<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi there.<br>
<br>
I'm running a replicated volume with 2 bricks, using GlusterFS
3.5.3. Both servers are running CentOS 7.0.<br>
<br>
The GlusterFS volume is only used to store VM disks images (not a
lot of files, but big files). I'm trying to debug a problem where
sometime, I/O from the VM POV is completely stuck for several
minutes (usually ~10 minutes, I'll open a new thread for this).
While looking at the volume status, I see something strange:<br>
<br>
gluster vol heal vmstore info healed<br>
<br>
This command should only show healed files, but I see tons of
entries (1024, which I think is the limit). Every files in use is
listed, as if self-heal would run continuously, eg:<br>
<br>
[...]<br>
2016-04-18 13:13:19 /qual/tse2k12_sys.qcow2<br>
2016-04-18 13:23:19 /prod/compta2015_sys.qcow2<br>
2016-04-18 13:23:19 /qual/tse2k12_sys.qcow2<br>
2016-04-18 13:33:19 /prod/syslog_sys.qcow2<br>
2016-04-18 13:33:19 /prod/ipasserelle_data.qcow2<br>
2016-04-18 13:33:19 /qual/tse2k12_sys.qcow2<br>
2016-04-18 13:43:19 /qual/wintest_sys.qcow2<br>
2016-04-18 13:43:19 /qual/tse2k12_sys.qcow2<br>
2016-04-18 13:43:20 /prod/ipasserelle_sys.qcow2<br>
2016-04-18 13:53:19 /qual/tse2k12_sys.qcow2<br>
2016-04-18 13:53:20 /qual/wintest_sys.qcow2<br>
2016-04-18 14:03:20 /prod/tel_var.qcow2<br>
2016-04-18 14:03:20 /qual/tse2k12_sys.qcow2<br>
2016-04-18 14:13:19 /qual/tse2k12_sys.qcow2<br>
2016-04-18 14:13:20 /prod/ipasserelle_data.qcow2<br>
2016-04-18 14:13:20 /prod/ipasserelle_sys.qcow2<br>
2016-04-18 14:13:21 /qual/report_sys.qcow2<br>
2016-04-18 14:23:19 /qual/tse2k12_sys.qcow2<br>
2016-04-18 14:23:21 /prod/ipasserelle_sys.qcow2<br>
2016-04-18 14:33:20 /qual/parana_sys.qcow2<br>
2016-04-18 14:33:21 /prod/ipasserelle_sys.qcow2<br>
2016-04-18 14:33:22 /qual/tse2k12_sys.qcow2<br>
2016-04-18 14:53:19 /prod/ipasserelle_var.qcow2<br>
2016-04-18 14:53:19 /qual/tse2k12_sys.qcow2<br>
2016-04-18 15:03:20 /qual/tse2k12_sys.qcow2<br>
2016-04-18 15:03:20 /prod/tel_var.qcow2<br>
2016-04-18 15:13:20 /prod/ipasserelle_data.qcow2<br>
2016-04-18 15:13:20 /prod/vigo_sys.qcow2<br>
2016-04-18 15:23:20 /prod/ipasserelle_data.qcow2<br>
2016-04-18 15:33:19 /qual/tse2k12_sys.qcow2<br>
2016-04-18 15:43:21 /prod/tel_var.qcow2<br>
2016-04-18 15:53:20 /qual/tse2k12_sys.qcow2<br>
2016-04-18 15:53:21 /qual/print_sys.qcow2<br>
2016-04-18 16:03:20 /prod/ged_sys.qcow2<br>
2016-04-18 16:13:21 /prod/ged_sys.qcow2<br>
2016-04-18 16:13:22 /qual/tse2k12_sys.qcow2<br>
<br>
Note: healed entries are not the same on brick1 and brick2<br>
<br>
<br>
But gluster vol heal vmstore info never shows any heal in progress
(I run it every 15 minutes). As most of those files are big (between
2GB and 2.8TB), if heal would take place, I'd notice it as it'd take
several hours to complete. I see nothing in /var/log/gluster related
to heal happening.<br>
<br>
It seems to be very regular, 10 min interval, and each time a
different file is marked as healed.<br>
<br>
I've digged a bit at what is running at this time, and I see that I
have a monitoring script which query S.M.A.R.T data exactly at this
time. I can't see any reason this could be linked to GlusterFS
seeing healed file while no heal took place, but that's the only
thing I see running at this interval.<br>
<br>
Any idea what this could be ? What can make gluster think files are
healed while nothing is logged, and no data heal took place ?<br>
<br>
Cheers<br>
Daniel<br>
<br>
<div class="moz-signature">-- <br>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<table>
<tbody>
<tr>
<td>
<p> <img
src="cid:part1.03070300.09060904@firewall-services.com"
alt="Logo FWS" width="275"> </p>
</td>
<td> <font face="Verdana, Geneva, sans-serif" size="2"> <strong>Daniel
Berteaud</strong><br>
<br>
FIREWALL-SERVICES SAS.<br>
Société de Services en Logiciels Libres<br>
Tel : 05 56 64 15 32<br>
Visio : <a class="moz-txt-link-freetext" href="http://vroom.im/dani">http://vroom.im/dani</a><br>
<em><a class="moz-txt-link-abbreviated" href="http://www.firewall-services.com">www.firewall-services.com</a></em> </font> </td>
</tr>
</tbody>
</table>
</div>
</body>
</html>