<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div>Hi Nicolas,<br></div><div><br></div><div>I think this issue has already been raised where we are seeing different heal info from different servers.<br></div><div><br></div><div><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1335429"><a>https://bugzilla.redhat.com/show_bug.cgi?id=1335429</a></a></div><div>Patch for this is under review.<br></div><div><br></div><div>Ashish<br></div><div><br></div><div><br></div><div><br></div><hr id="zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Nicolas Ecarnot" <nicolas@ecarnot.net><br><b>To: </b>"gluster-users" <Gluster-users@gluster.org><br><b>Sent: </b>Thursday, May 12, 2016 8:38:53 PM<br><b>Subject: </b>[Gluster-users] Continual heals happening on sharded cluster, me too...<br><div><br></div>Hello,<br><div><br></div>On a replica 3 sharded cluster in gluster 3.7.11 (CentOS 7.2), I've <br>witnessed a very sound behaviour for months.<br>Recently, I've upgraded from [something quite recent, like <br>3.7.something] to 3.7.11, and immediately, I'm seeing continual healing <br>appearing.<br><div><br></div>I'm using this cluster to host 3 oVirt VMs with very low I/O.<br><div><br></div>I read and tried to use informations here :<br>https://www.mail-archive.com/gluster-users%40gluster.org/msg24598.html<br>but I'm missing knowledge to know how to fix things.<br><div><br></div>Here what I see when running "gluster volume heal data-shard-03 info" :<br><div><br></div>* serv-vm-al01<br><div><br></div>Brick serv-vm-al01:/gluster/data/brick03<br>Status: Connected<br>Number of entries: 0<br><div><br></div>Brick serv-vm-al02:/gluster/data/brick03<br>Status: Connected<br>Number of entries: 0<br><div><br></div>Brick serv-vm-al03:/gluster/data/brick03<br>Status: Connected<br>Number of entries: 0<br><div><br></div>* serv-vm-al02<br><div><br></div>gluster volume heal data-shard-03 info<br>Brick serv-vm-al01:/gluster/data/brick03<br><gfid:75ab0b1c-258f-4349-85bd-49ef80592919><br><gfid:39bdcf05-f5b1-4df2-9941-614838b50e18><br><gfid:7e0d84a0-0749-4a2d-9390-e95d489ec66a><br>Status: Connected<br>Number of entries: 3<br><div><br></div>Brick serv-vm-al02:/gluster/data/brick03<br>/.shard/41573624-feb9-4ea6-bbd4-f0a912429b2f.1003<br>/.shard/26ec536a-8919-478c-834c-f6ac70882ee6.2351<br>/.shard/26ec536a-8919-478c-834c-f6ac70882ee6.1113<br>Status: Connected<br>Number of entries: 3<br><div><br></div>Brick serv-vm-al03:/gluster/data/brick03<br>/.shard/34948b18-5991-4761-b867-2bd4fe4879d4.2807<br>/.shard/26ec536a-8919-478c-834c-f6ac70882ee6.2351<br>/.shard/34948b18-5991-4761-b867-2bd4fe4879d4.2062<br>/.shard/26ec536a-8919-478c-834c-f6ac70882ee6.1113<br>/.shard/41573624-feb9-4ea6-bbd4-f0a912429b2f.1833<br>Status: Connected<br>Number of entries: 5<br><div><br></div>* serv-vm-al03<br><div><br></div>Brick serv-vm-al01:/gluster/data/brick03<br>/.shard/41573624-feb9-4ea6-bbd4-f0a912429b2f.882<br><gfid:39bdcf05-f5b1-4df2-9941-614838b50e18><br><gfid:93c83d37-71e5-4f17-b757-ff00a4384f2f><br><gfid:75fcdab7-6d18-4371-bbb2-be156dd3da86><br><gfid:75ab0b1c-258f-4349-85bd-49ef80592919><br><gfid:f933a07a-4f89-4bf9-ab82-a05432a0162f><br>/.shard/34948b18-5991-4761-b867-2bd4fe4879d4.2329<br>Status: Connected<br>Number of entries: 7<br><div><br></div>Brick serv-vm-al02:/gluster/data/brick03<br>/.shard/26ec536a-8919-478c-834c-f6ac70882ee6.1113<br>/.shard/34948b18-5991-4761-b867-2bd4fe4879d4.2807<br>/.shard/41573624-feb9-4ea6-bbd4-f0a912429b2f.190<br>/.shard/26ec536a-8919-478c-834c-f6ac70882ee6.2351<br>Status: Connected<br>Number of entries: 4<br><div><br></div>Brick serv-vm-al03:/gluster/data/brick03<br>/.shard/26ec536a-8919-478c-834c-f6ac70882ee6.1113<br>/.shard/26ec536a-8919-478c-834c-f6ac70882ee6.185<br>/.shard/26ec536a-8919-478c-834c-f6ac70882ee6.183<br>/.shard/26ec536a-8919-478c-834c-f6ac70882ee6.2351<br>Status: Connected<br>Number of entries: 4<br><div><br></div>We are monitoring healed entries in Nagios/centreon, and history graph <br>are showing that from the time I upgraded to 3.7.11, it went from around <br>0~1 files to heal every 5 minutes, to an average of 6 to 10 files to <br>heal every minutes.<br>The VMs are still doing the exact same nothing as previous.<br><div><br></div>Questions :<br>- how comes?<br>- how to fix?<br><div><br></div>-- <br>Nicolas ECARNOT<br>_______________________________________________<br>Gluster-users mailing list<br>Gluster-users@gluster.org<br>http://www.gluster.org/mailman/listinfo/gluster-users<br></div><div><br></div></div></body></html>