<html><head></head><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:12px">I updated from 3.7.6 to 3.7.8 a few days ago, and now it looks like a number of things are broken including healing. <br><br>This is a cluster of 3 servers. One server is Ubuntu 14.04 using the PPA repo, and the other two are Proxmox 4 using the Debian Jessie repo.<br><br>"heal info" and "heal statistics" do not show any healing activity; everything shows as zero. But I have broken files that are not getting healed.<br><br>Doing "heal", "heal full", and "heal enable" all say success. But none seem to fix anything.<br><br><div id="yui_3_16_0_1_1456786439885_63659">I have tried with entry-self-heal/metdata-self-heal/data-self-heal set both on and off; neither seems to make a difference.</div><div id="yui_3_16_0_1_1456786439885_63711"><br></div><div id="yui_3_16_0_1_1456786439885_63712">I replaced a brick on a replicated volume. Some of the files are just not being replaced/updated on the second brick. Others have a few blocks written on the second brick but are not complete.<br></div><br>I don't know what to look for in the logs, but I do see a lot of messages in glustershd.log like this:<br><br>[2016-02-29 23:13:27.001474] W [MSGID: 108034] [afr-self-heald.c:445:afr_shd_index_sweep] 0-vmdisk2-replicate-0: unable to get index-dir on vmdisk2-client-1<br>[2016-02-29 23:13:27.001524] W [MSGID: 108034] [afr-self-heald.c:445:afr_shd_index_sweep] 0-public-replicate-0: unable to get index-dir on public-client-3<br>[2016-02-29 23:13:27.001547] W [MSGID: 108034] [afr-self-heald.c:445:afr_shd_index_sweep] 0-users-replicate-0: unable to get index-dir on users-client-6<br>[2016-02-29 23:13:27.001876] W [MSGID: 108034] [afr-self-heald.c:445:afr_shd_index_sweep] 0-vmdisk1-replicate-0: unable to get index-dir on vmdisk1-client-2<br>[2016-02-29 23:13:35.001555] W [MSGID: 108034] [afr-self-heald.c:445:afr_shd_index_sweep] 0-backups-local-replicate-0: unable to get index-dir on backups-local-client-2<br><br>On at least one replicated/distributed volume, I see duplicate directory entries (one with the actual file, and one zero-length placeholder)<br><br>-rw-rwSrw- 1 root 1004 255744366 Oct 18 2013 S03E05 - The One with Frank Jr.mp4<br>---------T 1 root 1004 0 Feb 22 08:55 S03E05 - The One with Frank Jr.mp4<br>-rw-rwSrw- 1 root 1004 255705796 Oct 18 2013 S03E06 - The One with the Flashback.mp4<br>---------T 1 root 1004 0 Feb 22 08:55 S03E06 - The One with the Flashback.mp4<br><br>This is *through the FUSE mount*, not looking directly at the bricks.<br><br>Anyone have any ideas on what I should look at? Thanks<br><br>- Alan<br><div id="yui_3_16_0_1_1456786439885_63715" dir="ltr"><br></div><div id="yui_3_16_0_1_1456786439885_63721"><br></div></div></body></html>