<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">I noticed that my new brick (replacement disk) did not have a .shard directory created on the brick, if that helps. <div class=""><br class=""></div><div class="">I removed the affected brick from the volume and then wiped the disk, did an add-brick, and everything healed right up. I didn’t try and set any attrs or anything else, just removed and added the brick as new.</div><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On Aug 29, 2016, at 9:49 AM, Darrell Budic <<a href="mailto:budic@onholyground.com" class="">budic@onholyground.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Just to let you know I’m seeing the same issue under 3.7.14 on CentOS 7. Some content was healed correctly, now all the shards are queued up in a heal list, but nothing is healing. Got similar brick errors logged to the ones David was getting on the brick that isn’t healing:<div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures;" class="">[2016-08-29 03:31:40.436110] E [MSGID: 115050] [server-rpc-fops.c:179:server_lookup_cbk] 0-gv0-rep-server: 1613822: LOOKUP (null) (</span><span style="font-variant-ligatures: no-common-ligatures;" class="">000000</span><span style="font-variant-ligatures: no-common-ligatures;" class="">00-0000-0000-0000-</span><span style="font-variant-ligatures: no-common-ligatures;" class="">000000000000</span><span style="font-variant-ligatures: no-common-ligatures;" class="">/0f61bf63-8ef1-4e53-8bc3-6d46590c4fb1.29) ==> (Invalid argument) [Invalid argument]</span></div><div style="margin: 0px; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2016-08-29 03:31:43.005013] E [MSGID: 115050] [server-rpc-fops.c:179:server_lookup_cbk] 0-gv0-rep-server: 1616802: LOOKUP (null) (</span><span style="font-variant-ligatures: no-common-ligatures;" class="">000000</span><span style="font-variant-ligatures: no-common-ligatures" class="">00-0000-0000-0000-</span><span style="font-variant-ligatures: no-common-ligatures;" class="">000000000000</span><span style="font-variant-ligatures: no-common-ligatures" class="">/0f61bf63-8ef1-4e53-8bc3-6d46590c4fb1.40) ==> (Invalid argument) [Invalid argument]</span></div><div class=""><br class=""></div><div class="">This was after replacing the drive the brick was on and trying to get it back into the system by setting the volume's fattr on the brick dir. I’ll try the suggested method here on it it shortly.</div><div class=""><br class=""></div><div class=""> -Darrell</div><div class=""><br class=""></div><div class=""><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Aug 29, 2016, at 7:25 AM, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" class="">kdhananj@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class=""><div class="">Got it. Thanks.<br class=""><br class=""></div>I tried the same test and shd crashed with SIGABRT (well, that's because I compiled from src with -DDEBUG).<br class="">In any case, this error would prevent full heal from proceeding further.<br class=""></div><div class="">I'm debugging the crash now. Will let you know when I have the RC.<br class=""></div><div class=""><br class=""></div>-Krutika<br class=""></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Mon, Aug 29, 2016 at 5:47 PM, David Gossage <span dir="ltr" class=""><<a href="mailto:dgossage@carouselchecks.com" target="_blank" class="">dgossage@carouselchecks.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class=""><div class="gmail_extra"><div class=""><div data-smartmail="gmail_signature" class=""><div dir="ltr" class=""><div class=""><br class=""></div></div></div></div><div class="gmail_quote"><span class="">On Mon, Aug 29, 2016 at 7:14 AM, David Gossage <span dir="ltr" class=""><<a href="mailto:dgossage@carouselchecks.com" target="_blank" class="">dgossage@carouselchecks.com</a>></span> wrote:<br class=""></span><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class=""><div class="gmail_extra"><div class=""><div data-smartmail="gmail_signature" class=""><div dir="ltr" class=""><div class="">On Mon, Aug 29, 2016 at 5:25 AM, Krutika Dhananjay <span dir="ltr" class=""><<a href="mailto:kdhananj@redhat.com" target="_blank" class="">kdhananj@redhat.com</a>></span> wrote:<br class=""></div></div></div></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class=""><div class="">Could you attach both client and brick logs? Meanwhile I will try these steps out on my machines and see if it is easily recreatable.<br class=""><br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">Hoping 7z files are accepted by mail server.</div></div></div></div></blockquote><div class=""><br class=""></div></span><div class="">looks like zip file awaiting approval due to size </div><div class=""><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class=""><div class=""></div>-Krutika<br class=""></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Mon, Aug 29, 2016 at 2:31 PM, David Gossage <span dir="ltr" class=""><<a href="mailto:dgossage@carouselchecks.com" target="_blank" class="">dgossage@carouselchecks.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="">Centos 7 Gluster 3.8.3<div class=""><br class=""></div><div class=""><div class="">Brick1: ccgl1.gl.local:/gluster1/BRICK<wbr class="">1/1</div><div class="">Brick2: ccgl2.gl.local:/gluster1/BRICK<wbr class="">1/1</div><div class="">Brick3: ccgl4.gl.local:/gluster1/BRICK<wbr class="">1/1</div><div class="">Options Reconfigured:</div><div class="">cluster.data-self-heal-algorit<wbr class="">hm: full</div><div class="">cluster.self-heal-daemon: on</div><div class="">cluster.locking-scheme: granular</div><div class="">features.shard-block-size: 64MB</div><div class="">features.shard: on</div><div class="">performance.readdir-ahead: on</div><div class="">storage.owner-uid: 36</div><div class="">storage.owner-gid: 36</div><div class="">performance.quick-read: off</div><div class="">performance.read-ahead: off</div><div class=""><a href="http://performance.io/" class="">performance.io</a>-cache: off</div><div class="">performance.stat-prefetch: on</div><div class="">cluster.eager-lock: enable</div><div class="">network.remote-dio: enable</div><div class="">cluster.quorum-type: auto</div><div class="">cluster.server-quorum-type: server</div><div class="">server.allow-insecure: on</div><div class="">cluster.self-heal-window-size: 1024</div><div class="">cluster.background-self-heal-c<wbr class="">ount: 16</div><div class="">performance.strict-write-order<wbr class="">ing: off</div><div class="">nfs.disable: on</div><div class="">nfs.addr-namelookup: off</div><div class="">nfs.enable-ino32: off</div><div class="">cluster.granular-entry-heal: on</div><div class=""><br class=""></div><div class="">Friday did rolling upgrade from 3.8.3->3.8.3 no issues.</div><div class="">Following steps detailed in previous recommendations began proces of replacing and healngbricks one node at a time.</div><div class=""><br class=""></div><div class="">1) kill pid of brick</div><div class="">2) reconfigure brick from raid6 to raid10</div><div class="">3) recreate directory of brick</div><div class="">4) gluster volume start <> force</div><div class="">5) gluster volume heal <> full</div><div class=""><br class=""></div><div class="">1st node worked as expected took 12 hours to heal 1TB data. Load was little heavy but nothing shocking.</div><div class=""><br class=""></div><div class="">About an hour after node 1 finished I began same process on node2. Heal proces kicked in as before and the files in directories visible from mount and .glusterfs healed in short time. Then it began crawl of .shard adding those files to heal count at which point the entire proces ground to a halt basically. After 48 hours out of 19k shards it has added 5900 to heal list. Load on all 3 machnes is negligible. It was suggested to change this value to full cluster.data-self-heal-algorit<wbr class="">hm and restart volume which I did. No efffect. Tried relaunching heal no effect, despite any node picked. I started each VM and performed a stat of all files from within it, or a full virus scan and that seemed to cause short small spikes in shards added, but not by much. Logs are showing no real messages indicating anything is going on. I get hits to brick log on occasion of null lookups making me think its not really crawling shards directory but waiting for a shard lookup to add it. I'll get following in brick log but not constant and sometime multiple for same shard.</div><div class=""><br class=""></div><div class=""><div class="">[2016-08-29 08:31:57.478125] W [MSGID: 115009] [server-resolve.c:569:server_r<wbr class="">esolve] 0-GLUSTER1-server: no resolution type for (null) (LOOKUP)</div><div class="">[2016-08-29 08:31:57.478170] E [MSGID: 115050] [server-rpc-fops.c:156:server_<wbr class="">lookup_cbk] 0-GLUSTER1-server: 12591783: LOOKUP (null) (00000000-0000-0000-00</div><div class="">00-000000000000/241a55ed-f0d5-<wbr class="">4dbc-a6ce-ab784a0ba6ff.221) ==> (Invalid argument) [Invalid argument]</div></div><div class=""><br class=""></div><div class="">This one repeated about 30 times in row then nothing for 10 minutes then one hit for one different shard by itself.</div><div class=""><br class=""></div><div class="">How can I determine if Heal is actually running? How can I kill it or force restart? Does node I start it from determine which directory gets crawled to determine heals?</div><span class=""><font color="#888888" class=""><div class=""><br class=""></div><div class=""><div data-smartmail="gmail_signature" class=""><div dir="ltr" class=""><span class=""><font color="#888888" class=""><span style="" class=""><b class=""><i class="">David Gossage</i></b></span><font class=""><i class=""><span style="color:rgb(51,51,51)" class=""><b class=""><br class="">
</b></span></i></font></font></span><div class=""><span class=""><font color="#888888" class=""><font class=""><i class=""><span style="color:rgb(51,51,51)" class=""></span></i><font size="1" class=""><b style="color:rgb(153,0,0)" class="">Carousel Checks Inc.<span style="color:rgb(204,204,204)" class=""> | System Administrator</span></b></font></font><font style="color:rgb(153,153,153)" class=""><font size="1" class=""><br class="">
</font></font><font class=""><font size="1" class=""><span style="color:rgb(51,51,51)" class=""><b style="color:rgb(153,153,153)" class="">Office</b><span style="color:rgb(153,153,153)" class=""> <a value="+17086132426" class="">708.613.2284<font color="#888888" class=""><font size="1" class=""><br class=""></font></font></a></span></span></font></font></font></span></div></div></div></div>
</font></span></div></div>
<br class="">______________________________<wbr class="">_________________<br class="">
Gluster-users mailing list<br class="">
<a href="mailto:Gluster-users@gluster.org" target="_blank" class="">Gluster-users@gluster.org</a><br class="">
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank" class="">http://www.gluster.org/mailman<wbr class="">/listinfo/gluster-users</a><br class=""></blockquote></div><br class=""></div>
</blockquote></div><br class=""></div></div>
</blockquote></div></div></div><br class=""></div></div>
</blockquote></div><br class=""></div>
_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class=""><a href="http://www.gluster.org/mailman/listinfo/gluster-users" class="">http://www.gluster.org/mailman/listinfo/gluster-users</a></div></blockquote></div><br class=""></div></div></div>_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">http://www.gluster.org/mailman/listinfo/gluster-users</div></blockquote></div><br class=""></div></body></html>