<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Aug 30, 2016 at 8:52 AM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div><div data-smartmail="gmail_signature"><div dir="ltr"><div>On Tue, Aug 30, 2016 at 8:01 AM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br></div></div></div></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 30, 2016 at 6:20 PM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Tue, Aug 30, 2016 at 6:07 PM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Tue, Aug 30, 2016 at 7:18 AM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Could you also share the glustershd logs?<br></div></div></div></blockquote><div><br></div></span><div>I'll get them when I get to work sure </div></div></div></div></blockquote></span></div></div></div></blockquote></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><br></div>I tried the same steps that you mentioned multiple times, but heal is running to completion without any issues.<br><br></div><div>It must be said that 'heal full' traverses the files and directories in a depth-first order and does heals also in the same order. But if it gets interrupted in the middle (say because self-heal-daemon was either intentionally or unintentionally brought offline and then brought back up), self-heal will only pick up the entries that are so far marked as new-entries that need heal which it will find in indices/xattrop directory. What this means is that those files and directories that were not visited during the crawl, will remain untouched and unhealed in this second iteration of heal, unless you execute a 'heal-full' again.<br></div></div></blockquote><div><br></div></span><div>So should it start healing shards as it crawls or not until after it crawls the entire .shard directory? At the pace it was going that could be a week with one node appearing in the cluster but with no shard files if anything tries to access a file on that node. From my experience other day telling it to heal full again did nothing regardless of node used.</div></div></div></div></blockquote></span></div></div></div></blockquote><div><br></div><div>Crawl is started from '/' of the volume. Whenever self-heal detects during the crawl that a file or directory is present in some brick(s) and absent in others, it creates the file on the bricks where it is absent and marks the fact that the file or directory might need data/entry and metadata heal too (this also means that an index is created under .glusterfs/indices/xattrop of the src bricks). And the data/entry and metadata heal are picked up and done in </div></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>the background with the help of these indices.<br></div></div></div></div></blockquote><div><br></div><div>Looking at my 3rd node as example i find nearly an exact same number of files in xattrop dir as reported by heal count at time I brought down node2 to try and alleviate read io errors that seemed to occur from what I was guessing as attempts to use the node with no shards for reads.</div><div><br></div><div>Also attached are the glustershd logs from the 3 nodes, along with the test node i tried yesterday with same results.</div></div></div></div></blockquote><div><br></div><div>Is it possible you just need to spam the heal full command? Wait for a certain amount of time for it to time out? </div><div><br></div><div>The test server that I did yesterday that stopped at listing 33 shards then healing none of them stlll had 33 shards in list this morning. I issued another heal full and it jumped up and found the missing shards.</div><div><br></div><div>On the one hand its reassuring that if I just spam the command enough eventually it will heal. It's also disconcerting that if I spam the command enough times the heal will start.</div><div><br></div><div>I can't test if same behavior would occur on live node as I expect if it did kick in heals I'd have 12 hours of high load during copy again perhaps. But I can test if it happens after last shift. Though I lost track of how many times I tried restarting heal full over Saturday and Sunday when it looked to be doing nothing from all heal tracking commands documented.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>My suspicion is that this is what happened on your setup. Could you confirm if that was the case?<br></div></div></blockquote><div><br></div></span><div>Brick was brought online with force start then a full heal launched. Hours later after it became evident that it was not adding new files to heal I did try restarting self-heal daemon and relaunching full heal again. But this was after the heal had basically already failed to work as intended.</div></div></div></div></blockquote><div><br></div></span><div>OK. How did you figure it was not adding any new files? I need to know what places you were monitoring to come to this conclusion.<span><font color="#888888"><br><br></font></span></div><span><font color="#888888"><div>-Krutika<br> <br></div></font></span><div><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>As for those logs, I did manager to do something that caused these warning messages you shared earlier to appear in my client and server logs.<br>Although these logs are annoying and a bit scary too, they didn't do any harm to the data in my volume. Why they appear just after a brick is replaced and under no other circumstances is something I'm still investigating.<br></div><div><br></div><div>But for future, it would be good to follow the steps Anuradha gave as that would allow self-heal to at least detect that it has some repairing to do whenever it is restarted whether intentionally or otherwise.<br></div></div></blockquote><div><br></div></span><div>I followed those steps as described on my test box and ended up with exact same outcome of adding shards at an agonizing slow pace and no creation of .shard directory or heals on shard directory. Directories visible from mount healed quickly. This was with one VM so it has only 800 shards as well. After hours at work it had added a total of 33 shards to be healed. I sent those logs yesterday as well though not the glustershd.</div><div><br></div><div>Does replace-brick command copy files in same manner? For these purposes I am contemplating just skipping the heal route.</div><div><div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div><div><br></div><div>-Krutika<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 30, 2016 at 2:22 AM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div><div data-smartmail="gmail_signature"><div dir="ltr"><div>attached brick and client logs from test machine where same behavior occurred not sure if anything new is there. its still on 3.8.2</div><div><br></div><div><div>Number of Bricks: 1 x 3 = 3</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: 192.168.71.10:/gluster2/brick1<wbr>/1</div><div>Brick2: 192.168.71.11:/gluster2/brick2<wbr>/1</div><div>Brick3: 192.168.71.12:/gluster2/brick3<wbr>/1</div><div>Options Reconfigured:</div><div>cluster.locking-scheme: granular</div><div>performance.strict-o-direct: off</div><div>features.shard-block-size: 64MB</div><div>features.shard: on</div><div>server.allow-insecure: on</div><div>storage.owner-uid: 36</div><div>storage.owner-gid: 36</div><div>cluster.server-quorum-type: server</div><div>cluster.quorum-type: auto</div><div>network.remote-dio: on</div><div>cluster.eager-lock: enable</div><div>performance.stat-prefetch: off</div><div>performance.io-cache: off</div><div>performance.quick-read: off</div><div>cluster.self-heal-window-size: 1024</div><div>cluster.background-self-heal-c<wbr>ount: 16</div><div>nfs.enable-ino32: off</div><div>nfs.addr-namelookup: off</div><div>nfs.disable: on</div><div>performance.read-ahead: off</div><div>performance.readdir-ahead: on</div><div>cluster.granular-entry-heal: on</div></div><div><br></div><div><br></div><div><br></div></div></div></div><div><div><div class="gmail_quote">On Mon, Aug 29, 2016 at 2:20 PM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div><div data-smartmail="gmail_signature"><div dir="ltr"><div>On Mon, Aug 29, 2016 at 7:01 AM, Anuradha Talur <span dir="ltr"><<a href="mailto:atalur@redhat.com" target="_blank">atalur@redhat.com</a>></span> wrote:<br></div></div></div></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><br>
<br>
----- Original Message -----<br>
> From: "David Gossage" <<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>><br>
> To: "Anuradha Talur" <<a href="mailto:atalur@redhat.com" target="_blank">atalur@redhat.com</a>><br>
> Cc: "<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a> List" <<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>>, "Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>><br>
> Sent: Monday, August 29, 2016 5:12:42 PM<br>
> Subject: Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow<br>
><br>
> On Mon, Aug 29, 2016 at 5:39 AM, Anuradha Talur <<a href="mailto:atalur@redhat.com" target="_blank">atalur@redhat.com</a>> wrote:<br>
><br>
> > Response inline.<br>
> ><br>
> > ----- Original Message -----<br>
> > > From: "Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>><br>
> > > To: "David Gossage" <<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>><br>
> > > Cc: "<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a> List" <<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>><br>
> > > Sent: Monday, August 29, 2016 3:55:04 PM<br>
> > > Subject: Re: [Gluster-users] 3.8.3 Shards Healing Glacier Slow<br>
> > ><br>
> > > Could you attach both client and brick logs? Meanwhile I will try these<br>
> > steps<br>
> > > out on my machines and see if it is easily recreatable.<br>
> > ><br>
> > > -Krutika<br>
> > ><br>
> > > On Mon, Aug 29, 2016 at 2:31 PM, David Gossage <<br>
> > <a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a><br>
> > > > wrote:<br>
> > ><br>
> > ><br>
> > ><br>
> > > Centos 7 Gluster 3.8.3<br>
> > ><br>
> > > Brick1: ccgl1.gl.local:/gluster1/BRICK<wbr>1/1<br>
> > > Brick2: ccgl2.gl.local:/gluster1/BRICK<wbr>1/1<br>
> > > Brick3: ccgl4.gl.local:/gluster1/BRICK<wbr>1/1<br>
> > > Options Reconfigured:<br>
> > > cluster.data-self-heal-algorit<wbr>hm: full<br>
> > > cluster.self-heal-daemon: on<br>
> > > cluster.locking-scheme: granular<br>
> > > features.shard-block-size: 64MB<br>
> > > features.shard: on<br>
> > > performance.readdir-ahead: on<br>
> > > storage.owner-uid: 36<br>
> > > storage.owner-gid: 36<br>
> > > performance.quick-read: off<br>
> > > performance.read-ahead: off<br>
> > > performance.io-cache: off<br>
> > > performance.stat-prefetch: on<br>
> > > cluster.eager-lock: enable<br>
> > > network.remote-dio: enable<br>
> > > cluster.quorum-type: auto<br>
> > > cluster.server-quorum-type: server<br>
> > > server.allow-insecure: on<br>
> > > cluster.self-heal-window-size: 1024<br>
> > > cluster.background-self-heal-c<wbr>ount: 16<br>
> > > performance.strict-write-order<wbr>ing: off<br>
> > > nfs.disable: on<br>
> > > nfs.addr-namelookup: off<br>
> > > nfs.enable-ino32: off<br>
> > > cluster.granular-entry-heal: on<br>
> > ><br>
> > > Friday did rolling upgrade from 3.8.3->3.8.3 no issues.<br>
> > > Following steps detailed in previous recommendations began proces of<br>
> > > replacing and healngbricks one node at a time.<br>
> > ><br>
> > > 1) kill pid of brick<br>
> > > 2) reconfigure brick from raid6 to raid10<br>
> > > 3) recreate directory of brick<br>
> > > 4) gluster volume start <> force<br>
> > > 5) gluster volume heal <> full<br>
> > Hi,<br>
> ><br>
> > I'd suggest that full heal is not used. There are a few bugs in full heal.<br>
> > Better safe than sorry ;)<br>
> > Instead I'd suggest the following steps:<br>
> ><br>
> > Currently I brought the node down by systemctl stop glusterd as I was<br>
> getting sporadic io issues and a few VM's paused so hoping that will help.<br>
> I may wait to do this till around 4PM when most work is done in case it<br>
> shoots load up.<br>
><br>
><br>
> > 1) kill pid of brick<br>
> > 2) to configuring of brick that you need<br>
> > 3) recreate brick dir<br>
> > 4) while the brick is still down, from the mount point:<br>
> > a) create a dummy non existent dir under / of mount.<br>
> ><br>
><br>
> so if noee 2 is down brick, pick node for example 3 and make a test dir<br>
> under its brick directory that doesnt exist on 2 or should I be dong this<br>
> over a gluster mount?<br>
You should be doing this over gluster mount.<br>
><br>
> > b) set a non existent extended attribute on / of mount.<br>
> ><br>
><br>
> Could you give me an example of an attribute to set? I've read a tad on<br>
> this, and looked up attributes but haven't set any yet myself.<br>
><br>
Sure. setfattr -n "user.some-name" -v "some-value" <path-to-mount><br>
> Doing these steps will ensure that heal happens only from updated brick to<br>
> > down brick.<br>
> > 5) gluster v start <> force<br>
> > 6) gluster v heal <><br>
> ><br>
><br>
> Will it matter if somewhere in gluster the full heal command was run other<br>
> day? Not sure if it eventually stops or times out.<br>
><br>
full heal will stop once the crawl is done. So if you want to trigger heal again,<br>
run gluster v heal <>. Actually even brick up or volume start force should<br>
trigger the heal.<br>
</blockquote><div><br></div><div>Did this on test bed today. its one server with 3 bricks on same machine so take that for what its worth. also it still runs 3.8.2. Maybe ill update and re-run test.</div><div><br></div><div>killed brick</div><div>deleted brick dir</div><div>recreated brick dir</div><div>created fake dir on gluster mount</div><div>set suggested fake attribute on it</div><div>ran volume start <> force</div><div><br></div><div>looked at files it said needed healing and it was just 8 shards that were modified for few minutes I ran through steps</div><div><br></div><div>gave it few minutes and it stayed same</div><div>ran gluster volume <> heal</div><div><br></div><div>it healed all the directories and files you can see over mount including fakedir. </div><div><br></div><div>same issue for shards though. it adds more shards to heal at glacier pace. slight jump in speed if I stat every file and dir in VM running but not all shards.</div><div><br></div><div>It started with 8 shards to heal and is now only at 33 out of 800 and probably wont finish adding for few days at rate it goes.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">> ><br>
> > > 1st node worked as expected took 12 hours to heal 1TB data. Load was<br>
> > little<br>
> > > heavy but nothing shocking.<br>
> > ><br>
> > > About an hour after node 1 finished I began same process on node2. Heal<br>
> > > proces kicked in as before and the files in directories visible from<br>
> > mount<br>
> > > and .glusterfs healed in short time. Then it began crawl of .shard adding<br>
> > > those files to heal count at which point the entire proces ground to a<br>
> > halt<br>
> > > basically. After 48 hours out of 19k shards it has added 5900 to heal<br>
> > list.<br>
> > > Load on all 3 machnes is negligible. It was suggested to change this<br>
> > value<br>
> > > to full cluster.data-self-heal-algorit<wbr>hm and restart volume which I<br>
> > did. No<br>
> > > efffect. Tried relaunching heal no effect, despite any node picked. I<br>
> > > started each VM and performed a stat of all files from within it, or a<br>
> > full<br>
> > > virus scan and that seemed to cause short small spikes in shards added,<br>
> > but<br>
> > > not by much. Logs are showing no real messages indicating anything is<br>
> > going<br>
> > > on. I get hits to brick log on occasion of null lookups making me think<br>
> > its<br>
> > > not really crawling shards directory but waiting for a shard lookup to<br>
> > add<br>
> > > it. I'll get following in brick log but not constant and sometime<br>
> > multiple<br>
> > > for same shard.<br>
> > ><br>
> > > [2016-08-29 08:31:57.478125] W [MSGID: 115009]<br>
> > > [server-resolve.c:569:server_r<wbr>esolve] 0-GLUSTER1-server: no resolution<br>
> > type<br>
> > > for (null) (LOOKUP)<br>
> > > [2016-08-29 08:31:57.478170] E [MSGID: 115050]<br>
> > > [server-rpc-fops.c:156:server_<wbr>lookup_cbk] 0-GLUSTER1-server: 12591783:<br>
> > > LOOKUP (null) (00000000-0000-0000-00<br>
> > > 00-000000000000/241a55ed-f0d5-<wbr>4dbc-a6ce-ab784a0ba6ff.221) ==> (Invalid<br>
> > > argument) [Invalid argument]<br>
> > ><br>
> > > This one repeated about 30 times in row then nothing for 10 minutes then<br>
> > one<br>
> > > hit for one different shard by itself.<br>
> > ><br>
> > > How can I determine if Heal is actually running? How can I kill it or<br>
> > force<br>
> > > restart? Does node I start it from determine which directory gets<br>
> > crawled to<br>
> > > determine heals?<br>
> > ><br>
> > > David Gossage<br>
> > > Carousel Checks Inc. | System Administrator<br>
> > > Office <a href="tel:708.613.2284" value="+17086132284" target="_blank">708.613.2284</a><br>
> > ><br>
> > > ______________________________<wbr>_________________<br>
> > > Gluster-users mailing list<br>
> > > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> > > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a><br>
> > ><br>
> > ><br>
> > > ______________________________<wbr>_________________<br>
> > > Gluster-users mailing list<br>
> > > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> > > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a><br>
> ><br>
> > --<br>
> > Thanks,<br>
> > Anuradha.<br>
> ><br>
><span class="HOEnZb"><font color="#888888"><span><font color="#888888"><span><font color="#888888"><span><font color="#888888"><br>
<span><font color="#888888"><br>
--<br>
Thanks,<br>
Anuradha.<br>
</font></span></font></span></font></span></font></span></font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><span><font color="#888888"><span><font color="#888888"><br></font></span></font></span></font></span></div></div><span class="HOEnZb"><font color="#888888"><span><font color="#888888"><span><font color="#888888">
</font></span></font></span></font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><span><font color="#888888"><span><font color="#888888"><br></font></span></font></span></font></span></div></div></div></div><span class="HOEnZb"><font color="#888888"><span><font color="#888888">
</font></span></font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><span><font color="#888888"><br></font></span></font></span></div><span class="HOEnZb"><font color="#888888"><span><font color="#888888">
</font></span></font></span></blockquote></div></div></div><span class="HOEnZb"><font color="#888888"><span><font color="#888888"><br></font></span></font></span></div></div><span class="HOEnZb"><font color="#888888"><span><font color="#888888">
</font></span></font></span></blockquote></div></div></div><span class="HOEnZb"><font color="#888888"><span><font color="#888888"><br></font></span></font></span></div></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br></font></span></div></div>
</blockquote></div><br></div></div>
</blockquote></div><br></div></div>