<div dir="ltr"><div class="gmail_extra"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>On Mon, Aug 29, 2016 at 5:25 AM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br></div></div></div></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Could you attach both client and brick logs? Meanwhile I will try these steps out on my machines and see if it is easily recreatable.<br><br></div></div></blockquote><div><br></div><div>Hoping 7z files are accepted by mail server.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div>-Krutika<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 29, 2016 at 2:31 PM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Centos 7 Gluster 3.8.3<div><br></div><div><div>Brick1: ccgl1.gl.local:/gluster1/BRICK<wbr>1/1</div><div>Brick2: ccgl2.gl.local:/gluster1/BRICK<wbr>1/1</div><div>Brick3: ccgl4.gl.local:/gluster1/BRICK<wbr>1/1</div><div>Options Reconfigured:</div><div>cluster.data-self-heal-algorit<wbr>hm: full</div><div>cluster.self-heal-daemon: on</div><div>cluster.locking-scheme: granular</div><div>features.shard-block-size: 64MB</div><div>features.shard: on</div><div>performance.readdir-ahead: on</div><div>storage.owner-uid: 36</div><div>storage.owner-gid: 36</div><div>performance.quick-read: off</div><div>performance.read-ahead: off</div><div>performance.io-cache: off</div><div>performance.stat-prefetch: on</div><div>cluster.eager-lock: enable</div><div>network.remote-dio: enable</div><div>cluster.quorum-type: auto</div><div>cluster.server-quorum-type: server</div><div>server.allow-insecure: on</div><div>cluster.self-heal-window-size: 1024</div><div>cluster.background-self-heal-c<wbr>ount: 16</div><div>performance.strict-write-order<wbr>ing: off</div><div>nfs.disable: on</div><div>nfs.addr-namelookup: off</div><div>nfs.enable-ino32: off</div><div>cluster.granular-entry-heal: on</div><div><br></div><div>Friday did rolling upgrade from 3.8.3->3.8.3 no issues.</div><div>Following steps detailed in previous recommendations began proces of replacing and healngbricks one node at a time.</div><div><br></div><div>1) kill pid of brick</div><div>2) reconfigure brick from raid6 to raid10</div><div>3) recreate directory of brick</div><div>4) gluster volume start <> force</div><div>5) gluster volume heal <> full</div><div><br></div><div>1st node worked as expected took 12 hours to heal 1TB data. Load was little heavy but nothing shocking.</div><div><br></div><div>About an hour after node 1 finished I began same process on node2. Heal proces kicked in as before and the files in directories visible from mount and .glusterfs healed in short time. Then it began crawl of .shard adding those files to heal count at which point the entire proces ground to a halt basically. After 48 hours out of 19k shards it has added 5900 to heal list. Load on all 3 machnes is negligible. It was suggested to change this value to full cluster.data-self-heal-algorit<wbr>hm and restart volume which I did. No efffect. Tried relaunching heal no effect, despite any node picked. I started each VM and performed a stat of all files from within it, or a full virus scan and that seemed to cause short small spikes in shards added, but not by much. Logs are showing no real messages indicating anything is going on. I get hits to brick log on occasion of null lookups making me think its not really crawling shards directory but waiting for a shard lookup to add it. I'll get following in brick log but not constant and sometime multiple for same shard.</div><div><br></div><div><div>[2016-08-29 08:31:57.478125] W [MSGID: 115009] [server-resolve.c:569:server_r<wbr>esolve] 0-GLUSTER1-server: no resolution type for (null) (LOOKUP)</div><div>[2016-08-29 08:31:57.478170] E [MSGID: 115050] [server-rpc-fops.c:156:server_<wbr>lookup_cbk] 0-GLUSTER1-server: 12591783: LOOKUP (null) (00000000-0000-0000-00</div><div>00-000000000000/241a55ed-f0d5-<wbr>4dbc-a6ce-ab784a0ba6ff.221) ==> (Invalid argument) [Invalid argument]</div></div><div><br></div><div>This one repeated about 30 times in row then nothing for 10 minutes then one hit for one different shard by itself.</div><div><br></div><div>How can I determine if Heal is actually running? How can I kill it or force restart? Does node I start it from determine which directory gets crawled to determine heals?</div><span><font color="#888888"><div><br></div><div><div data-smartmail="gmail_signature"><div dir="ltr"><span><font color="#888888"><span style="color:rgb(0,0,0)"><b><i>David Gossage</i></b></span><font><i><span style="color:rgb(51,51,51)"><b><br>
</b></span></i></font></font></span><div><span><font color="#888888"><font><i><span style="color:rgb(51,51,51)"></span></i><font size="1"><b style="color:rgb(153,0,0)">Carousel Checks Inc.<span style="color:rgb(204,204,204)"> | System Administrator</span></b></font></font><font style="color:rgb(153,153,153)"><font size="1"><br>
</font></font><font><font size="1"><span style="color:rgb(51,51,51)"><b style="color:rgb(153,153,153)">Office</b><span style="color:rgb(153,153,153)"> <a value="+17086132426">708.613.2284<font color="#888888"><font size="1"><br></font></font></a></span></span></font></font></font></span></div></div></div></div>
</font></span></div></div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div><br></div></div>