<div dir="ltr">If I read this correctly <div><br></div><div>[2016-07-20 09:06:39.072114] E [MSGID: 108008] [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch] 0-GLUSTER1-replicate-0: Gfid mismatch detected for <be318638-e8a0-4c6d-977d-7a937aa84806/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217>, bc2b3e90-8efe-4d75-acda-9c4cc9cf2800 on GLUSTER1-client-2 and a5cf59d7-c38a-4612-835d-3a294e70084d on GLUSTER1-client-0. Skipping conservative merge on the file.<br></div><div><br></div><div>the mismatch is between brick1 ccgl1 or client 0 and the new brick ccgl4 or client-2</div><div><br></div><div>Can I just tell it to use a specific brick as correct one as I expect client0/ccgl1 which had the correct data prior to kicking off post move brick heal to have the correct data. Or will this still involve powering off VM and reseting <span style="color:rgb(51,51,51);font-family:Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:13.6px;line-height:20.4px;background-color:rgba(0,0,0,0.0392157)">trusted.afr.GLUSTER1-client-# values? That method leaves me puzzled unless I am looking at old docs as I would expect to see 3 lines of </span>trusted.afr.GLUSTER1-client-#</div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><span><font color="#888888"><span style="color:rgb(0,0,0)"><b><i>David Gossage</i></b></span><font><i><span style="color:rgb(51,51,51)"><b><br>
</b></span></i></font></font></span><div><span><font color="#888888"><font><i><span style="color:rgb(51,51,51)"></span></i><font size="1"><b style="color:rgb(153,0,0)">Carousel Checks Inc.<span style="color:rgb(204,204,204)"> | System Administrator</span></b></font></font><font style="color:rgb(153,153,153)"><font size="1"><br>
</font></font><font><font size="1"><span style="color:rgb(51,51,51)"><b style="color:rgb(153,153,153)">Office</b><span style="color:rgb(153,153,153)"> <a value="+17086132426">708.613.2284<font color="#888888"><font size="1"><br></font></font></a></span></span></font></font></font></span></div></div></div></div>
<br><div class="gmail_quote">On Wed, Jul 20, 2016 at 4:29 AM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">picking one shard at random trying to follow <a href="https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md" target="_blank">https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md</a> to see if I can figure out procedure.<div>Shouldn't each shard have a trusted.afr.GLUSTER1-client-# for each of 3 bricks? First 2 servers have just 1, and 3rd new server has 2. So sort of at loss how to interpret that at the moment.<br><div><br></div><div><div>[root@ccgl1 ~]# getfattr -d -m . -e hex /gluster1/BRICK1/1/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: gluster1/BRICK1/1/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217</div><div>security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</div><div>trusted.afr.GLUSTER1-client-2=0x00001be30000000100000000</div><div>trusted.afr.dirty=0x000000000000000000000000</div><div>trusted.bit-rot.version=0x020000000000000057813fe9000a51b4</div><div>trusted.gfid=0xa5cf59d7c38a4612835d3a294e70084d</div></div><div><br></div><div><div>[root@ccgl2 ~]# getfattr -d -m . -e hex /gluster1/BRICK1/1/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: gluster1/BRICK1/1/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217</div><div>security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</div><div>trusted.afr.GLUSTER1-client-2=0x00001be30000000100000000</div><div>trusted.afr.dirty=0x000000000000000000000000</div><div>trusted.bit-rot.version=0x020000000000000057813fe9000b206f</div><div>trusted.gfid=0xa5cf59d7c38a4612835d3a294e70084d</div></div><div><br></div><div><div>[root@ccgl4 ~]# getfattr -d -m . -e hex /gluster1/BRICK1/1/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: gluster1/BRICK1/1/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217</div><div>security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65645f743a733000</div><div>trusted.afr.GLUSTER1-client-0=0x000000010000000100000000</div><div>trusted.afr.GLUSTER1-client-1=0x000000010000000100000000</div><div>trusted.gfid=0xbc2b3e908efe4d75acda9c4cc9cf2800</div></div><div><br></div><div><br></div></div></div><div class="gmail_extra"><br clear="all"><div><div data-smartmail="gmail_signature"><div dir="ltr"><span><font color="#888888"><span style="color:rgb(0,0,0)"><b><i>David Gossage</i></b></span><font><i><span style="color:rgb(51,51,51)"><b><br>
</b></span></i></font></font></span><div><span><font color="#888888"><font><i><span style="color:rgb(51,51,51)"></span></i><font size="1"><b style="color:rgb(153,0,0)">Carousel Checks Inc.<span style="color:rgb(204,204,204)"> | System Administrator</span></b></font></font><font style="color:rgb(153,153,153)"><font size="1"><br>
</font></font><font><font size="1"><span style="color:rgb(51,51,51)"><b style="color:rgb(153,153,153)">Office</b><span style="color:rgb(153,153,153)"> <a value="+17086132426">708.613.2284<font color="#888888"><font size="1"><br></font></font></a></span></span></font></font></font></span></div></div></div></div>
<br><div class="gmail_quote">On Wed, Jul 20, 2016 at 4:13 AM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Plenty of these in glustershd.log <div><br></div><div><div>[2016-07-20 09:06:39.072114] E [MSGID: 108008] [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch] 0-GLUSTER1-replicate-0: Gfid mismatch detected for <be318638-e8a0-4c6d-977d-7a937aa84806/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217>, bc2b3e90-8efe-4d75-acda-9c4cc9cf2800 on GLUSTER1-client-2 and a5cf59d7-c38a-4612-835d-3a294e70084d on GLUSTER1-client-0. Skipping conservative merge on the file.</div><div>[2016-07-20 09:06:41.546543] E [MSGID: 108008] [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch] 0-GLUSTER1-replicate-0: Gfid mismatch detected for <be318638-e8a0-4c6d-977d-7a937aa84806/18843fb4-e31c-4fc3-b519-cc6e5e947813.192>, 45ac549c-25e1-4842-b9ed-ca3205101465 on GLUSTER1-client-2 and 4c44b432-0b4d-4304-b030-e7a35d0dafc3 on GLUSTER1-client-0. Skipping conservative merge on the file.</div><div>[2016-07-20 09:06:42.484537] E [MSGID: 108008] [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch] 0-GLUSTER1-replicate-0: Gfid mismatch detected for <be318638-e8a0-4c6d-977d-7a937aa84806/91e12c0a-2557-4410-bf7d-834604f221f0.95>, dea6f79c-56e2-4831-9542-57f034ad2afb on GLUSTER1-client-2 and 6626519c-67f6-4e11-92a9-bf3c1b67cafd on GLUSTER1-client-0. Skipping conservative merge on the file.</div><div>[2016-07-20 09:06:43.799848] E [MSGID: 108008] [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch] 0-GLUSTER1-replicate-0: Gfid mismatch detected for <be318638-e8a0-4c6d-977d-7a937aa84806/d5a328be-03d0-42f7-a443-248290849e7d.48>, 844384f5-4443-445f-b4ea-b0641d1c9fae on GLUSTER1-client-2 and 97ac4b05-ac40-411d-981b-092556021aad on GLUSTER1-client-0. Skipping conservative merge on the file.</div><div>[2016-07-20 09:06:44.572868] E [MSGID: 108008] [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch] 0-GLUSTER1-replicate-0: Gfid mismatch detected for <be318638-e8a0-4c6d-977d-7a937aa84806/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10>, 59894562-d896-40dd-aa36-e612e74bac43 on GLUSTER1-client-2 and bcb2ab91-3bf6-477e-89db-66a69fac650e on GLUSTER1-client-0. Skipping conservative merge on the file.</div><div>[2016-07-20 09:06:47.279036] E [MSGID: 108008] [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch] 0-GLUSTER1-replicate-0: Gfid mismatch detected for <be318638-e8a0-4c6d-977d-7a937aa84806/585c8361-f6a9-48d4-9295-260ec657de1e.48>, 6fcab9a8-add1-4e05-affb-612b96457351 on GLUSTER1-client-2 and 0675dde6-e978-4d9d-aae9-eb8e3d11596e on GLUSTER1-client-0. Skipping conservative merge on the file.</div><div>[2016-07-20 09:06:47.706885] E [MSGID: 108008] [afr-self-heal-entry.c:253:afr_selfheal_detect_gfid_and_type_mismatch] 0-GLUSTER1-replicate-0: Gfid mismatch detected for <be318638-e8a0-4c6d-977d-7a937aa84806/18843fb4-e31c-4fc3-b519-cc6e5e947813.226>, 4314c4ad-d2a6-4352-b7f1-37cd3381fdf7 on GLUSTER1-client-2 and 5540f9cd-0f91-44e7-bfaa-74870a908030 on GLUSTER1-client-0. Skipping conservative merge on the file.</div></div><div><br></div><div>also as opposed to earlier I know see a result in split brain info</div><div><br></div><div><div>Brick ccgl1.gl.local:/gluster1/BRICK1/1</div><div>/.shard</div><div>Status: Connected</div><div>Number of entries in split-brain: 1</div><div><br></div><div>Brick ccgl2.gl.local:/gluster1/BRICK1/1</div><div>/.shard</div><div>Status: Connected</div><div>Number of entries in split-brain: 1</div><div><br></div><div>Brick ccgl4.gl.local:/gluster1/BRICK1/1</div><div>Status: Connected</div><div>Number of entries in split-brain: 0</div></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div data-smartmail="gmail_signature"><div dir="ltr"><span><font color="#888888"><span style="color:rgb(0,0,0)"><b><i>David Gossage</i></b></span><font><i><span style="color:rgb(51,51,51)"><b><br>
</b></span></i></font></font></span><div><span><font color="#888888"><font><i><span style="color:rgb(51,51,51)"></span></i><font size="1"><b style="color:rgb(153,0,0)">Carousel Checks Inc.<span style="color:rgb(204,204,204)"> | System Administrator</span></b></font></font><font style="color:rgb(153,153,153)"><font size="1"><br>
</font></font><font><font size="1"><span style="color:rgb(51,51,51)"><b style="color:rgb(153,153,153)">Office</b><span style="color:rgb(153,153,153)"> <a value="+17086132426">708.613.2284<font color="#888888"><font size="1"><br></font></font></a></span></span></font></font></font></span></div></div></div></div>
<br><div class="gmail_quote">On Wed, Jul 20, 2016 at 1:29 AM, David Gossage <span dir="ltr"><<a href="mailto:dgossage@carouselchecks.com" target="_blank">dgossage@carouselchecks.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">So I have enabled shading on 3.7.11, moved all VM images off and on and everything seemed fine no issues. <div><br></div><div><div>Volume Name: GLUSTER1</div><div>Type: Replicate</div><div>Volume ID: 167b8e57-28c3-447a-95cc-8410cbdf3f7f</div><div>Status: Started</div><div>Number of Bricks: 1 x 3 = 3</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: ccgl1.gl.local:/gluster1/BRICK1/1</div><div>Brick2: ccgl2.gl.local:/gluster1/BRICK1/1</div><div>Brick3: ccgl4.gl.local:/gluster1/BRICK1/1</div><div>Options Reconfigured:</div><div>nfs.enable-ino32: off</div><div>nfs.addr-namelookup: off</div><div>nfs.disable: on</div><div>performance.strict-write-ordering: off</div><div>cluster.background-self-heal-count: 16</div><div>cluster.self-heal-window-size: 1024</div><div>server.allow-insecure: on</div><div>cluster.server-quorum-type: server</div><div>cluster.quorum-type: auto</div><div>network.remote-dio: enable</div><div>cluster.eager-lock: enable</div><div>performance.stat-prefetch: on</div><div>performance.io-cache: off</div><div>performance.read-ahead: off</div><div>performance.quick-read: off</div><div>storage.owner-gid: 36</div><div>storage.owner-uid: 36</div><div>performance.readdir-ahead: on</div><div>features.shard: on</div><div>features.shard-block-size: 64MB</div><div>diagnostics.brick-log-level: WARNING</div><div><br></div><div><br></div><div>I added a new server(ccgl4) to replace one that nic died. Probed, and moved brick then kicked off heal.</div><div><br></div><div>When I did so on the smaller volume that just hosts my hosted engine it completed so fast that engine didnt even hiccup.</div><div><br></div><div>When I did same procedure on larger volume all VM's paused. I'm guesing just too many shards healing at once maybe no biggie. I let heal go went to bed for few hours while it healed 600GB of shards.</div><div><br></div><div>This morning looks like I may have some split-brain heals going on. All VM's started just fine after I unpausd/re-started them as needed. So far they have been up services seem ok, nothign seems read only etc.</div><div><br></div><div>But I have shards that still seem healing after a bit.</div><div>split-brain currently shows 0 entries on all 3 nodes (1 example below)</div><div><div>[root@ccgl1 ~]# gluster volume heal GLUSTER1 info split-brain</div><div>Brick ccgl1.gl.local:/gluster1/BRICK1/1</div><div>Status: Connected</div><div>Number of entries in split-brain: 0</div><div><br></div><div>Brick ccgl2.gl.local:/gluster1/BRICK1/1</div><div>Status: Connected</div><div>Number of entries in split-brain: 0</div><div><br></div><div>Brick ccgl4.gl.local:/gluster1/BRICK1/1</div><div>Status: Connected</div><div>Number of entries in split-brain: 0</div><div><br></div><div><br></div><div>However for example I get different results running (and sorry for the copy/paste storm incoming)</div><div><br></div><div><div>[root@ccgl1 ~]# gluster volume heal GLUSTER1 info</div><div>Brick ccgl1.gl.local:/gluster1/BRICK1/1</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1</div><div>/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47</div><div>/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226</div><div>/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192</div><div>/.shard - Possibly undergoing heal</div><div><br></div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274</div><div>/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32</div><div>/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49</div><div>/.shard - Possibly undergoing heal</div><div><br></div><div>Status: Connected</div><div>Number of entries: 21</div><div><br></div><div>Brick ccgl2.gl.local:/gluster1/BRICK1/1</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274</div><div>/.shard - Possibly undergoing heal</div><div><br></div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47</div><div>/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47</div><div>/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226</div><div>/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48</div><div>/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218</div><div>/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95</div><div>/.shard - Possibly undergoing heal</div><div><br></div><div>Status: Connected</div><div>Number of entries: 21</div><div><br></div><div>Brick ccgl4.gl.local:/gluster1/BRICK1/1</div><div>/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49</div><div>/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47</div><div>/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209</div><div>/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217</div><div>/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32</div><div>Status: Connected</div><div>Number of entries: 19</div></div><div><br></div><div><div>[root@ccgl2 ~]# gluster volume heal GLUSTER1 info</div><div>Brick ccgl1.gl.local:/gluster1/BRICK1/1</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1</div><div>/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47</div><div>/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226</div><div>/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192</div><div>/.shard - Possibly undergoing heal</div><div><br></div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274</div><div>/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32</div><div>/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49</div><div>/.shard/1cec1185-ad25-414b-9ae3-c17b5ef0d064.3</div><div>/.shard - Possibly undergoing heal</div><div><br></div><div>Status: Connected</div><div>Number of entries: 22</div><div><br></div><div>Brick ccgl2.gl.local:/gluster1/BRICK1/1</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274</div><div>/.shard - Possibly undergoing heal</div><div><br></div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47</div><div>/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47</div><div>/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226</div><div>/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48</div><div>/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218</div><div>/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95</div><div>/.shard - Possibly undergoing heal</div><div><br></div><div>/.shard/900421ff-b10d-404a-bbe7-173af11a69dd.377</div><div>/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.248</div><div>Status: Connected</div><div>Number of entries: 23</div><div><br></div><div>Brick ccgl4.gl.local:/gluster1/BRICK1/1</div><div>/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49</div><div>/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47</div><div>/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209</div><div>/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217</div><div>/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.45</div><div>/.shard/996ba563-c7c6-4448-9d94-2dee6c90b8c3.200</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.47</div><div>/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.47</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.311</div><div>Status: Connected</div><div>Number of entries: 24</div></div><div><br></div><div><br></div><div><br></div><div><div>[root@ccgl4 ~]# gluster volume heal GLUSTER1 info</div><div>Brick ccgl1.gl.local:/gluster1/BRICK1/1</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1</div><div>/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47</div><div>/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226</div><div>/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192</div><div>/.shard - Possibly undergoing heal</div><div><br></div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274</div><div>/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32</div><div>/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.123</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.45</div><div>/.shard - Possibly undergoing heal</div><div><br></div><div>Status: Connected</div><div>Number of entries: 23</div><div><br></div><div>Brick ccgl2.gl.local:/gluster1/BRICK1/1</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274</div><div>/.shard - Possibly undergoing heal</div><div><br></div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47</div><div>/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47</div><div>/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226</div><div>/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48</div><div>/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218</div><div>/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95</div><div>/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.2</div><div>/.shard - Possibly undergoing heal</div><div><br></div><div>/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.248</div><div>Status: Connected</div><div>Number of entries: 23</div><div><br></div><div>Brick ccgl4.gl.local:/gluster1/BRICK1/1</div><div>/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49</div><div>/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47</div><div>/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209</div><div>/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47</div><div>/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48</div><div>/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217</div><div>/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226</div><div>/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.45</div><div>/.shard/996ba563-c7c6-4448-9d94-2dee6c90b8c3.200</div><div>/.shard/996ba563-c7c6-4448-9d94-2dee6c90b8c3.160</div><div>/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.46</div><div>Status: Connected</div><div>Number of entries: 23</div></div><div><br></div><div><br></div><div>Should I just let it keep going? </div><span><font color="#888888"><div><br></div><div><div data-smartmail="gmail_signature"><div dir="ltr"><span><font color="#888888"><span style="color:rgb(0,0,0)"><b><i>David Gossage</i></b></span><font><i><span style="color:rgb(51,51,51)"><b><br>
</b></span></i></font></font></span><div><span><font color="#888888"><font><i><span style="color:rgb(51,51,51)"></span></i><font size="1"><b style="color:rgb(153,0,0)">Carousel Checks Inc.<span style="color:rgb(204,204,204)"> | System Administrator</span></b></font></font><font style="color:rgb(153,153,153)"><font size="1"><br>
</font></font><font><font size="1"><span style="color:rgb(51,51,51)"><b style="color:rgb(153,153,153)">Office</b><span style="color:rgb(153,153,153)"> <a value="+17086132426">708.613.2284<font color="#888888"><font size="1"><br></font></font></a></span></span></font></font></font></span></div></div></div></div>
</font></span></div></div></div>
</blockquote></div><br></div>
</blockquote></div><br></div>
</blockquote></div><br></div></div>