<div dir="ltr">Hi Glusterfs Experts,<div><br></div><div>We are testing glusterfs 3.7.0 tarball on our 10 Node glusterfs cluster. Each node has 36 dirves and please find the volume info below</div><div><br></div><div><div>Volume Name: vaulttest5</div><div>Type: Distributed-Disperse</div><div>Volume ID: 68e082a6-9819-4885-856c-1510cd201bd9</div><div>Status: Started</div><div>Number of Bricks: 36 x (8 + 2) = 360</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: 10.1.2.1:/media/disk1</div><div>Brick2: 10.1.2.2:/media/disk1</div><div>Brick3: 10.1.2.3:/media/disk1</div><div>Brick4: 10.1.2.4:/media/disk1</div><div>Brick5: 10.1.2.5:/media/disk1</div><div>Brick6: 10.1.2.6:/media/disk1</div><div>Brick7: 10.1.2.7:/media/disk1</div><div>Brick8: 10.1.2.8:/media/disk1</div><div>Brick9: 10.1.2.9:/media/disk1</div><div>Brick10: 10.1.2.10:/media/disk1</div><div>Brick11: 10.1.2.1:/media/disk2</div><div>Brick12: 10.1.2.2:/media/disk2</div><div>Brick13: 10.1.2.3:/media/disk2</div><div>Brick14: 10.1.2.4:/media/disk2</div><div>Brick15: 10.1.2.5:/media/disk2</div><div>Brick16: 10.1.2.6:/media/disk2</div><div>Brick17: 10.1.2.7:/media/disk2</div><div>Brick18: 10.1.2.8:/media/disk2</div><div>Brick19: 10.1.2.9:/media/disk2</div><div>Brick20: 10.1.2.10:/media/disk2</div><div>...</div><div>....</div><div><div>Brick351: 10.1.2.1:/media/disk36</div><div>Brick352: 10.1.2.2:/media/disk36</div><div>Brick353: 10.1.2.3:/media/disk36</div><div>Brick354: 10.1.2.4:/media/disk36</div><div>Brick355: 10.1.2.5:/media/disk36</div><div>Brick356: 10.1.2.6:/media/disk36</div><div>Brick357: 10.1.2.7:/media/disk36</div><div>Brick358: 10.1.2.8:/media/disk36</div><div>Brick359: 10.1.2.9:/media/disk36</div><div>Brick360: 10.1.2.10:/media/disk36</div><div>Options Reconfigured:</div><div>performance.readdir-ahead: on</div></div><div><br></div><div>We did some performance testing and simulated the proactive self healing for Erasure coding. Disperse volume has been created across nodes.</div><div><br></div><div><u><b>Description of problem</b></u></div><div><br></div><div>I disconnected the <b>network of two nodes</b> and tried to write some video files and <b>glusterfs</b> <b>wrote the video files on balance 8 nodes perfectly</b>. I tried to download the uploaded file and it was downloaded perfectly. Then i enabled the network of two nodes, the pro active self healing mechanism worked perfectly and wrote the unavailable junk of data to the recently enabled node from the other 8 nodes. But when i tried to download the same file node, it showed Input/Output error. I couldn't download the file. I think there is an issue in pro active self healing.</div><div><br></div><div>Also we tried the simulation with one node network failure. We faced same I/O error issue while downloading the file</div><div><br></div><div><br></div><div><u>Error while downloading file </u></div><div><u><br></u></div><div><p class="MsoNormal">root@master02:/home/admin# rsync -r
--progress /mnt/gluster/file13_AN ./1/file13_AN-2</p>
<p class="MsoNormal">sending incremental file list</p>
<p class="MsoNormal">file13_AN</p>
<p class="MsoNormal"> 3,342,355,597 100%
4.87MB/s 0:10:54 (xfr#1, to-chk=0/1)</p>
<p class="MsoNormal">rsync: read errors mapping "/mnt/gluster/file13_AN":
Input/output error (5)</p>
<p class="MsoNormal">WARNING: file13_AN failed verification -- update discarded
(will try again).</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> root@master02:/home/admin# cp
/mnt/gluster/file13_AN ./1/file13_AN-3</p><p class="MsoNormal"></p>
<p class="MsoNormal">cp: error reading ‘/mnt/gluster/file13_AN’: Input/output error</p>
<span style="font-size:11pt;font-family:Calibri,sans-serif">cp:
failed to extend ‘./1/file13_AN-3’: Input/output error</span><u><br></u></div><div><span style="font-size:11pt;font-family:Calibri,sans-serif"><br></span></div><div><br></div><div>We can't conclude the issue with glusterfs 3.7.0 or our glusterfs configuration.</div><div><br></div><div>Any help would be greatly appreciated</div><div><br></div>-- <br><div class="gmail_signature">Cheers</div><div class="gmail_signature">Backer<br><br></div>
</div></div>