<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-IN" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi All,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Glusterfs does self-healing when replace brick command is issued to change brick from one to another.
<o:p></o:p></p>
<p class="MsoNormal">But it also can happen that we can replace a brick which was attached earlier to the replica pair and have old data in it.
<o:p></o:p></p>
<p class="MsoNormal">There could be some modification of files in the new brick and then moved to old brick. The files which are modified
<o:p></o:p></p>
<p class="MsoNormal">new brick are not healed to old brick but newly added files are healed immediately.
<br>
<br>
Below example that I tried<o:p></o:p></p>
<p class="MsoNormal">Replica 3 pair of nodes as below: <o:p></o:p></p>
<p class="MsoNormal">Node 1: 10.132.10.50 (blr-7)<o:p></o:p></p>
<p class="MsoNormal">Node 2: 10.132.10.110 (blr-20)<o:p></o:p></p>
<p class="MsoNormal">Node 3: 10.132.10.56 (blr-12)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>Step 1: Issue brick replace to new node(blr-12)<br>
</b>[root@v3blr7 ~]# gluster volume replace-brick vs 10.132.10.110:/home/sameer/setup_2/blr-20 10.132.10.56:/home/sameer/setup_2/blr-12 commit force<o:p></o:p></p>
<p class="MsoNormal">volume replace-brick: success: replace-brick commit force operation successful<o:p></o:p></p>
<p class="MsoNormal">[root@v3blr7 ~]#<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>Step 2: healing started<o:p></o:p></b></p>
<p class="MsoNormal">[root@v3blr12 blr-12]# gluster volume heal vs info<o:p></o:p></p>
<p class="MsoNormal">Brick 10.132.10.50:/home/sameer/setup_2/blr-7<o:p></o:p></p>
<p class="MsoNormal">/heal_issue.0.0 - Possibly undergoing heal<o:p></o:p></p>
<p class="MsoNormal">/heal_issue.1.0<o:p></o:p></p>
<p class="MsoNormal">Number of entries: 2<o:p></o:p></p>
<p class="MsoNormal">Brick 10.132.10.56:/home/sameer/setup_2/blr-12<o:p></o:p></p>
<p class="MsoNormal">Number of entries: 0<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal"><b>After Healing is finished: <o:p></o:p></b></p>
<p class="MsoNormal">[root@v3blr12 blr-12]# [root@v3blr12 blr-12]# ls -lh<o:p></o:p></p>
<p class="MsoNormal">total 2.1G<o:p></o:p></p>
<p class="MsoNormal">-rw-r--r--. 2 root root 1.0G Dec 13 23:26 heal_issue.0.0<o:p></o:p></p>
<p class="MsoNormal">-rw-r--r--. 2 root root 1.0G Dec 13 23:26 heal_issue.1.0<o:p></o:p></p>
<p class="MsoNormal">[root@v3blr12 blr-12]#<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>Step 3: Run FIO to update the files and create two more files.
<o:p></o:p></b></p>
<p class="MsoNormal">[root@v3blr7 blr-7]# ls -lh<o:p></o:p></p>
<p class="MsoNormal">total 8.1G<o:p></o:p></p>
<p class="MsoNormal">-rw-r--r--. 2 root root 2.0G Dec 13 23:33 heal_issue.0.0<o:p></o:p></p>
<p class="MsoNormal">-rw-r--r--. 2 root root 2.0G Dec 13 23:33 heal_issue.1.0<o:p></o:p></p>
<p class="MsoNormal">-rw-r--r--. 2 root root 2.0G Dec 13 23:33 heal_issue.2.0<o:p></o:p></p>
<p class="MsoNormal">-rw-r--r--. 2 root root 2.0G Dec 13 23:33 heal_issue.3.0<o:p></o:p></p>
<p class="MsoNormal">[root@v3blr7 blr-7]#<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>Step 4: Issue replace brick to old brick(from blr-12 to blr-20)<o:p></o:p></b></p>
<p class="MsoNormal">[root@v3blr7 ~]# gluster volume replace-brick vs 10.132.10.56:/home/sameer/setup_2/blr-12 10.132.10.110:/home/sameer/setup_2/blr-20 commit force<o:p></o:p></p>
<p class="MsoNormal">volume replace-brick: success: replace-brick commit force operation successful<o:p></o:p></p>
<p class="MsoNormal"><b><o:p> </o:p></b></p>
<p class="MsoNormal"><b>Step 4: Healing finished to failed node<o:p></o:p></b></p>
<p class="MsoNormal">[root@v3blr20 blr-20]# gluster v heal vs info<o:p></o:p></p>
<p class="MsoNormal">Brick 10.132.10.50:/home/sameer/setup_2/blr-7<o:p></o:p></p>
<p class="MsoNormal">Number of entries: 0<o:p></o:p></p>
<p class="MsoNormal">Brick 10.132.10.110:/home/sameer/setup_2/blr-20<o:p></o:p></p>
<p class="MsoNormal">Number of entries: 0<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>Step 5: File size of failed nodes<o:p></o:p></b></p>
<p class="MsoNormal">[root@v3blr20 blr-20]# ls -lh<o:p></o:p></p>
<p class="MsoNormal">total 6.1G<o:p></o:p></p>
<p class="MsoNormal">-rw-r--r--. 2 root root 1.0G Dec 14 10:00 heal_issue.0.0 <=====
<b><span style="color:#C00000">File size is 1gb. did not heal</span></b><o:p></o:p></p>
<p class="MsoNormal">-rw-r--r--. 2 root root 1.0G Dec 14 10:00 heal_issue.1.0 <=====
<b><span style="color:#C00000">File size is 1gb. did not heal</span></b><o:p></o:p></p>
<p class="MsoNormal">-rw-r--r--. 2 root root 2.0G Dec 14 10:03 heal_issue.2.0<o:p></o:p></p>
<p class="MsoNormal">-rw-r--r--. 2 root root 2.0G Dec 14 10:03 heal_issue.3.0<o:p></o:p></p>
<p class="MsoNormal">[root@v3blr20 blr-20]#<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">The old files which are modified to other replica is not synced here and even it won’t show as split-brain too.
<o:p></o:p></p>
<p class="MsoNormal">How will users get informed that modified files are not synced? This seems an issue !!!!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Does glusterfs not allow to reuse(replace-brick) old brick which could have data? If not, there is need of
<o:p></o:p></p>
<p class="MsoNormal">healing data 2 times if there gets some issue in one brick and adding that fixing the brick. Seems this is not good idea to do.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Does anyone face this issue? Or this could be issue in glusterfs itself. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks and regards,<o:p></o:p></p>
<p class="MsoNormal">Samir Biswas<br>
<br>
<br>
<o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial",sans-serif"><strong>HGST E-mail
Confidentiality Notice & Disclaimer:</strong><br>This e-mail and any files transmitted with it may contain confidential
or legally privileged information of HGST and are intended solely for the use
of the individual or entity to which they are addressed. If you are not the
intended recipient, any disclosure, copying, distribution or any action taken
or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error,
please notify the sender immediately and delete the e-mail in its entirety from
your system.<o:p></o:p></span></p></body>
</html>