<html><body><div style="font-family: garamond,new york,times,serif; font-size: 12pt; color: #000000"><div>Hi,<br></div><div><br></div><div>Could you provide the xattrs in hex format?</div><div><br></div><div>You can execute `getfattr -d -m . -e hex <path-to-the-directory/file-on-the-brick(s)>`</div><div><br></div><div>-Krutika</div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Alessandro Ipe" <Alessandro.Ipe@meteo.be><br><b>To: </b>"Krutika Dhananjay" <kdhananj@redhat.com><br><b>Cc: </b>gluster-users@gluster.org<br><b>Sent: </b>Thursday, March 12, 2015 5:15:08 PM<br><b>Subject: </b>Re: [Gluster-users] Input/output error when trying to access a file on client<br><div><br></div>
<style>
p, li { white-space: pre-wrap; }
</style>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Hi,</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Actually, my gluster volume is distribute-replicate so I should provide the attributes on all the bricks. Here they are:</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">1. # file: data/glusterfs/md1/brick1/root</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.afr.md1-client-0=0sAAAAAAAAAAAAAAAA</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.afr.md1-client-1=0sAAAAAAAAAAAAAAAA</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw==</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.glusterfs.dht=0sAAAAAQAAAACqqqqq/////w==</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">2. # file: data/glusterfs/md1/brick1/root</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.afr.md1-client-0=0sAAAAAAAAAAAAAAAA</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.afr.md1-client-1=0sAAAAAAAAAAAAAAAA</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw==</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.glusterfs.dht=0sAAAAAQAAAACqqqqq/////w==</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">3. # file: data/glusterfs/md1/brick1/root</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.afr.md1-client-2=0sAAAAAAAAAAAAAAAA</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.afr.md1-client-3=0sAAAAAAAAAAEAAAAA</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw==</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.glusterfs.dht=0sAAAAAQAAAAAAAAAAVVVVVA==</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">4. # file: data/glusterfs/md1/brick1/root</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.afr.md1-client-2=0sAAAAAAAAAAEAAAAA</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.afr.md1-client-3=0sAAAAAAAAAAAAAAAA</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw==</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.glusterfs.dht=0sAAAAAQAAAAAAAAAAVVVVVA==</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">5. # file: data/glusterfs/md1/brick1/root</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.afr.md1-client-4=0sAAAAAAAAAAAAAAAA</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.afr.md1-client-5=0sAAAAAAAAAAEAAAAA</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw==</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.glusterfs.dht=0sAAAAAQAAAABVVVVVqqqqqQ==</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">6. # file: data/glusterfs/md1/brick1/root</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.afr.md1-client-4=0sAAAAAAAAAAEAAAAA</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.afr.md1-client-5=0sAAAAAAAAAAAAAAAA</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw==</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.glusterfs.dht=0sAAAAAQAAAABVVVVVqqqqqQ==</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">so it seems in fact that there are discrepancies between 3-4 and 5-6 (replicate pairs).</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">A.</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">On Thursday 12 March 2015 11:33:00 Alessandro Ipe wrote:<br></p>
<p style=" margin-top:12px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Hi,</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">"gluster volume heal md1 info split-brain" returns approximatively 2000 files (already divided by 2</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"> due to replicate volume). So manually repairing each split-brain is unfeasable. Before scripting some</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"> procedure, I need to be sure that I will not harm further the gluster system.</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Moreover, I noticed that the messages printed in the logs are all about directories,</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"> e.g.</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">[2015-03-12 10:06:53.423856] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-md1-replicate-1: Unable to self-heal contents of '/root' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 1 ] [ 1 0 ] ]</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">[2015-03-12 10:06:53.424005] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-md1-replicate-2: Unable to self-heal contents of '/root' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 1 ] [ 1 0 ] ]</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">[2015-03-12 10:06:53.424110] E [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-md1-replicate-1: metadata self heal failed, on /root</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">[2015-03-12 10:06:53.424290] E [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-md1-replicate-2: metadata self heal failed, on /root</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Getting the attributes of that directory on each brick gives me for the first</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"># file: data/glusterfs/md1/brick1/root</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.afr.md1-client-0=0sAAAAAAAAAAAAAAAA</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.afr.md1-client-1=0sAAAAAAAAAAAAAAAA</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw==</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.glusterfs.dht=0sAAAAAQAAAACqqqqq/////w==</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">and for the second</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"># file: data/glusterfs/md1/brick1/root</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.afr.md1-client-0=0sAAAAAAAAAAAAAAAA</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.afr.md1-client-1=0sAAAAAAAAAAAAAAAA</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw==</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">trusted.glusterfs.dht=0sAAAAAQAAAACqqqqq/////w==</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">so it seems that there are both rigorously identical. However, according to your split -brain tutorial, </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"> none of them has 0x000000000000000000000000. What 0sAAAAAAAAAAAAAAAA means</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"> in fact ?</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Should I change both attributes on each directory to 0x000000000000000000000000 ?</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">Many thanks,</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">A.</p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">On Wednesday 11 March 2015 08:02:56 Krutika Dhananjay wrote:<br></p>
<p style=" margin-top:12px; margin-bottom:0px; margin-left:80px; margin-right:80px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'garamond,new york,times,serif'; color:#000000;">Hi,<br></span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:80px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'garamond,new york,times,serif'; color:#000000;"><br></span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:80px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'garamond,new york,times,serif'; color:#000000;">Have you gone through </span><a href="https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md" target="_blank"><span style=" font-family:'garamond,new york,times,serif'; text-decoration: underline; color:#006e28;">https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md</span></a><span style=" font-family:'garamond,new york,times,serif'; color:#000000;"> ?</span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:80px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'garamond,new york,times,serif'; color:#000000;">If not, could you go through that once and try the steps given there? Do let us know if something is not clear in the doc.</span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:80px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'garamond,new york,times,serif'; color:#000000;"><br></span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:80px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'garamond,new york,times,serif'; color:#000000;">-Krutika</span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:80px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'garamond,new york,times,serif'; color:#000000;"><br></span></p>
<hr>
<p style=" margin-top:12px; margin-bottom:12px; margin-left:85px; margin-right:120px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'Helvetica,Arial,sans-serif'; font-weight:600; color:#000000;">From: </span><span style=" font-family:'Helvetica,Arial,sans-serif'; color:#000000;">"Alessandro Ipe" <Alessandro.Ipe@meteo.be><br></span><span style=" font-family:'Helvetica,Arial,sans-serif'; font-weight:600; color:#000000;">To: </span><span style=" font-family:'Helvetica,Arial,sans-serif'; color:#000000;">gluster-users@gluster.org<br></span><span style=" font-family:'Helvetica,Arial,sans-serif'; font-weight:600; color:#000000;">Sent: </span><span style=" font-family:'Helvetica,Arial,sans-serif'; color:#000000;">Wednesday, March 11, 2015 4:54:09 PM<br></span><span style=" font-family:'Helvetica,Arial,sans-serif'; font-weight:600; color:#000000;">Subject: </span><span style=" font-family:'Helvetica,Arial,sans-serif'; color:#000000;">Re: [Gluster-users] Input/output error when trying to access a file on client<br></span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:85px; margin-right:120px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'Helvetica,Arial,sans-serif'; color:#000000;"><br></span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'Helvetica,Arial,sans-serif'; color:#000000;">Well, it is even worse. Now when doing a "ls -R" on the volume results in a lot of <br></span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:85px; margin-right:120px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'Helvetica,Arial,sans-serif'; color:#000000;"><br></span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'Helvetica,Arial,sans-serif'; color:#000000;">[2015-03-11 11:18:31.957505] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-md1-replicate-2: Unable to self-heal contents of '/library' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 2 ] [ 1 0 ] ]<br>[2015-03-11 11:18:31.957692] E [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status] 0-md1-replicate-2: metadata self heal failed, on /library<br></span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:85px; margin-right:120px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'Helvetica,Arial,sans-serif'; color:#000000;"><br></span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'Helvetica,Arial,sans-serif'; color:#000000;">I am desperate...<br></span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:85px; margin-right:120px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'Helvetica,Arial,sans-serif'; color:#000000;"><br></span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'Helvetica,Arial,sans-serif'; color:#000000;"><br>A.<br></span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:85px; margin-right:120px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'Helvetica,Arial,sans-serif'; color:#000000;"><br></span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'Helvetica,Arial,sans-serif'; color:#000000;"><br>On Wednesday 11 March 2015 12:05:33 you wrote:<br>> Hi,<br>> <br>> <br>> When trying to access a file on a gluster client (through fuse), I get an<br>> "Input/output error" message.<br>> <br>> Getting the attributes for the file gives me for the first brick<br>> # file: data/glusterfs/md1/brick1/kvm/hail/hail_home.qcow2<br>> trusted.afr.md1-client-2=0sAAAAAAAAAAAAAAAA<br>> trusted.afr.md1-client-3=0sAAABdAAAAAAAAAAA<br>> trusted.gfid=0sOCFPGCdrQ9uyq2yTTPCKqQ==<br>> <br>> while for the second (replicate) brick<br>> # file: data/glusterfs/md1/brick1/kvm/hail/hail_home.qcow2<br>> trusted.afr.md1-client-2=0sAAABJAAAAAAAAAAA<br>> trusted.afr.md1-client-3=0sAAAAAAAAAAAAAAAA<br>> trusted.gfid=0sOCFPGCdrQ9uyq2yTTPCKqQ==<br>> <br>> It seems that I have a split-brain. How can I solve this issue by resetting<br>> the attributes, please ?<br>> <br>> <br>> Thanks,<br>> <br>> <br>> Alessandro.<br>> <br>> ==================<br>> gluster volume info md1<br>> <br>> Volume Name: md1<br>> Type: Distributed-Replicate<br>> Volume ID: 6da4b915-1def-4df4-a41c-2f3300ebf16b<br>> Status: Started<br>> Number of Bricks: 3 x 2 = 6<br>> Transport-type: tcp<br>> Bricks:<br>> Brick1: tsunami1:/data/glusterfs/md1/brick1<br>> Brick2: tsunami2:/data/glusterfs/md1/brick1<br>> Brick3: tsunami3:/data/glusterfs/md1/brick1<br>> Brick4: tsunami4:/data/glusterfs/md1/brick1<br>> Brick5: tsunami5:/data/glusterfs/md1/brick1<br>> Brick6: tsunami6:/data/glusterfs/md1/brick1<br>> Options Reconfigured:<br>> server.allow-insecure: on<br>> cluster.read-hash-mode: 2<br>> features.quota: off<br>> performance.write-behind: on<br>> performance.write-behind-window-size: 4MB<br>> performance.flush-behind: off<br>> performance.io-thread-count: 64<br>> performance.cache-size: 512MB<br>> nfs.disable: on<br>> cluster.lookup-unhashed: off<br></span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:85px; margin-right:120px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'Helvetica,Arial,sans-serif'; color:#000000;"><br></span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'Helvetica,Arial,sans-serif'; color:#000000;">_______________________________________________<br>Gluster-users mailing list<br>Gluster-users@gluster.org<br>http://www.gluster.org/mailman/listinfo/gluster-users<br></span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:80px; margin-right:80px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><span style=" font-family:'garamond,new york,times,serif'; color:#000000;"><br></span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:40px; margin-right:40px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p><div><br></div><p></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"><br></p><div><br></div><p></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;">-- </p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"> Dr. Ir. Alessandro Ipe </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"> Department of Observations Tel. +32 2 373 06 31</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"> Remote Sensing from Space Fax. +32 2 374 67 88 </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"> Royal Meteorological Institute </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"> Avenue Circulaire 3 Email: </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"> B-1180 Brussels Belgium Alessandro.Ipe@meteo.be </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; -qt-user-state:0;"> Web: http://gerb.oma.be </p>
<p style="-qt-paragraph-type:empty; margin-top:0px; margin-bottom:0px; margin-left:0px; margin-right:0px; -qt-block-indent:0; text-indent:0px; "> </p></blockquote><div><br></div></div></body></html>