<div dir="ltr">Hi All,<div><br></div><div>I tried many things to troubleshoot my issues; recreating volumes with different configurations, using different installation medium for the OS, re-installing the gluster environment several times.</div><div>My issues were resolved by reformatting my /opt partition from XFS to EXT4, then recreating the gluster volumes with EXT4-backed bricks rather than XFS-backed bricks.</div><div><br></div><div>Is there a reason to believe that XFS is an improper file system to be used for gluster bricks?</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 6, 2015 at 9:20 AM, Cobin Bluth <span dir="ltr"><<a href="mailto:cbluth@gmail.com" target="_blank">cbluth@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I appreciate your response, Joe.<div><br></div><div>Even before asking for help in IRC and this mailing list, I had googled and come across information about healing and split-brain, but I had not encountered any information that had helped.</div><div><br></div><div>I tried "gluster volume heal DSR info" and this is the output I received:</div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><font face="monospace, monospace"><br></font></div><div><div><font face="monospace, monospace">[root@glusterfs1 ~]# gluster volume heal DSR info</font></div></div><div><div><font face="monospace, monospace">Brick glusterfs1:/opt/dsr/</font></div></div><div><div><font face="monospace, monospace">Number of entries: 0</font></div></div><div><div><font face="monospace, monospace"><br></font></div></div><div><div><font face="monospace, monospace">Brick glusterfs2:/opt/dsr/</font></div></div><div><div><font face="monospace, monospace">Number of entries: 0</font></div></div><div><div><font face="monospace, monospace"><br></font></div></div><div><div><font face="monospace, monospace">Brick glusterfs3:/opt/dsr/</font></div></div><div><div><font face="monospace, monospace">Number of entries: 0</font></div></div><div><div><font face="monospace, monospace"><br></font></div></div><div><div><font face="monospace, monospace">Brick glusterfs4:/opt/dsr/</font></div></div><div><div><font face="monospace, monospace">Number of entries: 0</font></div></div><div><div><font face="monospace, monospace"><br></font></div></div><div><div><font face="monospace, monospace">Brick glusterfs5:/opt/dsr/</font></div></div><div><div><font face="monospace, monospace">Number of entries: 0</font></div></div><div><div><font face="monospace, monospace"><br></font></div></div><div><div><font face="monospace, monospace">Brick glusterfs6:/opt/dsr/</font></div></div><div><div><font face="monospace, monospace">Number of entries: 0</font></div></div><div><div><font face="monospace, monospace"><br></font></div></div><div><div><font face="monospace, monospace">Brick glusterfs7:/opt/dsr/</font></div></div><div><div><font face="monospace, monospace">Number of entries: 0</font></div></div><div><div><font face="monospace, monospace"><br></font></div></div><div><div><font face="monospace, monospace">Brick glusterfs8:/opt/dsr/</font></div></div><div><div><font face="monospace, monospace">Number of entries: 0</font></div></div><div><div><font face="monospace, monospace"><br></font></div></div><div><div><font face="monospace, monospace">[root@glusterfs1 ~]# gluster volume heal DSR info split-brain</font></div></div><div><div><font face="monospace, monospace">Brick glusterfs1:/opt/dsr/</font></div></div><div><div><font face="monospace, monospace">Number of entries in split-brain: 0</font></div></div><div><div><font face="monospace, monospace"><br></font></div></div><div><div><font face="monospace, monospace">Brick glusterfs2:/opt/dsr/</font></div></div><div><div><font face="monospace, monospace">Number of entries in split-brain: 0</font></div></div><div><div><font face="monospace, monospace"><br></font></div></div><div><div><font face="monospace, monospace">Brick glusterfs3:/opt/dsr/</font></div></div><div><div><font face="monospace, monospace">Number of entries in split-brain: 0</font></div></div><div><div><font face="monospace, monospace"><br></font></div></div><div><div><font face="monospace, monospace">Brick glusterfs4:/opt/dsr/</font></div></div><div><div><font face="monospace, monospace">Number of entries in split-brain: 0</font></div></div><div><div><font face="monospace, monospace"><br></font></div></div><div><div><font face="monospace, monospace">Brick glusterfs5:/opt/dsr/</font></div></div><div><div><font face="monospace, monospace">Number of entries in split-brain: 0</font></div></div><div><div><font face="monospace, monospace"><br></font></div></div><div><div><font face="monospace, monospace">Brick glusterfs6:/opt/dsr/</font></div></div><div><div><font face="monospace, monospace">Number of entries in split-brain: 0</font></div></div><div><div><font face="monospace, monospace"><br></font></div></div><div><div><font face="monospace, monospace">Brick glusterfs7:/opt/dsr/</font></div></div><div><div><font face="monospace, monospace">Number of entries in split-brain: 0</font></div></div><div><div><font face="monospace, monospace"><br></font></div></div><div><div><font face="monospace, monospace">Brick glusterfs8:/opt/dsr/</font></div></div><div><div><font face="monospace, monospace">Number of entries in split-brain: 0</font></div></div><div><div><font face="monospace, monospace"><br></font></div></div><div><div><font face="monospace, monospace">[root@glusterfs1 ~]# </font></div></div></blockquote><div><br></div><div><br></div><div>If it helps, I am using CentOS7 and the latest repo found here: <a rel="nofollow" href="http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo" style="color:rgb(102,17,204);font-family:'Roboto Slab','Times New Roman',serif;font-size:14px;line-height:19px;white-space:pre-wrap;background-color:rgb(250,250,250)" target="_blank">http://download.gluster.org/pub/gluster/glusterfs/LATEST/CentOS/glusterfs-epel.repo</a></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font face="monospace, monospace"><br></font></div><div><div><font face="monospace, monospace">[root@glusterfs1 ~]# cat /etc/centos-release</font></div></div><div><div><font face="monospace, monospace">CentOS Linux release 7.1.1503 (Core) </font></div></div><div><div><font face="monospace, monospace">[root@glusterfs1 ~]# yum repolist</font></div></div><div><div><font face="monospace, monospace">Loaded plugins: fastestmirror</font></div></div><div><div><font face="monospace, monospace">Determining fastest mirrors</font></div></div><div><div><font face="monospace, monospace"> * base: <a href="http://lug.mtu.edu" target="_blank">lug.mtu.edu</a></font></div></div><div><div><font face="monospace, monospace"> * epel: <a href="http://mirrors.cat.pdx.edu" target="_blank">mirrors.cat.pdx.edu</a></font></div></div><div><div><font face="monospace, monospace"> * extras: <a href="http://centos.mirrors.tds.net" target="_blank">centos.mirrors.tds.net</a></font></div></div><div><div><font face="monospace, monospace"> * updates: <a href="http://mirrors.cat.pdx.edu" target="_blank">mirrors.cat.pdx.edu</a></font></div></div><div><div><font face="monospace, monospace">repo id repo name status</font></div></div><div><div><font face="monospace, monospace">!base/7/x86_64 CentOS-7 - Base 8,652</font></div></div><div><div><font face="monospace, monospace">!epel/x86_64 Extra Packages for Enterprise Linux 7 - x86_64 8,524</font></div></div><div><div><font face="monospace, monospace">!extras/7/x86_64 CentOS-7 - Extras 214</font></div></div><div><div><font face="monospace, monospace">!glusterfs-epel/7/x86_64 GlusterFS is a clustered file-system capable of scaling to several petabytes. 14</font></div></div><div><div><font face="monospace, monospace">!glusterfs-noarch-epel/7 GlusterFS is a clustered file-system capable of scaling to several petabytes. 2</font></div></div><div><div><font face="monospace, monospace">!updates/7/x86_64 CentOS-7 - Updates 1,486</font></div></div><div><div><font face="monospace, monospace">repolist: 18,892</font></div></div><div><div><font face="monospace, monospace">[root@glusterfs1 ~]# yum list installed | grep -i gluster</font></div></div><div><div><font face="monospace, monospace">glusterfs.x86_64 3.7.4-2.el7 @glusterfs-epel</font></div></div><div><div><font face="monospace, monospace">glusterfs-api.x86_64 3.7.4-2.el7 @glusterfs-epel</font></div></div><div><div><font face="monospace, monospace">glusterfs-cli.x86_64 3.7.4-2.el7 @glusterfs-epel</font></div></div><div><div><font face="monospace, monospace">glusterfs-client-xlators.x86_64 3.7.4-2.el7 @glusterfs-epel</font></div></div><div><div><font face="monospace, monospace">glusterfs-fuse.x86_64 3.7.4-2.el7 @glusterfs-epel</font></div></div><div><div><font face="monospace, monospace">glusterfs-libs.x86_64 3.7.4-2.el7 @glusterfs-epel</font></div></div><div><div><font face="monospace, monospace">glusterfs-rdma.x86_64 3.7.4-2.el7 @glusterfs-epel</font></div></div><div><div><font face="monospace, monospace">glusterfs-server.x86_64 3.7.4-2.el7 @glusterfs-epel</font></div></div><div><div><font face="monospace, monospace">samba-vfs-glusterfs.x86_64 4.1.12-23.el7_1 @updates </font></div></div><div><div><font face="monospace, monospace">[root@glusterfs1 ~]# </font></div></div></blockquote><div><br></div><div><br></div><div>And I am still experiencing the same error. </div><div><br></div><div>I am not largely familiar with troubleshooting this, is there something that I am missing?</div><div>I have set up another cluster in the same fashion, and I experienced the issue again.</div><div>I appreciate the information regarding the split-brain, but for me it doesnt seem to be the case, could there be another culprit?</div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 5, 2015 at 7:44 PM, Joe Julian <span dir="ltr"><<a href="mailto:joe@julianfamily.org" target="_blank">joe@julianfamily.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
[19:33] <JoeJulian> AceFacee check "gluster volume heal DSR
info"<br>
[19:33] <JoeJulian> it sounds like split-brain.<br>
[19:34] <JoeJulian> @split-brain<br>
[19:34] <glusterbot> JoeJulian: To heal split-brains, see
<a href="https://gluster.readthedocs.org/en/release-3.7.0/Features/heal-info-and-split-brain-resolution/" target="_blank">https://gluster.readthedocs.org/en/release-3.7.0/Features/heal-info-and-split-brain-resolution/</a>
For additional information, see this older article
<a href="https://joejulian.name/blog/fixing-split-brain-with-glusterfs-33/" target="_blank">https://joejulian.name/blog/fixing-split-brain-with-glusterfs-33/</a>
Also see splitmount
<a href="https://joejulian.name/blog/glusterfs-split-brain-recovery-made-easy/" target="_blank">https://joejulian.name/blog/glusterfs-split-brain-recovery-made-easy/</a><span><br>
<br>
<div>On 10/05/2015 05:43 PM, Cobin Bluth
wrote:<br>
</div>
</span><blockquote type="cite"><div><div>
<div dir="ltr">
<div>Please see the following:</div>
<blockquote style="margin:0 0 0 40px;border:none;padding:0px">
<div><font face="monospace, monospace"><br>
</font></div>
<div>
<div><font face="monospace, monospace">root@Asus:/mnt/GlusterFS-POC#
mount</font></div>
</div>
<div>
<div><font face="monospace, monospace">[ ...truncated... ]</font></div>
</div>
<div>
<div><font face="monospace, monospace">GlusterFS1:DSR on
/mnt/GlusterFS-POC type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)</font></div>
</div>
<div>
<div><font face="monospace, monospace">root@Asus:/mnt/GlusterFS-POC#
pwd</font></div>
</div>
<div>
<div><font face="monospace, monospace">/mnt/GlusterFS-POC</font></div>
</div>
<div>
<div><font face="monospace, monospace">root@Asus:/mnt/GlusterFS-POC#
for i in {1..10}; do cat /mnt/10MB-File > $i.tmp;
done</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">root@Asus:/mnt/GlusterFS-POC#
cat /mnt/10MB-File > test-file</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">cat: write error:
Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">root@Asus:/mnt/GlusterFS-POC#
dd if=/dev/zero of=test-file bs=1M count=10</font></div>
</div>
<div>
<div><font face="monospace, monospace">dd: error writing
‘test-file’: Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">dd: closing output
file ‘test-file’: Input/output error</font></div>
</div>
<div>
<div><font face="monospace, monospace">root@Asus:/mnt/GlusterFS-POC#
ls -lsha</font></div>
</div>
<div>
<div><font face="monospace, monospace">total 444K</font></div>
</div>
<div>
<div><font face="monospace, monospace">9.0K drwxr-xr-x 4
root root 4.1K Oct 5 17:32 .</font></div>
</div>
<div>
<div><font face="monospace, monospace">4.0K drwxr-xr-x 10
root root 4.0K Oct 5 17:16 ..</font></div>
</div>
<div>
<div><font face="monospace, monospace"> 13K -rw-r--r-- 1
root root 257K Oct 5 17:30 10.tmp</font></div>
</div>
<div>
<div><font face="monospace, monospace">1.0K -rw-r--r-- 1
root root 129K Oct 5 17:30 1.tmp</font></div>
</div>
<div>
<div><font face="monospace, monospace"> 13K -rw-r--r-- 1
root root 257K Oct 5 17:30 2.tmp</font></div>
</div>
<div>
<div><font face="monospace, monospace">1.0K -rw-r--r-- 1
root root 129K Oct 5 17:30 3.tmp</font></div>
</div>
<div>
<div><font face="monospace, monospace"> 13K -rw-r--r-- 1
root root 257K Oct 5 17:30 4.tmp</font></div>
</div>
<div>
<div><font face="monospace, monospace"> 13K -rw-r--r-- 1
root root 257K Oct 5 17:30 5.tmp</font></div>
</div>
<div>
<div><font face="monospace, monospace"> 13K -rw-r--r-- 1
root root 257K Oct 5 17:30 6.tmp</font></div>
</div>
<div>
<div><font face="monospace, monospace"> 13K -rw-r--r-- 1
root root 257K Oct 5 17:30 7.tmp</font></div>
</div>
<div>
<div><font face="monospace, monospace"> 13K -rw-r--r-- 1
root root 257K Oct 5 17:30 8.tmp</font></div>
</div>
<div>
<div><font face="monospace, monospace"> 13K -rw-r--r-- 1
root root 257K Oct 5 17:30 9.tmp</font></div>
</div>
<div>
<div><font face="monospace, monospace">329K -rw-r--r-- 1
root root 385K Oct 5 17:32 test-file</font></div>
</div>
<div>
<div><font face="monospace, monospace"> 0 drwxr-xr-x 3
root root 48 Oct 5 16:05 .trashcan</font></div>
</div>
<div>
<div><font face="monospace, monospace">root@Asus:/mnt/GlusterFS-POC#</font></div>
</div>
</blockquote>
<div><br>
</div>
<div><br>
</div>
<div>I am trying to do tests on my gluster volume to see how
well it will work for me.</div>
<div>I am getting that error when I try to use dd on it.</div>
<div><br>
</div>
<div>What would be the best way to troubleshoot this?</div>
<div><br>
</div>
<br clear="all">
<div>
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Thanks,</div>
<div><br>
</div>
<div>Cobin</div>
<div dir="ltr">
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
</div></div><span><pre>_______________________________________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</span></blockquote>
<br>
</div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>