<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 5 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Garamond;
panose-1:2 2 4 4 3 3 1 1 8 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Times New Roman",serif;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=EN-AU link=blue vlink="#954F72"><div class=WordSection1><p class=MsoNormal>gluster volume set datastore1 group virt</p><p class=MsoNormal>Unable to open file '/var/lib/glusterd/groups/virt'. Error: No such file or directory</p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Not sure I understand this one – couldn’t find any docs for it.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Sent from <a href="http://go.microsoft.com/fwlink/?LinkId=550986">Mail</a> for Windows 10</p><p><o:p> </o:p></p><p><o:p> </o:p></p><div style='mso-element:para-border-div;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0cm'><br><b>From: </b>Krutika Dhananjay<br><b>Sent: </b>Saturday, 14 November 2015 1:45 PM<br><b>To: </b>Lindsay Mathieson<br><b>Cc: </b>gluster-users<br><b>Subject: </b>Re: [Gluster-users] File Corruption with shards - 100% reproducable</p></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman",serif'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Garamond",serif;color:black'>The logs are at /var/log/glusterfs/<hyphenated-path-to-the-mountpoint>.log</span><span style='font-size:12.0pt;font-family:"Garamond",serif;color:black'><o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Garamond",serif;color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Garamond",serif;color:black'>OK. So what do you observe when you set group virt to on?<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Garamond",serif;color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Garamond",serif;color:black'># gluster volume set <VOL> group virt<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Garamond",serif;color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Garamond",serif;color:black'>-Krutika<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Garamond",serif;color:black'><o:p> </o:p></span></p></div><p class=MsoNormal><span lang=EN-GB style='font-size:12.0pt;font-family:"Garamond",serif;color:black'><img border=0 width=708 height=1 id="Horizontal_x0020_Line_x0020_1" src="cid:image001.png@01D11EF0.1EDF4CB0"></span><span style='font-size:12.0pt;font-family:"Garamond",serif;color:black'><o:p></o:p></span></p><blockquote style='border:none;border-left:solid #1010FF 1.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal><b><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>From: </span></b><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>"Lindsay Mathieson" <lindsay.mathieson@gmail.com><br><b>To: </b>"Krutika Dhananjay" <kdhananj@redhat.com><br><b>Cc: </b>"gluster-users" <gluster-users@gluster.org><br><b>Sent: </b>Friday, November 13, 2015 11:57:15 AM<br><b>Subject: </b>Re: [Gluster-users] File Corruption with shards - 100% reproducable<o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div><div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>On 12 November 2015 at 15:46, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>> wrote:<o:p></o:p></span></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>OK. What do the client logs say?<o:p></o:p></span></p></div></blockquote><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>Dumb question - Which logs are those? <o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>Could you share the exact steps to recreate this, and I will try it locally on my setup?<o:p></o:p></span></p></div></blockquote><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>I'm running this on a 3 node proxmox cluster, which makes the vm creation & migration easy to test.<o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>Steps:<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>- Create 3 node gluster datastore using proxmox vm host nodes<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><br>- Add gluster datastore as a storage dvice to proxmox<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'> * qemu vms use the gfapi to access the datastore<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'> * proxmox also adds a fuse mount for easy acces<o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>- create a VM on the gluster storage, QCOW2 format. I just created a simple debain Mate vm<o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>- start the vm, open a console to it.<o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>- live migrate the VM to a another node<o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>- It will rapdily barf itself with disk errors<o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>- stop the VM<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>- qemu will show file corruption (many many errors)<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'> * qemu-img check <vm disk image><br> * qemu-img info <vm disk image><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>Repeating the process with sharding off has no errors.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><br> <o:p></o:p></span></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>Also, want to see the output of 'gluster volume info'.<o:p></o:p></span></p></div></blockquote></div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><br>I've trimmed settings down to a bare minimum. This is a test gluster cluster so I can do with it as I wish.<o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Courier New";color:black'><br>gluster volume info<br> <br>Volume Name: datastore1<br>Type: Replicate<br>Volume ID: 238fddd0-a88c-4edb-8ac5-ef87c58682bf<br>Status: Started<br>Number of Bricks: 1 x 3 = 3<br>Transport-type: tcp<br>Bricks:<br>Brick1: vnb.proxmox.softlog:/mnt/ext4<br>Brick2: vng.proxmox.softlog:/mnt/ext4<br>Brick3: vna.proxmox.softlog:/mnt/ext4<br>Options Reconfigured:<br>performance.strict-write-ordering: on<br>performance.readdir-ahead: off<br>cluster.quorum-type: auto<br>features.shard: on</span><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><o:p> </o:p></span></p></div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'><br clear=all><br>-- <o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black'>Lindsay<o:p></o:p></span></p></div></div></div></blockquote><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Garamond",serif;color:black'><o:p> </o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>