<html><body><div style="font-family: garamond,new york,times,serif; font-size: 12pt; color: #000000"><div>You should be able to find a file named group-virt.example under /etc/glusterfs/<br></div><div>Copy that as /var/lib/glusterd/virt.<br></div><div><br></div><div>Then execute `gluster volume set datastore1 group virt`.<br></div><div>Now with this configuration, could you try your test case and let me know whether the file corruption still exists?<br></div><div><br></div><div>-Krutika<br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Lindsay Mathieson" <lindsay.mathieson@gmail.com><br><b>To: </b>"Krutika Dhananjay" <kdhananj@redhat.com><br><b>Cc: </b>"gluster-users" <gluster-users@gluster.org><br><b>Sent: </b>Saturday, November 14, 2015 10:51:26 AM<br><b>Subject: </b>RE: [Gluster-users] File Corruption with shards - 100% reproducable<br><div><br></div><style><!--
@font-face
{font-family:Helvetica;
panose-1:2 11 5 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Garamond;
panose-1:2 2 4 4 3 3 1 1 8 3;}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Times New Roman",serif;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><div class="WordSection1"><p class="MsoNormal">gluster volume set datastore1 group virt</p><p class="MsoNormal">Unable to open file '/var/lib/glusterd/groups/virt'. Error: No such file or directory</p><p class="MsoNormal"> </p><p class="MsoNormal">Not sure I understand this one – couldn’t find any docs for it.</p><p class="MsoNormal"> </p><p class="MsoNormal">Sent from <a href="http://go.microsoft.com/fwlink/?LinkId=550986" target="_blank" data-mce-href="http://go.microsoft.com/fwlink/?LinkId=550986">Mail</a> for Windows 10</p><p> </p><p> </p><div style="mso-element:para-border-div;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm" data-mce-style="mso-element: para-border-div; border: none; border-top: solid #E1E1E1 1.0pt; padding: 3.0pt 0cm 0cm 0cm;"><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;border:none;padding:0cm" data-mce-style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; border: none; padding: 0cm;"><br><b>From: </b>Krutika Dhananjay<br><b>Sent: </b>Saturday, 14 November 2015 1:45 PM<br><b>To: </b>Lindsay Mathieson<br><b>Cc: </b>gluster-users<br><b>Subject: </b>Re: [Gluster-users] File Corruption with shards - 100% reproducable</p></div><p class="MsoNormal"> </p><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif" data-mce-style="font-size: 12.0pt; font-family: 'Times New Roman',serif;"> </span></p><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Garamond",serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Garamond',serif; color: black;">The logs are at /var/log/glusterfs/<hyphenated-path-to-the-mountpoint>.log</span><span style="font-size:12.0pt;font-family:"Garamond",serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Garamond',serif; color: black;"></span></p><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Garamond",serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Garamond',serif; color: black;"> </span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Garamond",serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Garamond',serif; color: black;">OK. So what do you observe when you set group virt to on?</span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Garamond",serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Garamond',serif; color: black;"> </span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Garamond",serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Garamond',serif; color: black;"># gluster volume set <VOL> group virt</span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Garamond",serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Garamond',serif; color: black;"> </span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Garamond",serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Garamond',serif; color: black;">-Krutika</span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Garamond",serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Garamond',serif; color: black;"> </span></p></div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Garamond",serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Garamond',serif; color: black;" lang="EN-GB"><img id="Horizontal_x0020_Line_x0020_1" src="cid:image001.png@01D11EF0.1EDF4CB0" data-mce-src="cid:image001.png@01D11EF0.1EDF4CB0" border="0" height="1" width="708"></span><span style="font-size:12.0pt;font-family:"Garamond",serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Garamond',serif; color: black;"></span></p><blockquote style="border:none;border-left:solid #1010FF 1.5pt;padding:0cm 0cm 0cm 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-bottom:5.0pt" data-mce-style="border: none; border-left: solid #1010FF 1.5pt; padding: 0cm 0cm 0cm 4.0pt; margin-left: 3.75pt; margin-top: 5.0pt; margin-bottom: 5.0pt;"><p class="MsoNormal"><b><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">From: </span></b><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">"Lindsay Mathieson" <lindsay.mathieson@gmail.com><br><b>To: </b>"Krutika Dhananjay" <kdhananj@redhat.com><br><b>Cc: </b>"gluster-users" <gluster-users@gluster.org><br><b>Sent: </b>Friday, November 13, 2015 11:57:15 AM<br><b>Subject: </b>Re: [Gluster-users] File Corruption with shards - 100% reproducable</span></p><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div><div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">On 12 November 2015 at 15:46, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank" data-mce-href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>> wrote:</span></p><blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm" data-mce-style="border: none; border-left: solid #CCCCCC 1.0pt; padding: 0cm 0cm 0cm 6.0pt; margin-left: 4.8pt; margin-right: 0cm;"><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">OK. What do the client logs say?</span></p></div></blockquote><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">Dumb question - Which logs are those? </span></p><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div><blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm" data-mce-style="border: none; border-left: solid #CCCCCC 1.0pt; padding: 0cm 0cm 0cm 6.0pt; margin-left: 4.8pt; margin-right: 0cm;"><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">Could you share the exact steps to recreate this, and I will try it locally on my setup?</span></p></div></blockquote><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">I'm running this on a 3 node proxmox cluster, which makes the vm creation & migration easy to test.</span></p><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">Steps:</span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">- Create 3 node gluster datastore using proxmox vm host nodes</span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"><br>- Add gluster datastore as a storage dvice to proxmox</span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> * qemu vms use the gfapi to access the datastore</span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> * proxmox also adds a fuse mount for easy acces</span></p><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">- create a VM on the gluster storage, QCOW2 format. I just created a simple debain Mate vm</span></p><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">- start the vm, open a console to it.</span></p><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">- live migrate the VM to a another node</span></p><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">- It will rapdily barf itself with disk errors</span></p><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">- stop the VM</span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">- qemu will show file corruption (many many errors)</span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> * qemu-img check <vm disk image><br> * qemu-img info <vm disk image></span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">Repeating the process with sharding off has no errors.</span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"><br> </span></p></div><blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm" data-mce-style="border: none; border-left: solid #CCCCCC 1.0pt; padding: 0cm 0cm 0cm 6.0pt; margin-left: 4.8pt; margin-right: 0cm;"><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">Also, want to see the output of 'gluster volume info'.</span></p></div></blockquote></div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"><br>I've trimmed settings down to a bare minimum. This is a test gluster cluster so I can do with it as I wish.</span></p><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Courier New";color:black" data-mce-style="font-size: 12.0pt; font-family: 'Courier New'; color: black;"><br>gluster volume info<br> <br>Volume Name: datastore1<br>Type: Replicate<br>Volume ID: 238fddd0-a88c-4edb-8ac5-ef87c58682bf<br>Status: Started<br>Number of Bricks: 1 x 3 = 3<br>Transport-type: tcp<br>Bricks:<br>Brick1: vnb.proxmox.softlog:/mnt/ext4<br>Brick2: vng.proxmox.softlog:/mnt/ext4<br>Brick3: vna.proxmox.softlog:/mnt/ext4<br>Options Reconfigured:<br>performance.strict-write-ordering: on<br>performance.readdir-ahead: off<br>cluster.quorum-type: auto<br>features.shard: on</span><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"></span></p><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"> </span></p></div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;"><br clear="all"><br>-- </span></p><div><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Helvetica',sans-serif; color: black;">Lindsay</span></p></div></div></div></blockquote><p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Garamond",serif;color:black" data-mce-style="font-size: 12.0pt; font-family: 'Garamond',serif; color: black;"> </span></p><p class="MsoNormal"> </p><p class="MsoNormal"> </p></div></blockquote><div><br></div></div></body></html>