<div>Hi Olav,</div>
<div><br class="Apple-interchange-newline" />I have a hunch that our problem was caused by improper unmounting of the gluster volume, and have since found that the proper order should be: kill all jobs using volume -> unmount volume on clients -> gluster volume stop -> stop gluster service (if necessary)</div>
<div> </div>
<div>In my case, I wrote a Python script to find duplicate files on the mounted volume, then delete the corresponding link files on the bricks (making sure to also delete files in the .glusterfs directory)</div>
<div> </div>
<div>However, your find command was also suggested to me and I think it's a simpler solution. I believe removing all link files (even ones that are not causing duplicates) is fine since the next file access gluster will do a lookup on all bricks and recreate any link files if necessary. Hopefully a gluster expert can chime in on this point as I'm not completely sure.</div>
<div> </div>
<div>Keep in mind your setup is somewhat different than mine as I have only 5 bricks with no replication.</div>
<div> </div>
<div>Regards,</div>
<div>Tom</div>
<div> </div>
<blockquote class="threadBlockQuote" style="border-left: 2px solid #C2C2C2; padding-left: 3px; margin-left: 4px;">--------- Original Message ---------
<div>Subject: Re: [Gluster-users] Hundreds of duplicate files<br />From: "Olav Peeters" <opeeters@gmail.com><br />Date: 2/18/15 10:52 am<br />To: gluster-users@gluster.org, tbenzvi@3vgeomatics.com<br /><br />
<div class="moz-cite-prefix">Hi all,<br /> I'm have this problem after upgrading from 3.5.3 to 3.6.2.<br /> At the moment I am still waiting for a heal to finish (on a 31TB volume with 42 bricks, replicated over three nodes).<br /> <br /> Tom,<br /> how did you remove the duplicates?<br /> with 42 bricks I will not be able to do this manually..<br /> Did a:<br /> find $brick_root -type f -size 0 -perm 1000 -exec /bin/rm {} \;<br /> work for you?<br /> <br /> Should this type of thing ideally not be checked and mended by a heal?<br /> <br /> Does anyone have an idea yet how this happens in the first place? Can it be connected to upgrading?<br /> <br /> Cheers,<br /> Olav<br />
<pre class="moz-signature"> </pre>
On 01/01/15 03:07, <a class="moz-txt-link-abbreviated" href="mailto:tbenzvi@3vgeomatics.com">tbenzvi@3vgeomatics.com</a> wrote:</div>
<blockquote cite="mid:20141231190720.b2b02683b6fce9ed61e10e2e9bfae354.1100adfcd4.mailapi@email04.secureserver.net">
<div>No, the files can be read on a newly mounted client! I went ahead and deleted all of the link files associated with these duplicates, and then remounted the volume. The problem is fixed!</div>
<div>Thanks again for the help, Joe and Vijay.</div>
<div> </div>
<div>Tom</div>
<div> </div>
<blockquote class="threadBlockQuote" style="border-left: 2px solid
#C2C2C2; padding-left: 3px; margin-left: 4px;">--------- Original Message ---------
<div>Subject: Re: [Gluster-users] Hundreds of duplicate files<br /> From: "Vijay Bellur" <a class="moz-txt-link-rfc2396E" href="mailto:vbellur@redhat.com"><vbellur@redhat.com></a><br /> Date: 12/28/14 3:23 am<br /> To: <a class="moz-txt-link-abbreviated" href="mailto:tbenzvi@3vgeomatics.com">tbenzvi@3vgeomatics.com</a>, <a class="moz-txt-link-abbreviated" href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br /> <br /> On 12/28/2014 01:20 PM, <a class="moz-txt-link-abbreviated" href="mailto:tbenzvi@3vgeomatics.com">tbenzvi@3vgeomatics.com</a> wrote:<br /> > Hi Vijay,<br /> > Yes the files are still readable from the .glusterfs path.<br /> > There is no explicit error. However, trying to read a text file in<br /> > python simply gives me null characters:<br /> ><br /> > >>> open('ott_mf_itab').readlines()<br /> > ['\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00']<br /> ><br /> > And reading binary files does the same<br /> ><br /> <br /> Is this behavior seen with a freshly mounted client too?<br /> <br /> -Vijay<br /> <br /> > --------- Original Message ---------<br /> > Subject: Re: [Gluster-users] Hundreds of duplicate files<br /> > From: "Vijay Bellur" <a class="moz-txt-link-rfc2396E" href="mailto:vbellur@redhat.com"><vbellur@redhat.com></a><br /> > Date: 12/27/14 9:57 pm<br /> > To: <a class="moz-txt-link-abbreviated" href="mailto:tbenzvi@3vgeomatics.com">tbenzvi@3vgeomatics.com</a>, <a class="moz-txt-link-abbreviated" href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br /> ><br /> > On 12/28/2014 10:13 AM, <a class="moz-txt-link-abbreviated" href="mailto:tbenzvi@3vgeomatics.com">tbenzvi@3vgeomatics.com</a> wrote:<br /> > > Thanks Joe, I've read your blog post as well as your post<br /> > regarding the<br /> > > .glusterfs directory.<br /> > > I found some unneeded duplicate files which were not being read<br /> > > properly. I then deleted the link file from the brick. This always<br /> > > removes the duplicate file from the listing, but the file does not<br /> > > always become readable. If I also delete the associated file in the<br /> > > .glusterfs directory on that brick, then some more files become<br /> > > readable. However this solution still doesn't work for all files.<br /> > > I know the file on the brick is not corrupt as it can be read<br /> > directly<br /> > > from the brick directory.<br /> ><br /> > For files that are not readable from the client, can you check if the<br /> > file is readable from the .glusterfs/ path?<br /> ><br /> > What is the specific error that is seen while trying to read one such<br /> > file from the client?<br /> ><br /> > Thanks,<br /> > Vijay<br /> ><br /> ><br /> ><br /> > _______________________________________________<br /> > Gluster-users mailing list<br /> > <a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br /> > <a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><br /> ></div>
</blockquote>
<br /><fieldset class="mimeAttachmentHeader"></fieldset><br />
<pre>_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
</div>
</blockquote>