<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Please truncate the client and brick logs then mount the fuse client
and demonstrate the error, then share the brick logs and the client
log up to that point.<br>
<br>
Also, share the extended attributes for both the dht linkfile and
the actual file.<br>
<br>
<div class="moz-cite-prefix">On 12/26/2014 06:34 PM,
<a class="moz-txt-link-abbreviated" href="mailto:tbenzvi@3vgeomatics.com">tbenzvi@3vgeomatics.com</a> wrote:<br>
</div>
<blockquote
cite="mid:20141226193452.b2b02683b6fce9ed61e10e2e9bfae354.c099383950.mailapi@email04.secureserver.net"
type="cite">
<div>Hello everyone and happy holidays,</div>
<div> </div>
<div>I upgraded both servers so that they are now both running
Gluster 3.5.3, in fact they are both running Fedora 20 with the
same kernel version. We have only one client and that is the
first server itself, with plans to change this in the future..</div>
<div>As per a previous suggestion, I also ran xfs_repair on each
of the five bricks, which reported no errors.</div>
<div> </div>
<div>So to recap: Doing a file listing on the mounted Gluster
volume shows the same filename appearing twice. Trying to access
the file either gives me the link file (and an error trying to
read it), or the file on the actual brick location (this is
entirely random, sometimes it will not work and then a few
seconds later trying to read the file again works).</div>
<div>Additionally, there are some cases in which two versions
(different content) with the same filename appear on two bricks
on the different servers.</div>
<div> </div>
<div>I would much appreciate it if someone could shed some light
on this issue.</div>
<div> </div>
<div>Best Regards,</div>
<div>Tom</div>
<div> </div>
<blockquote class="threadBlockQuote" style="border-left: 2px solid
#C2C2C2; padding-left: 3px; margin-left: 4px;">---------
Original Message ---------
<div>Subject: Re: [Gluster-users] Hundreds of duplicate files<br>
From: "Joe Julian" <a class="moz-txt-link-rfc2396E" href="mailto:joe@julianfamily.org"><joe@julianfamily.org></a><br>
Date: 12/21/14 10:34 pm<br>
To: <a class="moz-txt-link-abbreviated" href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
<br>
Have you tried upgrading the older server so all are running
the same version? Even though it's supposed to work with mixed
versions, the goal should always be to have everything running
the same version (clients and servers).<br>
<br>
<div class="moz-cite-prefix">On 12/20/2014 09:37 PM, <a
moz-do-not-send="true" class="moz-txt-link-abbreviated"
href="mailto:tbenzvi@3vgeomatics.com">tbenzvi@3vgeomatics.com</a>
wrote:</div>
<blockquote
cite="mid:20141220223744.b2b02683b6fce9ed61e10e2e9bfae354.e9aad02fe8.mailapi@email04.secureserver.net">
<div>Hi Joe,</div>
<div> </div>
<div>Thanks for the reply. That worked; I probably forgot to
do this as root last time. Yet, the files still show up
twice in a directory listing on the mounted volume. And it
seems to be random whether reading the file will succeed
or not. I've tried with several files and it sometimes
works and sometimes fails; I assume this depends on
whether it locates the actual file on the brick or the
link file. Let me know if you have any idea what's going
on.</div>
<div> </div>
<div>Output of the command:</div>
<div> </div>
<div>
<p>$ getfattr -m . -d -e hex
/data/glusterfs/safari/brick01/brick/rsc/tsx/montreal_smaller/sm_asc/stack/slc/20130210.slc.ras<br>
getfattr: Removing leading '/' from absolute path names<br>
# file:
data/glusterfs/safari/brick01/brick/rsc/tsx/montreal_smaller/sm_asc/stack/slc/20130210.slc.ras<br>
system.posix_acl_access=0x0200000001000600ffffffff04000600ffffffff10000600ffffffff20000400ffffffff<br>
trusted.SGI_ACL_FILE=0x0000000400000001ffffffff0006000000000004ffffffff0006000000000010ffffffff0006000000000020ffffffff00040000<br>
trusted.gfid=0x52c2aed77d09412d8bfd7ca70e87b196<br>
trusted.glusterfs.dht.linkto=0x7361666172692d636c69656e742d3200</p>
</div>
<div> </div>
<div>Cheers,</div>
<div>Tom</div>
<div> </div>
<blockquote class="threadBlockQuote" style="border-left: 2px
solid #C2C2C2; padding-left: 3px; margin-left: 4px;">---------
Original Message ---------
<div>Subject: Re: [Gluster-users] Hundreds of duplicate
files<br>
From: "Joe Julian" <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:joe@julianfamily.org"><joe@julianfamily.org></a><br>
Date: 12/20/14 8:53 pm<br>
To: <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br>
<br>
Try 'getfattr -m . -d -e hex' (dot instead of dash) and,
of course, do that as root.<br>
<br>
<div class="moz-cite-prefix">On 12/20/2014 06:02 PM, <a
moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:tbenzvi@3vgeomatics.com">tbenzvi@3vgeomatics.com</a>
wrote:</div>
<blockquote
cite="mid:20141220190235.b2b02683b6fce9ed61e10e2e9bfae354.9213a98af5.mailapi@email04.secureserver.net">
<div>Hi everyone,</div>
<div> </div>
<div>We have a distributed Gluster volume on five
bricks over two servers (first server running
gluster 3.4.2, second server running gluster 3.5.1,
both running Fedora 20)</div>
<div>Starting last week, doing a file listing on the
mounted volume shows many files with the same name
appearing twice (and they are listed with the same
inode). Doing a search for these files, I have found
290,000 of them!!</div>
<div> </div>
<div>If I do a listing of these files on the bricks
themselves, it looks like most are link files (du
will show the file on the first server as 0 bytes,
and the sticky bit set). The file is fine on the
second server. Unfortunately, running "getfattr -m -
-e hex -d" on the file shows NO gluster-related
attributes and I believe this is why both files
appear in the listing. The files cannot be read by
any programs as it is trying to read the link file.
I assume the metadata became corrupted. This is a
production server so we really need to know:</div>
<div> </div>
<div>1. How did this happen, and how can we prevent it
going forward? There was a server crash a week ago
and I believe that was the cause.</div>
<div>2. How can we heal the Gluster volume/bricks and
link files. If there is some straightforward way of
restoring the link file pointer I can write a script
to do it, obviously doing this manually will be
impossible.</div>
<div> </div>
<div>Thanks very much for any and all help - much
appreciated!</div>
<div> </div>
<div>Regards,</div>
<div>Tom</div>
<div> </div>
<div> </div>
<div>On Wed, Dec 17, 2014 at 4:07 AM, <a
moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:tbenzvi@3vgeomatics.com"><tbenzvi@3vgeomatics.com></a>
wrote:</div>
<div>
<div>> Hi everyone, we have noticed some
extremely odd behaviour with our<br>
> distributed Gluster volume where duplicate
files (same name, same or<br>
> different content) are being created and
stored on multiple bricks. The only<br>
> consistent clue is that one of the duplicate
files has the sticky bit set. I<br>
> am hoping someone will be able to shed some
light on why this is happening<br>
> and how we can restore the volume as there
appear to be hundreds of such<br>
> files. I will try to provide as much
pertinent information as I can.<br>
><br>
> We have a 130TB Gluster volume consisting of
two 20TB bricks on server1, and<br>
> three 40TB bricks on a server2 which were
added at a later date (and<br>
> rebalancing was done). The volume is mounted
on server1, and accessed only<br>
> through this server but by many users. Both
servers went down due to power<br>
> loss several days ago after which this
problem was first noticed. We ran a<br>
> rebalance command on the volumes, this has
not fixed the problem.<br>
><br>
><br>
> Gluster volume info:<br>
> Volume Name: safari<br>
> Type: Distribute<br>
> Volume ID:
d48d0e6b-4389-4c2c-8fd1-cd2854121eda<br>
> Status: Started<br>
> Number of Bricks: 5<br>
> Transport-type: tcp<br>
> Bricks:<br>
> Brick1:
server1:/data/glusterfs/safari/brick00/brick<br>
> Brick2:
server1:/data/glusterfs/safari/brick01/brick<br>
> Brick3:
server2:/data/glusterfs/safari/brick02/brick<br>
> Brick4:
server2:/data/glusterfs/safari/brick03/brick<br>
> Brick5:
server2:/data/glusterfs/safari/brick04/brick<br>
><br>
><br>
> Size information:<br>
> /dev/sdc 37T 16T 22T 42%
/data/glusterfs/safari/brick02<br>
> /dev/sdd 37T 16T 22T 42%
/data/glusterfs/safari/brick03<br>
> /dev/sde 37T 17T 21T 45%
/data/glusterfs/safari/brick04<br>
> /dev/md126 11T 7.7T 2.8T 74%
/data/glusterfs/safari/brick00<br>
> /dev/md124 11T 8.0T 2.5T 77%
/data/glusterfs/safari/brick01<br>
> server2:/safari 130T 63T 68T 48% /sar<br>
><br>
><br>
> Example 1:<br>
> -Two files with the same name exist in one
directory<br>
> -They have different contents and attributes<br>
> -A file listing on the mounted volume shows
the same inode<br>
> -The newer file has sticky bit set<br>
> -Neither file is corrupted, they can both be
viewed by using the absolute<br>
> path (on the bricks)<br>
><br>
> File listing on the mounted volume<br>
> 13036730497538635177 -rw-rw-r-T 1 jon users
924 Dec 15 10:42 RSLC_tab<br>
> 13036730497538635177 -rw-rw-r-- 1 jon users
418 Mar 18 2013 RSLC_tab<br>
><br>
> Listing of the files on the bricks:<br>
> 8925798411 -rw-rw-r-T+ 2 jon users 924 Dec 15
10:42<br>
>
/data/glusterfs/safari/brick00/brick/complete/shm/rs2/ottawa/mf6_asc/stack_org/RSLC_tab<br>
> 51541886672 -rw-rw-r--+ 2 1002 users 418 Mar
18 2013<br>
>
/data/glusterfs/safari/brick02/brick/complete/shm/rs2/ottawa/mf6_asc/stack_org/RSLC_tab<br>
><br>
><br>
> Example 2:<br>
> -Two files with the same name exist in one
directory<br>
> -They have the same content and attributes<br>
> -No sticky bit is set when looking at file
listing on the mounted volume<br>
> -Sticky bit is set for one while when looking
at file listing on the bricks<br>
> -Files are corrupted<br>
><br>
> File listing on the mounted volume:<br>
> 13012555852904096080 -rw-rw-r-- 1 tom users
2393848 Dec 8 2013<br>
> ifg_lr/20130226_20130813.diff.phi.ras<br>
> 13012555852904096080 -rw-rw-r-- 1 tom users
2393848 Dec 8 2013<br>
> ifg_lr/20130226_20130813.diff.phi.ras<br>
><br>
> Listing of the files on the bricks:<br>
> 17058578 -rw-rw-r-T+ 2 tom users 2393848 Dec
13 17:11<br>
>
/data/glusterfs/safari/brick00/brick/rsc/rs2/calgary/u22_dsc/stack_org/ifg_lr/20130226_20130813.diff.phi.ras<br>
> 57986922129 -rw-rw-r--+ 2 1010 users 2393848
Dec 8 2013<br>
>
/data/glusterfs/safari/brick02/brick/rsc/rs2/calgary/u22_dsc/stack_org/ifg_lr/20130226_20130813.diff.phi.ras<br>
><br>
><br>
> Additionally, only some files in this
directory are duplicated. The<br>
> duplicated files are corrupted (can not be
viewed as Raster images: the<br>
> original file type)<br>
> The files which are not duplicated are not
corrupted.<br>
><br>
> File command: (notice duplicate and singleton
files)<br>
> ifg_lr/20091021_20100218.diff.phi.ras: Sun
raster image data, 1208 x 1981,<br>
> 8-bit, RGB colormap<br>
> ifg_lr/20091021_20101016.diff.phi.ras: data<br>
> ifg_lr/20091021_20101016.diff.phi.ras: data<br>
> ifg_lr/20091021_20101109.diff.phi.ras: Sun
raster image data, 1208 x 1981,<br>
> 8-bit, RGB colormap<br>
> ifg_lr/20091021_20101203.diff.phi.ras: Sun
raster image data, 1208 x 1981,<br>
> 8-bit, RGB colormap<br>
> ifg_lr/20091021_20101227.diff.phi.ras: Sun
raster image data, 1208 x 1981,<br>
> 8-bit, RGB colormap<br>
> ifg_lr/20091021_20110120.diff.phi.ras: Sun
raster image data, 1208 x 1981,<br>
> 8-bit, RGB colormap<br>
> ifg_lr/20091021_20110213.diff.phi.ras: data<br>
> ifg_lr/20091021_20110213.diff.phi.ras: data<br>
> ifg_lr/20091021_20110309.diff.phi.ras: data<br>
> ifg_lr/20091021_20110309.diff.phi.ras: sticky
data<br>
> ifg_lr/20091021_20110402.diff.phi.ras: Sun
raster image data, 1208 x 1981,<br>
> 8-bit, RGB colormap</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
_______________________________________________
Gluster-users mailing list <a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></div>
</blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre>_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
_______________________________________________ Gluster-users
mailing list <a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></div>
</blockquote>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>