<div>Actually we are using XFS for the bricks. Still haven't made any progress on this issue, unfortunately..</div>
<div> </div>
<blockquote class="threadBlockQuote" style="border-left: 2px solid #C2C2C2; padding-left: 3px; margin-left: 4px;">--------- Original Message ---------
<div>Subject: Re: [Gluster-users] Hundreds of duplicate files<br />From: "Anders Blomdell" <anders.blomdell@control.lth.se><br />Date: 12/21/14 7:42 pm<br />To: tbenzvi@3vgeomatics.com, gluster-users@gluster.org<br /><br /><br /> <br /> On 21 December 2014 06:37:44 CET, tbenzvi@3vgeomatics.com wrote:<br /> >Hi Joe,<br /> > <br /> >Thanks for the reply. That worked; I probably forgot to do this as root<br /> >last time. Yet, the files still show up twice in a directory listing on<br /> >the mounted volume. And it seems to be random whether reading the file<br /> >will succeed or not. I've tried with several files and it sometimes<br /> >works and sometimes fails; I assume this depends on whether it locates<br /> >the actual file on the brick or the link file. Let me know if you have<br /> >any idea what's going on.<br /> Does the brick filesystem happen to be ext4? I havs hed the similar problem with 3.6.x and <br /> ext4 (64 bit offset problem). <br /> <br /> > <br /> >Output of the command:<br /> > <br /> >$ getfattr -m . -d -e hex<br /> >/data/glusterfs/safari/brick01/brick/rsc/tsx/montreal_smaller/sm_asc/stack/slc/20130210.slc.ras<br /> >getfattr: Removing leading '/' from absolute path names<br /> ># file:<br /> >data/glusterfs/safari/brick01/brick/rsc/tsx/montreal_smaller/sm_asc/stack/slc/20130210.slc.ras<br /> >system.posix_acl_access=0x0200000001000600ffffffff04000600ffffffff10000600ffffffff20000400ffffffff<br /> >trusted.SGI_ACL_FILE=0x0000000400000001ffffffff0006000000000004ffffffff0006000000000010ffffffff0006000000000020ffffffff00040000<br /> >trusted.gfid=0x52c2aed77d09412d8bfd7ca70e87b196<br /> >trusted.glusterfs.dht.linkto=0x7361666172692d636c69656e742d3200<br /> > <br /> > <br /> >Cheers,<br /> >Tom<br /> > <br /> >--------- Original Message --------- Subject: Re: [Gluster-users]<br /> >Hundreds of duplicate files<br /> >From: "Joe Julian" <joe@julianfamily.org><br /> >Date: 12/20/14 8:53 pm<br /> >To: gluster-users@gluster.org<br /> ><br /> >Try 'getfattr -m . -d -e hex' (dot instead of dash) and, of course, do<br /> >that as root.<br /> > <br /> > On 12/20/2014 06:02 PM, tbenzvi@3vgeomatics.com wrote:<br /> > Hi everyone,<br /> > <br /> >We have a distributed Gluster volume on five bricks over two servers<br /> >(first server running gluster 3.4.2, second server running gluster<br /> >3.5.1, both running Fedora 20)<br /> >Starting last week, doing a file listing on the mounted volume shows<br /> >many files with the same name appearing twice (and they are listed with<br /> >the same inode). Doing a search for these files, I have found 290,000<br /> >of them!!<br /> > <br /> >If I do a listing of these files on the bricks themselves, it looks<br /> >like most are link files (du will show the file on the first server as<br /> >0 bytes, and the sticky bit set). The file is fine on the second<br /> >server. Unfortunately, running "getfattr -m - -e hex -d" on the file<br /> >shows NO gluster-related attributes and I believe this is why both<br /> >files appear in the listing. The files cannot be read by any programs<br /> >as it is trying to read the link file. I assume the metadata became<br /> >corrupted. This is a production server so we really need to know:<br /> > <br /> >1. How did this happen, and how can we prevent it going forward? There<br /> >was a server crash a week ago and I believe that was the cause.<br /> >2. How can we heal the Gluster volume/bricks and link files. If there<br /> >is some straightforward way of restoring the link file pointer I can<br /> >write a script to do it, obviously doing this manually will be<br /> >impossible.<br /> > <br /> >Thanks very much for any and all help - much appreciated!<br /> > <br /> >Regards,<br /> >Tom<br /> > <br /> > <br /> >On Wed, Dec 17, 2014 at 4:07 AM, <tbenzvi@3vgeomatics.com> wrote:<br /> >> Hi everyone, we have noticed some extremely odd behaviour with our<br /> > > distributed Gluster volume where duplicate files (same name, same or<br /> >> different content) are being created and stored on multiple bricks.<br /> >The only<br /> >> consistent clue is that one of the duplicate files has the sticky bit<br /> >set. I<br /> >> am hoping someone will be able to shed some light on why this is<br /> >happening<br /> >> and how we can restore the volume as there appear to be hundreds of<br /> >such<br /> > > files. I will try to provide as much pertinent information as I can.<br /> > ><br /> >> We have a 130TB Gluster volume consisting of two 20TB bricks on<br /> >server1, and<br /> > > three 40TB bricks on a server2 which were added at a later date (and<br /> >> rebalancing was done). The volume is mounted on server1, and accessed<br /> >only<br /> >> through this server but by many users. Both servers went down due to<br /> >power<br /> >> loss several days ago after which this problem was first noticed. We<br /> >ran a<br /> > > rebalance command on the volumes, this has not fixed the problem.<br /> > ><br /> > ><br /> > > Gluster volume info:<br /> > > Volume Name: safari<br /> > > Type: Distribute<br /> > > Volume ID: d48d0e6b-4389-4c2c-8fd1-cd2854121eda<br /> > > Status: Started<br /> > > Number of Bricks: 5<br /> > > Transport-type: tcp<br /> > > Bricks:<br /> > > Brick1: server1:/data/glusterfs/safari/brick00/brick<br /> > > Brick2: server1:/data/glusterfs/safari/brick01/brick<br /> > > Brick3: server2:/data/glusterfs/safari/brick02/brick<br /> > > Brick4: server2:/data/glusterfs/safari/brick03/brick<br /> > > Brick5: server2:/data/glusterfs/safari/brick04/brick<br /> > ><br /> > ><br /> > > Size information:<br /> > > /dev/sdc 37T 16T 22T 42% /data/glusterfs/safari/brick02<br /> > > /dev/sdd 37T 16T 22T 42% /data/glusterfs/safari/brick03<br /> > > /dev/sde 37T 17T 21T 45% /data/glusterfs/safari/brick04<br /> > > /dev/md126 11T 7.7T 2.8T 74% /data/glusterfs/safari/brick00<br /> > > /dev/md124 11T 8.0T 2.5T 77% /data/glusterfs/safari/brick01<br /> > > server2:/safari 130T 63T 68T 48% /sar<br /> > ><br /> > ><br /> > > Example 1:<br /> > > -Two files with the same name exist in one directory<br /> > > -They have different contents and attributes<br /> > > -A file listing on the mounted volume shows the same inode<br /> > > -The newer file has sticky bit set<br /> >> -Neither file is corrupted, they can both be viewed by using the<br /> >absolute<br /> > > path (on the bricks)<br /> > ><br /> > > File listing on the mounted volume<br /> >> 13036730497538635177 -rw-rw-r-T 1 jon users 924 Dec 15 10:42 RSLC_tab<br /> > > 13036730497538635177 -rw-rw-r-- 1 jon users 418 Mar 18 2013 RSLC_tab<br /> > ><br /> > > Listing of the files on the bricks:<br /> > > 8925798411 -rw-rw-r-T+ 2 jon users 924 Dec 15 10:42<br /> >><br /> >/data/glusterfs/safari/brick00/brick/complete/shm/rs2/ottawa/mf6_asc/stack_org/RSLC_tab<br /> > > 51541886672 -rw-rw-r--+ 2 1002 users 418 Mar 18 2013<br /> >><br /> >/data/glusterfs/safari/brick02/brick/complete/shm/rs2/ottawa/mf6_asc/stack_org/RSLC_tab<br /> > ><br /> > ><br /> > > Example 2:<br /> > > -Two files with the same name exist in one directory<br /> > > -They have the same content and attributes<br /> >> -No sticky bit is set when looking at file listing on the mounted<br /> >volume<br /> >> -Sticky bit is set for one while when looking at file listing on the<br /> >bricks<br /> > > -Files are corrupted<br /> > ><br /> > > File listing on the mounted volume:<br /> > > 13012555852904096080 -rw-rw-r-- 1 tom users 2393848 Dec 8 2013<br /> > > ifg_lr/20130226_20130813.diff.phi.ras<br /> > > 13012555852904096080 -rw-rw-r-- 1 tom users 2393848 Dec 8 2013<br /> > > ifg_lr/20130226_20130813.diff.phi.ras<br /> > ><br /> > > Listing of the files on the bricks:<br /> > > 17058578 -rw-rw-r-T+ 2 tom users 2393848 Dec 13 17:11<br /> >><br /> >/data/glusterfs/safari/brick00/brick/rsc/rs2/calgary/u22_dsc/stack_org/ifg_lr/20130226_20130813.diff.phi.ras<br /> > > 57986922129 -rw-rw-r--+ 2 1010 users 2393848 Dec 8 2013<br /> >><br /> >/data/glusterfs/safari/brick02/brick/rsc/rs2/calgary/u22_dsc/stack_org/ifg_lr/20130226_20130813.diff.phi.ras<br /> > ><br /> > ><br /> > > Additionally, only some files in this directory are duplicated. The<br /> >> duplicated files are corrupted (can not be viewed as Raster images:<br /> >the<br /> > > original file type)<br /> > > The files which are not duplicated are not corrupted.<br /> > ><br /> > > File command: (notice duplicate and singleton files)<br /> >> ifg_lr/20091021_20100218.diff.phi.ras: Sun raster image data, 1208 x<br /> >1981,<br /> > > 8-bit, RGB colormap<br /> > > ifg_lr/20091021_20101016.diff.phi.ras: data<br /> > > ifg_lr/20091021_20101016.diff.phi.ras: data<br /> >> ifg_lr/20091021_20101109.diff.phi.ras: Sun raster image data, 1208 x<br /> >1981,<br /> > > 8-bit, RGB colormap<br /> >> ifg_lr/20091021_20101203.diff.phi.ras: Sun raster image data, 1208 x<br /> >1981,<br /> > > 8-bit, RGB colormap<br /> >> ifg_lr/20091021_20101227.diff.phi.ras: Sun raster image data, 1208 x<br /> >1981,<br /> > > 8-bit, RGB colormap<br /> >> ifg_lr/20091021_20110120.diff.phi.ras: Sun raster image data, 1208 x<br /> >1981,<br /> > > 8-bit, RGB colormap<br /> > > ifg_lr/20091021_20110213.diff.phi.ras: data<br /> > > ifg_lr/20091021_20110213.diff.phi.ras: data<br /> > > ifg_lr/20091021_20110309.diff.phi.ras: data<br /> > > ifg_lr/20091021_20110309.diff.phi.ras: sticky data<br /> >> ifg_lr/20091021_20110402.diff.phi.ras: Sun raster image data, 1208 x<br /> >1981,<br /> > > 8-bit, RGB colormap<br /> ><br /> ><br /> ><br /> > _______________________________________________<br /> > Gluster-users mailing list<br /> > Gluster-users@gluster.org<br /> > http://www.gluster.org/mailman/listinfo/gluster-users <br /> >_______________________________________________ Gluster-users mailing<br /> >list Gluster-users@gluster.org<br /> >http://www.gluster.org/mailman/listinfo/gluster-users<br /> ><br /> ><br /> >------------------------------------------------------------------------<br /> ><br /> >_______________________________________________<br /> >Gluster-users mailing list<br /> >Gluster-users@gluster.org<br /> >http://www.gluster.org/mailman/listinfo/gluster-users<br /> <br /> -- <br /> Sent from my Android device with K-9 Mail. Please excuse my brevity.</div>
</blockquote>