<div dir="ltr"><div>'Failed moves" are still a problem on our backup system. Another instance is attached with gfids if it's helpful. In this case, the rename after explicitly removing the target location was successful.<br><br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"> mv the files from bkp01 --> bkp00 : <span tabindex="0" class=""><span class="">18:41:02</span></span><br>
/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4' to `../bkp00/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4': File exists<br>
/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4' to `../bkp00/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4': File exists<br></blockquote><br>Source:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
# file: data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4<br>
trusted.bit-rot.version=0x0200000000000000571632ea00056a16<br>
trusted.gfid=0x2246740774424bb78f408d46c1f2a13e<br>
trusted.pgfid.583e43d6-27ff-4978-90a8-d7057385cf72=0x00000001<br></blockquote>
<br>Target:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
getfattr: Removing leading '/' from absolute path names <br># file: data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4<br>
trusted.gfid=0x556cd0eabffe4549a5ae43a188491db3<br>
trusted.glusterfs.dht.linkto=0x6766736261636b75702d636c69656e742d3100<br>
trusted.pgfid.9e21e545-b254-4d93-ba34-8dd204b41160=0x00000001<br></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
getfattr: /data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4: No such file or directory<br></blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
getfattr: /data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4: No such file or directory<br>
getfattr: /data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4: No such file or directory<br>
getfattr: /data/brick03bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4: No such file or directory<br>
getfattr: /data/brick04bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4: No such file or directory<br>
getfattr: /data/brick05bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4: No such file or directory<br>
<br>
# file: data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4<br>
trusted.bit-rot.version=0x0200000000000000569bb8d700074a42<br>
trusted.gfid=0x556cd0eabffe4549a5ae43a188491db3<br>
trusted.pgfid.9e21e545-b254-4d93-ba34-8dd204b41160=0x00000001<br>
<br>
getfattr: /data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4: No such file or directory<br>
getfattr: /data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4: No such file or directory<br>
getfattr: /data/brick03bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4: No such file or directory<br>
getfattr: /data/brick04bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4: No such file or directory<br>
getfattr: /data/brick05bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4: No such file or directory<br>
stat: cannot stat `"../bkp00/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4"': No such file or directory<br>
retry: renaming ./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4 -> ../bkp00/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4<br>Rename Succeeded! ./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4 -> ../bkp00/./homegfs/hpc_shared/motorsports/gmics/Raven/p9/85/data_collected4<br></blockquote><div><br><br></div><div>Thanks for any assistance,<br></div><div>Patrick <br></div>
<br></div><div>
</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 3, 2016 at 5:02 PM, Glomski, Patrick <span dir="ltr"><<a href="mailto:patrick.glomski@corvidtec.com" target="_blank">patrick.glomski@corvidtec.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Attaching a text file with the same content that is easier to read.<span class="HOEnZb"><font color="#888888"><br><br></font></span></div><span class="HOEnZb"><font color="#888888">Patrick<br></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 3, 2016 at 4:59 PM, Glomski, Patrick <span dir="ltr"><<a href="mailto:patrick.glomski@corvidtec.com" target="_blank">patrick.glomski@corvidtec.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Raghavendra,<br><br></div>Last night the backup had four of these errors and only one of the 'retried moves' succeeded. The only one to succeed in moving the file the second time had target files on a different gluster peer (gfs01bkp). Not sure if that is significant. <br><br>Note that I cannot stat the target file over the FUSE mount for any of these, but it exists on the bricks. Running an 'ls' on the directory containing the file (via FUSE) does not fix the issue. Source and target xattrs are appended for all bricks on all machines in the distributed volume.<br></div><div><br></div><div>Let me know if there's any other information it would be useful to gather, as this issue seems to recur frequently.<br></div><div><br></div><div>Thanks,<br></div>Patrick<br><div><br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font size="1"><span style="font-family:monospace,monospace"># Move failures<br><br>/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/056-1/data_collected3' to `../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3': File exists<br>/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/090-1/data_collected3' to `../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3': File exists<br>/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/057-2/data_collected3' to `../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3': File exists<br>/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/54/data_collected4' to `../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4': File exists<br><br>/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/056-1/data_collected3' to `../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3': File exists<br>/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/090-1/data_collected3' to `../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3': File exists<br>/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/057-2/data_collected3' to `../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3': File exists<br>/bin/mv: cannot move `./homegfs/hpc_shared/motorsports/54/data_collected4' to `../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4': File exists<br><br>========================================================================================================================<br>retry: renaming ./homegfs/hpc_shared/motorsports/056-1/data_collected3 -> ../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3<br><br>source xattrs<br> gfs01bkp<br> getfattr: /data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory<br> getfattr: /data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory<br> <br> gfs02bkp<br> getfattr: /data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory<br> # file: data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/056-1/data_collected3<br> trusted.bit-rot.version=0x0200000000000000570308980001d157<br> trusted.gfid=0xe07abd8ae861442ebc0df8b20719af30<br> trusted.pgfid.1776adb6-2925-49d3-9cca-8a04c29f4c05=0x00000001<br> <br> getfattr: Removing leading '/' from absolute path names<br> getfattr: /data/brick03bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory<br> getfattr: /data/brick04bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory<br> getfattr: /data/brick05bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory<br><br>target xattrs<br> gfs01bkp<br> getfattr: /data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory<br> getfattr: /data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory<br> <br> gfs02bkp<br> # file: data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3<br> trusted.bit-rot.version=0x0200000000000000569bb8d20003ed00<br> trusted.gfid=0xaefffbd0676649cd95eb6dfc874d7a59<br> trusted.pgfid.f7c5eff3-f474-433b-b10e-480f8353c6b9=0x00000001<br> <br> getfattr: Removing leading '/' from absolute path names<br> # file: data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3<br> trusted.gfid=0xaefffbd0676649cd95eb6dfc874d7a59<br> trusted.glusterfs.dht.linkto=0x6766736261636b75702d636c69656e742d3200<br> trusted.pgfid.f7c5eff3-f474-433b-b10e-480f8353c6b9=0x00000001<br> <br> getfattr: Removing leading '/' from absolute path names<br> getfattr: /data/brick03bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory<br> getfattr: /data/brick04bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory<br> getfattr: /data/brick05bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3: No such file or directory<br> <br>stat: cannot stat `"../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3"': No such file or directory<br> <br>Rename Failed! ./homegfs/hpc_shared/motorsports/056-1/data_collected3 -> ../bkp00/./homegfs/hpc_shared/motorsports/056-1/data_collected3<br> <br>========================================================================================================================<br>retry: renaming ./homegfs/hpc_shared/motorsports/090-1/data_collected3 -> ../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3<br> <br>source xattrs<br> gfs01bkp<br> getfattr: /data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory<br> getfattr: /data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory<br><br> gfs02bkp<br> getfattr: /data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory<br> # file: data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/090-1/data_collected3<br> trusted.bit-rot.version=0x0200000000000000570308980001d157<br> trusted.gfid=0x93663986a2d84632b0b358289458add2<br> trusted.pgfid.15ff8972-e668-42f0-9254-cf5b5679e372=0x00000001<br> <br> getfattr: Removing leading '/' from absolute path names<br> getfattr: /data/brick03bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory<br> getfattr: /data/brick04bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory<br> getfattr: /data/brick05bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory<br> <br>target xattrs<br> gfs01bkp<br> getfattr: /data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory<br> getfattr: /data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory<br><br> gfs02bkp<br> # file: data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3<br> trusted.bit-rot.version=0x0200000000000000569bb8d20003ed00<br> trusted.gfid=0x830250ea6c08472e9d71a032818d4f9b<br> trusted.pgfid.3254d2bf-69e0-4931-8c1b-d41cd0b2eff7=0x00000001<br> <br> getfattr: Removing leading '/' from absolute path names<br> # file: data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3<br> trusted.gfid=0x830250ea6c08472e9d71a032818d4f9b<br> trusted.glusterfs.dht.linkto=0x6766736261636b75702d636c69656e742d3200<br> trusted.pgfid.3254d2bf-69e0-4931-8c1b-d41cd0b2eff7=0x00000001<br> <br> getfattr: Removing leading '/' from absolute path names<br> getfattr: /data/brick03bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory<br> getfattr: /data/brick04bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory<br> getfattr: /data/brick05bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3: No such file or directory<br> <br>stat: cannot stat `"../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3"': No such file or directory<br> <br>Rename Failed! ./homegfs/hpc_shared/motorsports/090-1/data_collected3 -> ../bkp00/./homegfs/hpc_shared/motorsports/090-1/data_collected3<br>========================================================================================================================<br>retry: renaming ./homegfs/hpc_shared/motorsports/057-2/data_collected3 -> ../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3<br> <br>source xattrs<br> gfs01bkp<br> getfattr: /data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory<br> getfattr: /data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory<br> <br> gfs02bkp<br> getfattr: /data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory<br> # file: data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/057-2/data_collected3<br> trusted.bit-rot.version=0x0200000000000000570308980001d157<br> trusted.gfid=0xa91444cec80d49cd98cbe53be4d5377e<br> trusted.pgfid.eee7bcf2-4f21-444d-a470-c37071e4d7ab=0x00000001<br> <br> getfattr: Removing leading '/' from absolute path names<br> getfattr: /data/brick03bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory<br> getfattr: /data/brick04bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory<br> getfattr: /data/brick05bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory<br> <br>target xattrs<br> gfs01bkp<br> getfattr: /data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory<br> getfattr: /data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory<br><br> gfs02bkp<br> # file: data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3<br> trusted.bit-rot.version=0x0200000000000000569bb8d20003ed00<br> trusted.gfid=0xdf95487705d34b2592b5eaed1e8dc4d3<br> trusted.pgfid.24a52e78-6e7c-4536-92b1-ee4419323e54=0x00000001<br> <br> getfattr: Removing leading '/' from absolute path names<br> # file: data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3<br> trusted.gfid=0xdf95487705d34b2592b5eaed1e8dc4d3<br> trusted.glusterfs.dht.linkto=0x6766736261636b75702d636c69656e742d3200<br> trusted.pgfid.24a52e78-6e7c-4536-92b1-ee4419323e54=0x00000001<br> <br> getfattr: Removing leading '/' from absolute path names<br> getfattr: /data/brick03bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory<br> getfattr: /data/brick04bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory<br> getfattr: /data/brick05bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3: No such file or directory<br> <br>stat: cannot stat `"../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3"': No such file or directory<br> <br>Rename Failed! ./homegfs/hpc_shared/motorsports/057-2/data_collected3 -> ../bkp00/./homegfs/hpc_shared/motorsports/057-2/data_collected3<br>========================================================================================================================<br>retry: renaming ./homegfs/hpc_shared/motorsports/54/data_collected4 -> ../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4<br> <br>source xattrs<br> gfs01bkp<br> # file: data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/54/data_collected4<br> trusted.bit-rot.version=0x02000000000000005702ecdc0001e78e<br> trusted.gfid=0x7860691318554b82a0f3ab8dd90fea98<br> trusted.pgfid.e3906384-e16a-4a5f-9b07-f156558954db=0x00000001<br> <br> getfattr: Removing leading '/' from absolute path names<br> getfattr: /data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory<br> <br> gfs02bkp<br> getfattr: /data/brick01bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory<br> getfattr: /data/brick02bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory<br> getfattr: /data/brick03bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory<br> getfattr: /data/brick04bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory<br> getfattr: /data/brick05bkp/gfsbackup/bkp01/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory<br> <br>target xattrs<br> gfs01bkp<br> # file: data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4<br> trusted.gfid=0x980c12097507431d953ee458ec14ca4a<br> trusted.glusterfs.dht.linkto=0x6766736261636b75702d636c69656e742d3100<br> trusted.pgfid.6a8ecf7c-5597-4725-9764-455f7e267667=0x00000001<br> <br> getfattr: Removing leading '/' from absolute path names<br> # file: data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4<br> trusted.bit-rot.version=0x0200000000000000569bb8d700074a42<br> trusted.gfid=0x980c12097507431d953ee458ec14ca4a<br> trusted.pgfid.6a8ecf7c-5597-4725-9764-455f7e267667=0x00000001<br> <br> getfattr: Removing leading '/' from absolute path names<br> <br> gfs02bkp<br> getfattr: /data/brick01bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory<br> getfattr: /data/brick02bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory<br> getfattr: /data/brick03bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory<br> getfattr: /data/brick04bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory<br> getfattr: /data/brick05bkp/gfsbackup/bkp01/../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4: No such file or directory<br> <br>stat: cannot stat `"../bkp00/./homegfs/hpc_shared/motorsports/54/data_collected4"': No such file or directory<br>Rename Succeeded!<br>========================================================================================================================<br> <br></span></font></blockquote><br><br></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 29, 2016 at 10:21 AM, Glomski, Patrick <span dir="ltr"><<a href="mailto:patrick.glomski@corvidtec.com" target="_blank">patrick.glomski@corvidtec.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Raghavendra,<br><br></div><div>This error is occurring in a shell script moving files between directories on a FUSE mount when overwriting an old file with a newer file (it's a backup script, moving an incremental backup of a file into a 'rolling full backup' directory).<br><br></div><div>As a temporary workaround, we parse the output of this shell script for move errors and handle the errors as they happen. Simply re-moving the files fails, so we stat the destination (to see if we can learn anything about the type of file that causes this behavior), delete the destination, and try the move again (success!). Typical output is as follows:<br><br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><font size="1"><span style="font-family:monospace,monospace"><span>/bin/mv: cannot move `./homegfs/hpc_shared/</span><span>motorsports/gmics/Raven/p11/</span><span>149/data_collected4' to `../bkp00/./homegfs/hpc_</span><span>shared/motorsports/gmics/</span><span>Raven/p11/149/data_collected4'</span><span>: File exists</span><br><span>/bin/mv: cannot move `./homegfs/hpc_shared/</span><span>motorsports/gmics/Raven/p11/</span><span>149/data_collected4' to `../bkp00/./homegfs/hpc_</span><span>shared/motorsports/gmics/</span><span>Raven/p11/149/data_collected4'</span><span>: File exists</span><br><span> File: `../bkp00/./homegfs/hpc_</span><span>shared/motorsports/gmics/</span><span>Raven/p11/149/data_collected4'</span><br><span> Size: 1714 Blocks: 4 IO Block: 131072 regular file</span><br><span>Device: 13h/19d Inode: 11051758947722304158 Links: 1</span><br><span>Access: (0660/-rw-rw----) Uid: ( 628/pkeistler) Gid: ( 2020/ gmirl)</span><br><span>Access: 2016-01-20 17:20:45.000000000 -0500</span><br><span>Modify: 2015-11-06 15:20:41.000000000 -0500</span><br><span>Change: 2016-01-27 03:35:00.434712146 -0500</span><br><span>retry: renaming ./homegfs/hpc_shared/</span><span>motorsports/gmics/Raven/p11/</span><span>149/data_collected4 -> ../bkp00/./homegfs/hpc_shared/</span><span>motorsports/gmics/Raven/p11/</span><span>149/data_collected4</span></span></font><br></blockquote><br></div><div>Not sure if that description rings any bells as to what the problem might be, but if not, I added some code to print out the 'getattr' for the source and destination file on all of the bricks (before we delete the destination) and will post to this thread the next time we have that issue.<br><br></div><div>Thanks,<br></div><div>Patrick<br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Fri, Apr 29, 2016 at 8:15 AM, Raghavendra G <span dir="ltr"><<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Wed, Apr 13, 2016 at 10:00 PM, David F. Robinson <span dir="ltr"><<a href="mailto:david.robinson@corvidtec.com" target="_blank">david.robinson@corvidtec.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div>I am running into two problems (possibly related?). </div><div><br></div><div>1) Every once in a while, when I do a 'rm -rf DIRNAME', it comes back with an error: </div><div> rm: cannot remove `DIRNAME` : Directory not empty</div><div><br></div><div> If I try the 'rm -rf' again after the error, it deletes the directory. The issue is that I have scripts that clean up directories, and they are failing unless I go through the deletes a 2nd time.</div></div></blockquote><div><br></div></span><div>What kind of mount are you using? Is it a FUSE or NFS mount? Recently we saw a similar issue on NFS clients on RHEL6 where rm -rf used to fail with ENOTEMPTY in some specific cases.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><br></div><div>2) I have different scripts to move a large numbers of files (5-25k) from one directory to another. Sometimes I receive an error: </div><div> /bin/mv: cannot move `xyz` to `../bkp00/xyz`: File exists</div></div></blockquote><div><br></div></span><div>Does ./bkp00/xyz exist on backend? If yes, what is the value of gfid xattr (key: "trusted.gfid") for "xyz" and "./bkp00/xyz" on backend bricks (I need gfid from all the bricks) when this issue happens?</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div><div><br></div><div> The move is done using '/bin/mv -f', so it should overwrite the file if it exists. I have tested this with hundreds of files, and it works as expected. However, every few days the script that moves the files will have problems with 1 or 2 files during the move. This is one move problem out of roughly 10,000 files that are being moved and I cannot figure out any reason for the intermittent problem.</div><div><br></div><div>Setup details for my gluster configuration shown below.</div><div><br></div><div><div>[root@gfs01bkp logs]# gluster volume info</div><div> </div><div>Volume Name: gfsbackup</div><div>Type: Distribute</div><div>Volume ID: e78d5123-d9bc-4d88-9c73-61d28abf0b41</div><div>Status: Started</div><div>Number of Bricks: 7</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: gfsib01bkp.corvidtec.com:/data/brick01bkp/gfsbackup</div><div>Brick2: gfsib01bkp.corvidtec.com:/data/brick02bkp/gfsbackup</div><div>Brick3: gfsib02bkp.corvidtec.com:/data/brick01bkp/gfsbackup</div><div>Brick4: gfsib02bkp.corvidtec.com:/data/brick02bkp/gfsbackup</div><div>Brick5: gfsib02bkp.corvidtec.com:/data/brick03bkp/gfsbackup</div><div>Brick6: gfsib02bkp.corvidtec.com:/data/brick04bkp/gfsbackup</div><div>Brick7: gfsib02bkp.corvidtec.com:/data/brick05bkp/gfsbackup</div><div>Options Reconfigured:</div><div>nfs.disable: off</div><div>server.allow-insecure: on</div><div>storage.owner-gid: 100</div><div>server.manage-gids: on</div><div>cluster.lookup-optimize: on</div><div>server.event-threads: 8</div><div>client.event-threads: 8</div><div>changelog.changelog: off</div><div>storage.build-pgfid: on</div><div>performance.readdir-ahead: on</div><div>diagnostics.brick-log-level: WARNING</div><div>diagnostics.client-log-level: WARNING</div><div>cluster.rebal-throttle: aggressive</div><div>performance.cache-size: 1024MB</div><div>performance.write-behind-window-size: 10MB</div><div><br></div><div><br></div><div>[root@gfs01bkp logs]# rpm -qa | grep gluster</div><div>glusterfs-server-3.7.9-1.el6.x86_64</div><div>glusterfs-debuginfo-3.7.9-1.el6.x86_64</div><div>glusterfs-api-3.7.9-1.el6.x86_64</div><div>glusterfs-resource-agents-3.7.9-1.el6.noarch</div><div>gluster-nagios-common-0.1.1-0.el6.noarch</div><div>glusterfs-libs-3.7.9-1.el6.x86_64</div><div>glusterfs-fuse-3.7.9-1.el6.x86_64</div><div>glusterfs-extra-xlators-3.7.9-1.el6.x86_64</div><div>glusterfs-geo-replication-3.7.9-1.el6.x86_64</div><div>glusterfs-3.7.9-1.el6.x86_64</div><div>glusterfs-cli-3.7.9-1.el6.x86_64</div><div>glusterfs-devel-3.7.9-1.el6.x86_64</div><div>glusterfs-rdma-3.7.9-1.el6.x86_64</div><div>samba-vfs-glusterfs-4.1.11-2.el6.x86_64</div><div>glusterfs-client-xlators-3.7.9-1.el6.x86_64</div><div>glusterfs-api-devel-3.7.9-1.el6.x86_64</div><div>python-gluster-3.7.9-1.el6.noarch</div></div><div><br></div>
</div><br></div></div>_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><span><font color="#888888"><br></font></span></blockquote></div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div>Raghavendra G<br></div>
</font></span></div></div>
<br></div></div>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>