<HTML><HEAD>
<STYLE id=eMClientCss>BLOCKQUOTE.cite {
PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px
}
BLOCKQUOTE.cite2 {
PADDING-TOP: 0px; PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; MARGIN-TOP: 3px; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px
}
.plain PRE {
FONT-SIZE: 100%; FONT-FAMILY: monospace; FONT-WEIGHT: normal; FONT-STYLE: normal
}
.plain TT {
FONT-SIZE: 100%; FONT-FAMILY: monospace; FONT-WEIGHT: normal; FONT-STYLE: normal
}
A IMG {
BORDER-TOP: 0px; BORDER-RIGHT: 0px; BORDER-BOTTOM: 0px; BORDER-LEFT: 0px
}
#x40322ca65d6d4278b73f5a07f0c10875 {
FONT-SIZE: 11pt; FONT-FAMILY: Calibri
}
.plain PRE {
FONT-SIZE: 11pt; FONT-FAMILY: Calibri
}
.plain TT {
FONT-SIZE: 11pt; FONT-FAMILY: Calibri
}
BODY {
FONT-SIZE: 11pt; FONT-FAMILY: Calibri
}
</STYLE>
</HEAD>
<BODY scroll=auto class>
<DIV>That did not fix the issue (see below). I also have run into another possibly related issue. After untarring the boost directory and compiling the software, I cannot delete the source directory structure. It says "direct not empty".</DIV>
<DIV><BR>corvidpost5:temp3/gfs> \rm -r boost_1_57_0<BR>rm: cannot remove `boost_1_57_0/libs/numeric/odeint/test': Directory not empty<BR>corvidpost5:temp3/gfs> cd boost_1_57_0/libs/numeric/odeint/test/<BR>corvidpost5:odeint/test> ls -al<BR>total 0<BR>drwxr-x--- 3 dfrobins users 94 Dec 20 01:51 ./<BR>drwx------ 3 dfrobins users 100 Dec 20 01:51 ../<BR></DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV><SPAN id=x40322ca65d6d4278b73f5a07f0c10875><STRONG><U>cluster.read-hash-mode to 2 results </U></STRONG></SPAN></DIV>
<DIV> </DIV>
<DIV>corvidpost5:TankExamples/DakotaList> ls -al<BR>total 5<BR>drwxr-x--- 2 dfrobins users 166 Dec 22 11:16 ./<BR>drwxr-x--- 6 dfrobins users 445 Dec 22 11:16 ../<BR>lrwxrwxrwx 1 dfrobins users 25 Dec 22 11:16 EvalTank.py -> ../tank_model/EvalTank.py*<BR><STRONG>---------- 1 dfrobins users 0 Dec 22 11:16 FEMTank.py</STRONG><BR>-rwx--x--- 1 dfrobins users 734 Nov 7 11:05 RunTank.sh*<BR>-rw------- 1 dfrobins users 1432 Nov 7 11:05 dakota_PandL_list.in<BR>-rw------- 1 dfrobins users 1860 Nov 7 11:05 dakota_Ponly_list.in<BR> </DIV>
<DIV>gluster> volume info homegfs<BR> <BR>Volume Name: homegfs<BR>Type: Distributed-Replicate<BR>Volume ID: 1e32672a-f1b7-4b58-ba94-58c085e59071<BR>Status: Started<BR>Number of Bricks: 4 x 2 = 8<BR>Transport-type: tcp<BR>Bricks:<BR>Brick1: gfsib01a.corvidtec.com:/data/brick01a/homegfs<BR>Brick2: gfsib01b.corvidtec.com:/data/brick01b/homegfs<BR>Brick3: gfsib01a.corvidtec.com:/data/brick02a/homegfs<BR>Brick4: gfsib01b.corvidtec.com:/data/brick02b/homegfs<BR>Brick5: gfsib02a.corvidtec.com:/data/brick01a/homegfs<BR>Brick6: gfsib02b.corvidtec.com:/data/brick01b/homegfs<BR>Brick7: gfsib02a.corvidtec.com:/data/brick02a/homegfs<BR>Brick8: gfsib02b.corvidtec.com:/data/brick02b/homegfs<BR>Options Reconfigured:<BR><STRONG><EM>cluster.read-hash-mode: 2</EM></STRONG><BR>performance.stat-prefetch: off<BR>performance.io-thread-count: 32<BR>performance.cache-size: 128MB<BR>performance.write-behind-window-size: 128MB<BR>server.allow-insecure: on<BR>network.ping-timeout: 10<BR>storage.owner-gid: 100<BR>geo-replication.indexing: off<BR>geo-replication.ignore-pid-check: on<BR>changelog.changelog: on<BR>changelog.fsync-interval: 3<BR>changelog.rollover-time: 15<BR>server.manage-gids: on<BR> </DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>------ Original Message ------</DIV>
<DIV>From: "Vijay Bellur" <<A href="mailto:vbellur@redhat.com">vbellur@redhat.com</A>></DIV>
<DIV>To: "David F. Robinson" <<A href="mailto:david.robinson@corvidtec.com">david.robinson@corvidtec.com</A>></DIV>
<DIV>Cc: "Justin Clift" <<A href="mailto:justin@gluster.org">justin@gluster.org</A>>; "Gluster Devel" <<A href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</A>></DIV>
<DIV>Sent: 12/22/2014 9:23:44 AM</DIV>
<DIV>Subject: Re: [Gluster-devel] 3.6.1 issue</DIV>
<DIV> </DIV>
<DIV id=xe6a78d76c57747c39502bbbc34f6e69b>
<BLOCKQUOTE class=cite2 type="cite">
<DIV>On 12/21/2014 11:10 PM, David F. Robinson wrote: </DIV>
<BLOCKQUOTE class=cite type="cite">
<DIV>So for now it is up to all of the individual users to know they cannot use tar without the -P switch if they are accessing a data storage system that uses gluster? </DIV>
<DIV> </DIV></BLOCKQUOTE>
<DIV> </DIV>
<DIV>Setting volume option cluster.read-hash-mode to 2 could help here. Can you please check if this resolves the problem without -P switch? </DIV>
<DIV> </DIV>
<DIV>-Vijay </DIV>
<DIV> </DIV>
<DIV> </DIV>
<BLOCKQUOTE class=cite2 type="cite">
<BLOCKQUOTE class=cite type="cite">
<DIV>On Dec 21, 2014, at 12:30 PM, Vijay Bellur <<A href="mailto:vbellur@redhat.com">vbellur@redhat.com</A>> wrote: </DIV>
<DIV> </DIV>
<BLOCKQUOTE class=cite2 type="cite">
<DIV>On 12/20/2014 12:09 PM, David F. Robinson wrote: </DIV>
<DIV>Seems to work with "-xPf". I obviously couldn't check all of the files, </DIV>
<DIV>but the two specific ones that I noted in my original email do not show </DIV>
<DIV>any problems when using -P... </DIV></BLOCKQUOTE>
<DIV> </DIV>
<DIV>This is related to the way tar extracts symbolic links by default & its interaction with GlusterFS. In a nutshell the following steps are involved in creation of symbolic links on the destination: </DIV>
<DIV> </DIV>
<DIV>a) Create an empty regular placeholder file with permission bits set to 0 and the name being that of the symlink source file. </DIV>
<DIV> </DIV>
<DIV>b) Record the device, inode numbers and the mtime of the placeholder file through stat. </DIV>
<DIV> </DIV>
<DIV>c) After the first pass of extraction is complete, there is a second pass involved to set right symbolic links. In this phase a stat is performed on the placeholder file. If all attributes recorded in b) are in sync with the latest information from stat buf, only then the placeholder is unlinked and a new symbolic link is created. If any attribute is out of sync, the unlink and creation of symbolic link do not happen. </DIV>
<DIV> </DIV>
<DIV>In the case of replicated GlusterFS volumes, the mtimes can vary across nodes during the creation of placeholder files. If the stat calls in steps b) and c) land on different nodes, then there is a very good likelihood that tar would skip creation of symbolic links and leave behind the placeholder files. </DIV>
<DIV> </DIV>
<DIV>A little more detail about this particular implementation behavior of symlinks for tar can be found at [1]. </DIV>
<DIV> </DIV>
<DIV>To overcome this behavior, we can make use of the P switch with tar command during extraction which will create the link file directly and not go ahead with the above set of steps. </DIV>
<DIV> </DIV>
<DIV>Keeping timestamps in sync across the cluster will help to an extent in preventing this situation. There are ongoing refinements in replicate's selection of read-child which will help in addressing this problem. </DIV>
<DIV> </DIV>
<DIV>-Vijay </DIV>
<DIV> </DIV>
<DIV>[1] <A href="http://lists.debian.org/debian-user/2003/03/msg03249.html">http://lists.debian.org/debian-user/2003/03/msg03249.html</A> </DIV>
<DIV> </DIV></BLOCKQUOTE>
<DIV> </DIV>
<DIV> </DIV></BLOCKQUOTE>
<DIV> </DIV></BLOCKQUOTE></DIV></BODY></HTML>