<html><body><div style="font-family: garamond,new york,times,serif; font-size: 12pt; color: #000000"><div>Hi,</div><div><br></div><div>So are the "submitter" clients exactly doing the same commands that you just pasted:</div><div>i.e., <span style="font-family: garamond, 'new york', times, serif; font-size: medium;">cp /localdir/job1234.txt /mnt/gv0/to_process/tmpspool followed by mv /mnt/gv0/to_process/tmpspool/job1234.txt /mnt/gv0/to_process in a loop?</span></div><div><span style="font-family: garamond, 'new york', times, serif; font-size: medium;">Or are they executing a hand-written program perhaps which open()s the file, write()s to the file, and then executes a rename() syscall?</span></div><div><span style="font-family: garamond, 'new york', times, serif; font-size: medium;"><br></span></div><div><span style="font-family: garamond, 'new york', times, serif; font-size: medium;">Also, is this issue hit if you turn off flush-behind (by doing a `gluster volume set <VOLNAME> performance.flush-behind off`)?</span></div><div><span style="font-family: garamond, 'new york', times, serif; font-size: medium;"><br></span></div><div><span style="font-family: garamond, 'new york', times, serif; font-size: medium;">-Krutika</span></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Kingsley" <gluster@gluster.dogwind.com><br><b>To: </b>gluster-users@gluster.org<br><b>Sent: </b>Friday, April 10, 2015 2:01:54 PM<br><b>Subject: </b>[Gluster-users] 3.6.2, file-write events out of order - data missing temporarily<br><div><br></div>Hi,<br><div><br></div>We're running gluster 3.6.2 on CentOS 7, using a replicate-only volume<br>with 4 way replication.<br><div><br></div>We have 10 hosts mounting the volume - 6 running CentOS 6 that submit<br>jobs to a "to-process" directory on the gluster volume, and 4 running<br>CentOS 7 that process entries from that directory.<br><div><br></div>So that the 4 "processor" machines don't read partly written files, the<br>submitting machines write to a tmpspool subdirectory first (subdirectory<br>of the to-process directory on the gluster volume) and then move it into<br>the main to-process directory once written, eg:<br><div><br></div>cp /localdir/job1234.txt /mnt/gv0/to_process/tmpspool<br>mv /mnt/gv0/to_process/tmpspool/job1234.txt /mnt/gv0/to_process<br><div><br></div>These job files are small (less than 500 bytes).<br><div><br></div>However, if one of the processor machines picks up one of the files<br>quite quickly after it appears, it sees a smaller (ie not fully written)<br>file. If it waits a few seconds and tries again, the file is complete.<br><div><br></div>Is this a known bug that might be fixed in 3.6.3, or is it a new issue?<br><div><br></div>One I recently saw was a 441 byte file that was moved from tmpspool into<br>to_process by the client machine, but was read from to_process as a 391<br>byte file by one of the processing machines with the last 2 lines<br>missing, but read again 3 seconds later with all of the data in place.<br><div><br></div>Curiously, when there is data missing, it's always whole lines; the<br>temporarily-short file never seems to end half way along a line of text.<br><div><br></div>Cheers,<br>Kingsley.<br><div><br></div>_______________________________________________<br>Gluster-users mailing list<br>Gluster-users@gluster.org<br>http://www.gluster.org/mailman/listinfo/gluster-users<br></blockquote><div><br></div></div></body></html>