<div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><div> Yes, I just used a text editor i.e. vi to test this out, but my real application is surveillance camera system. The video was stored in GlusterFS by FUSE and Samba. </div><div> Because of the surveillance camera system 7*24 working mode, it doesn't have to cache the files. So I added O_DIRECT flag in GlusterFS client protocol open() and create(). According my test, I wrote a video file into GlusterFS in windows, then I didn't find any cache in GlusterFS node, and I read a video file in windows, it also wasn't cached in GlusterFS node, but some frames in the video file(at the last) was lost, just like my vi test. </div><div> IIRC the page-alighed may be the reason of this question. How to solve this problem? Or in GlusterFS 3.6, 3.7, is there the option to enable io-direct mode to avoid the video cache in GlusterFS node?</div><div> </div><br><br><br><br><div style="position:relative;zoom:1"></div><div id="divNeteaseMailCard"></div><br>At 2016-07-11 18:51:02, "Krutika Dhananjay" <kdhananj@redhat.com> wrote:<br> <blockquote id="isReplyContent" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><div dir="ltr"><div><div><div>What's the application you are running? Sounds like you're using a text editor like vi(m) to test this out?<br>Is the application opening the files with O_DIRECT?<br></div>Do you have the strace output of the running application that confirms that it is open()ing the file with O_DIRECT?<br></div>Also, what are the offsets and sizes of the writes on this file by this application in the strace output?<br><br></div>-Krutika<br><div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 11, 2016 at 2:44 PM, Keiviw <span dir="ltr"><<a href="mailto:keiviw@163.com" target="_blank">keiviw@163.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial">I have checked the page-aligned, i.e. the file was larger than one page, a part of the file(one page size) was saved successfully, and the rest(more than one page but less than two pages) was lost.<div><div class="h5"><br><br><br><br><br><div style="zoom:1"></div><div></div><br>At 2016-07-11 12:53:32, "Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>> wrote:<br> <blockquote style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid"><div dir="ltr">Is it possible to share the test you are running? As per your volume, o-direct is not enabled on your volume, i.e. the file shouldn't be opened with o-direct but as per the logs it is giving Invalid Argument as if there is something wrong with the arguments when we do o-direct write with wrong size. so I would like to test out why exactly is it giving this problem. Please note that for o-direct write to succeed, both offset and size should be page-aligned, something like multiple of 512 is one way to check it.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jul 10, 2016 at 5:19 PM, Keiviw <span dir="ltr"><<a href="mailto:keiviw@163.com" target="_blank">keiviw@163.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><div>My volume info:</div><div> </div><div> Volume Name: test</div><div> Type: Distribute</div><div> Volume ID: 9294b122-d81e-4b12-9b5c-46e89ee0e40b</div><div> Status: Started</div><div> Number of Bricks: 2</div><div> Transport-type: tcp</div><div> Bricks:</div><div> Brick1: compute2:/home/brick1</div><div> Brick2: compute2:/home/brick2</div><div> Options Reconfigured:</div><div> performance.flush-behind: off</div><div> storage.linux-aio: off</div><div>My brick logs(I have cleaned up the history log):</div><div> [2016-07-10 11:42:50.577683] E [posix.c:2128:posix_writev] 0-test-posix: write failed: offset 0, Invalid argument</div><div> [2016-07-10 11:42:50.577735] I [server3_1-fops.c:1414:server_writev_cbk] 0-test-server: 8569840: WRITEV 5 (526a3118-9994-429e-afc0-4aa063606bde) ==> -1 (Invalid argument)</div><div> [2016-07-10 11:42:54.583038] E [posix.c:2128:posix_writev] 0-test-posix: write failed: offset 0, Invalid argument</div><div> [2016-07-10 11:42:54.583080] I [server3_1-fops.c:1414:server_writev_cbk] 0-test-server: 8569870: WRITEV 5 (c3d28f34-8f43-446d-8d0b-80841ae8ec5b) ==> -1 (Invalid argument)</div><div>My mnt-test-.logs:</div><div> [2016-07-10 11:42:50.577816] W [client3_1-fops.c:876:client3_1_writev_cbk] 0-test-client-1: remote operation failed: Invalid argument</div><div> [2016-07-10 11:42:50.578508] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 12398282: FSYNC() ERR => -1 (Invalid argument)</div><div> [2016-07-10 11:42:54.583156] W [client3_1-fops.c:876:client3_1_writev_cbk] 0-test-client-1: remote operation failed: Invalid argument</div><div> [2016-07-10 11:42:54.583762] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 12398317: FSYNC() ERR => -1 (Invalid argument)</div><div><div><div><br></div><div><br></div><br><br><br><div style="zoom:1"></div><div></div><br>在 2016-07-10 19:18:18，"Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>> 写道：<br> <blockquote style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid"><div dir="ltr"><div><br>To me it looks like a case of a flush triggering a write() that was cached by write-behind and because the write buffer<br></div><div>did not meet the page alignment requirement with o-direct write, it was failed with EINVAL and the trigger fop - i.e., flush() was failed with the 'Invalid argument' error code.<br><br></div><div>Could you attach the brick logs as well, so that we can confirm the theory?<br></div><div><br></div><div>-Krutika</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jul 9, 2016 at 9:31 PM, Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Pranith/Krutika,<div><br></div><div>Your inputs please, IIRC we'd need to turn on some o_direct option here?</div><div><div><div><br></div><div><br>On Saturday 9 July 2016, Keiviw <<a href="mailto:keiviw@163.com" target="_blank">keiviw@163.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>
<div style="line-height:1.5;text-align:justify;text-justify:inter-ideograph">
<div>The errors also occured in GlusterFS 3.6.7，I just add the O_DIRECT flag in client protocol open() and create()! How to explain and solve the problem?</div><div><br></div><div><span style="color:#888;font-size:15px">发自 </span><a href="http://u.163.com/signature" style="font-size:15px;color:#2e90eb" target="_blank">网易邮箱大师</a></div>
<div style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;margin-top:45px;margin-bottom:20px"><div style="font-size:14px;line-height:1.5;word-break:break-all;margin-left:10px;margin-right:10px">On 07/09/2016 17:58, <a style="text-decoration:none;color:#2a97ff">Atin Mukherjee</a> wrote:</div></div><blockquote style="margin:0">Any specific reason of using 3.3 given that its really quite old? We are at 3.6, 3.7 & 3.8 supportability matrix now.<div><span></span><br><br>On Saturday 9 July 2016, Keiviw <<a>keiviw@163.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><div>hi,</div><div> I have installed GlusterFS 3.3.0, and now I get Fsync failures when saving files with the O_DIRECT flag in open() and create().</div><div> 1, I tried to save a flie in vi and got this error:</div><div> "test" E667: Fsync failed</div><div> 2, I see this in the logs: </div><div> [2016-07-07 14:20:10.325400] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 102: FSYNC() ERR => -1 (Invalid argument)</div><div> [2016-07-07 14:20:13.930384] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 137: FSYNC() ERR => -1 (Invalid argument)</div><div> [2016-07-07 14:20:51.199448] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 174: FLUSH() ERR => -1 (Invalid argument)</div><div> [2016-07-07 14:21:32.804738] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 206: FLUSH() ERR => -1 (Invalid argument)</div><div> [2016-07-07 14:21:43.702146] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 276: FSYNC() ERR => -1 (Invalid argument)</div><div> [2016-07-07 14:21:51.296809] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 314: FSYNC() ERR => -1 (Invalid argument)</div><div> [2016-07-07 14:21:54.062687] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 349: FSYNC() ERR => -1 (Invalid argument)</div><div> [2016-07-07 14:22:54.678960] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 429: FSYNC() ERR => -1 (Invalid argument)</div><div> [2016-07-07 14:24:35.546980] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 505: FSYNC() ERR => -1 (Invalid argument)</div><div> [2016-07-07 14:24:48.696888] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 538: FSYNC() ERR => -1 (Invalid argument)</div><div> [2016-07-07 14:24:55.824752] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 575: FSYNC() ERR => -1 (Invalid argument)</div><div> [2016-07-07 14:25:04.364629] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 612: FSYNC() ERR => -1 (Invalid argument)</div><div> [2016-07-07 14:27:08.579052] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 713: FSYNC() ERR => -1 (Invalid argument)</div><div> [2016-07-07 14:46:44.635169] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 859: FLUSH() ERR => -1 (Invalid argument)</div><div> [2016-07-07 14:46:55.892041] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 989: FLUSH() ERR => -1 (Invalid argument)</div><div> [2016-07-07 14:51:45.904233] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 148747: FLUSH() ERR => -1 (Invalid argument)</div><div> [2016-07-07 14:54:45.603557] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 148986: FLUSH() ERR => -1 (Invalid argument)</div><div> [2016-07-07 14:59:18.818972] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 292106: FLUSH() ERR => -1 (Invalid argument)</div><div> [2016-07-07 15:48:32.708207] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 6753592: FLUSH() ERR => -1 (Invalid argument)</div><div> [2016-07-07 16:24:10.476228] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 10733703: FLUSH() ERR => -1 (Invalid argument)</div><div> [2016-07-07 16:30:42.388220] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 11497698: FLUSH() ERR => -1 (Invalid argument)</div><div> [2016-07-07 16:44:18.247511] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 12037185: FLUSH() ERR => -1 (Invalid argument)</div><div> [2016-07-08 01:00:55.131141] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 12181783: FLUSH() ERR => -1 (Invalid argument)</div><div> [2016-07-08 06:54:13.418607] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 12397060: FLUSH() ERR => -1 (Invalid argument)</div></div><br><br><span title="neteasefooter"><p> </p></span></blockquote></div><br><br>-- <br>Atin<br>Sent from iPhone<br>
</blockquote>
</div>
<br><br>
</div></blockquote></div><br><br>-- <br>Atin<br>Sent from iPhone<br>
</div></div></blockquote></div><br></div>
</blockquote></div></div></div><br><br><span title="neteasefooter"><p> </p></span></blockquote></div><br><br clear="all"><br>-- <br><div data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div>
</blockquote></div></div></div><br><br><span title="neteasefooter"><p> </p></span></blockquote></div><br></div>
</blockquote></div><br><br><span title="neteasefooter"><p> </p></span>