<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 25, 2016 at 6:44 PM, Raghavendra Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class=""><br>
<br>
----- Original Message -----<br>
> From: "Paul Cuzner" <<a href="mailto:pcuzner@redhat.com">pcuzner@redhat.com</a>><br>
> To: "Raghavendra Gowdappa" <<a href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a>><br>
> Cc: "Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>>, "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a>><br>
> Sent: Thursday, February 25, 2016 10:51:17 AM<br>
> Subject: Re: What's the correct way to enable direct-IO?<br>
><br>
> This is great info - with a lot of options to take in :)<br>
><br>
> To summarise, to enable direct-io and bypass the kernel filesystem cache<br>
> for a volume<br>
> 1. Mount the brick with direct-io-mode=enable option<br>
> 2. run vol set <vol> performance.strict-o-direct on<br>
<br>
</span>Would individual files be opened with O_DIRECT? If yes, this is sufficient. But if we want to do that for all files, the only way to do this is to disable write-behind.<br></blockquote><div><br></div><div>file opens will not be with o_direct BUT if I understand your earlier comments correctly the update to add o-direct to the storage/posix will force o-direct on open/create calls - which would mean vol set <bla> performance.write-behind off would not be required.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class=""><br>
> 3. update the vol files with 'o-direct' option in storage/posix (at least<br>
> for now)<br>
<br>
</span>4. Quick read should be made aware of O_DIRECT (it doesn't as of now). Again this is on the read-path. If you don't want O_DIRECT semantics for read path (is there one?), this is fine.<br>
<div class=""><div class="h5"><br></div></div></blockquote><div>performance.quick-read is off for the virt use case.<br><br></div><div>I'll try and test this out tomorrow.<br><br></div><div>Thanks!<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class="h5">
><br>
> Is that right?<br>
><br>
><br>
><br>
> On Thu, Feb 25, 2016 at 5:56 PM, Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a>><br>
> wrote:<br>
><br>
> ><br>
> ><br>
> > ----- Original Message -----<br>
> > > From: "Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>><br>
> > > To: "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a>>, "Raghavendra Gowdappa"<br>
> > <<a href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a>><br>
> > > Cc: "Paul Cuzner" <<a href="mailto:pcuzner@redhat.com">pcuzner@redhat.com</a>><br>
> > > Sent: Thursday, February 25, 2016 7:28:30 AM<br>
> > > Subject: What's the correct way to enable direct-IO?<br>
> > ><br>
> > > Hi,<br>
> > ><br>
> > > git-grep tells me there are multiple options in our code base for<br>
> > enabling<br>
> > > direct-IO on a gluster volume, at several layers in the translator stack:<br>
> > > i) use the mount option 'direct-io-mode=enable'<br>
> ><br>
> > This option is between kernel and glusterfs. Specifically it asks fuse<br>
> > kernel module to bypass page-cache. Note that when this option is set,<br>
> > direct-io is enabled for _all_ fds irrespective of whether applications<br>
> > have used O_DIRECT in their open/create calls or not.<br>
> ><br>
> > > ii) enable 'network.remote-dio' which is a protocol/client option using<br>
> > > volume set command<br>
> ><br>
> > This is an option introduced by [1] to _filter_ O_DIRECT flags in<br>
> > open/create calls before sending those requests to server. The option name<br>
> > is misleading here. However please note that this is the key (alias?) used<br>
> > by glusterd. The exact option name used by protocol/client is<br>
> > "filter_O_DIRECT" and its fine. Probably we should file a bug on glusterd<br>
> > to change the name?<br>
> ><br>
> > Coming to your use case, we don't want to filter O_DIRECT from reaching<br>
> > brick. Hence, we need to set this option to _off_ (by default its<br>
> > disabled).<br>
> ><br>
> > I am still not sure what is the relevance of this option against the bug<br>
> > it was introduced. If we need direct-io, we've to pass it to brick too, so<br>
> > that backend fs on brick is configured appropriately.<br>
> ><br>
> > [1] <a href="http://review.gluster.org/4206" rel="noreferrer" target="_blank">http://review.gluster.org/4206</a><br>
> > [2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=845213" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=845213</a><br>
> ><br>
> > > iii) enable performance.strict-o-direct which is a<br>
> > performance/write-behind<br>
> > > option using volume-set command<br>
> ><br>
> > Yes, write-behind honours O_DIRECT only if this option is set. So, we need<br>
> > to enable this for your use-case. Also, note that applications still need<br>
> > to use O_DIRECT in open/create calls.<br>
> ><br>
> > To summarize, following are the ways to bypass write-behind cache:<br>
> > 1. disable write-behind :).<br>
> > 2. applications use O_SYNC/O_DSYNC in open calls<br>
> > 3. enable performance.strict-o-direct _and_ applications should use<br>
> > O_DIRECT in open/create calls.<br>
> ><br>
> > > iv) use 'o-direct' option in storage/posix, volume-set on which reports<br>
> > that<br>
> > > the option doesn't exist.<br>
> ><br>
> > The option exists in storage/posix. But, there is no way to set it through<br>
> > cli (probably you can send a patch to do that if necessary). With this<br>
> > option, O_DIRECT is passed with _every_ open/create call on the brick.<br>
> ><br>
> > ><br>
> > > So then the question is - what is a surefire way to get direct-io-like<br>
> > > behavior on gluster volume(s)?<br>
> ><br>
> > There is no one global option. You need to configure various translators<br>
> > in the stack. Probably [2] was asking for such a feature. Also, as you<br>
> > might've noticed above the behavior/interpretation of these options is not<br>
> > same across all translators (like some are global and some are local only<br>
> > to an fd etc).<br>
> ><br>
> > Also note that apart from the options you listed above,<br>
> > 1. Quick-read is not aware of O_DIRECT. We need to make it to disable<br>
> > caching if open happens with O_DIRECT.<br>
> > 2. Handling of Quota Marker xattrs is not synchronous (though not exactly<br>
> > an O_DIRECT requirement) as marking is done after sending reply to calls<br>
> > like writev.<br>
> ><br>
> > On a related note, found article [3] to be informative.<br>
> ><br>
> > [1] <a href="http://review.gluster.org/4206" rel="noreferrer" target="_blank">http://review.gluster.org/4206</a><br>
> > [2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=845213" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=845213</a><br>
> > [3] <a href="https://lwn.net/Articles/457667/" rel="noreferrer" target="_blank">https://lwn.net/Articles/457667/</a><br>
> ><br>
> > regards,<br>
> > Raghavendra.<br>
> ><br>
><br>
</div></div></blockquote></div><br></div></div>