<font size=2 face="sans-serif">Hi </font><tt><font size=2>Pranith</font></tt><font size=2 face="sans-serif">,
it is appreciated for your reply.</font>
<br>
<br><tt><font size=2>Pranith Kumar Karampuri <pkarampu@redhat.com>
写于 2016/01/20 18:51:19:<br>
<br>
> 发件人: Pranith Kumar Karampuri <pkarampu@redhat.com></font></tt>
<br><tt><font size=2>> 收件人: li.ping288@zte.com.cn, gluster-devel@gluster.org,
</font></tt>
<br><tt><font size=2>> 日期: 2016/01/20 18:51</font></tt>
<br><tt><font size=2>> 主题: Re: [Gluster-devel] Gluster AFR volume
write performance has <br>
> been seriously affected by GLUSTERFS_WRITE_IS_APPEND in afr_writev</font></tt>
<br><tt><font size=2>> <br>
> Sorry for the delay in response.<br>
</font></tt>
<br><tt><font size=2>> On 01/15/2016 02:34 PM, li.ping288@zte.com.cn
wrote:</font></tt>
<br><tt><font size=2>> GLUSTERFS_WRITE_IS_APPEND Setting in afr_writev
function at <br>
> glusterfs client end makes the posix_writev in the server end deal
<br>
> IO write fops from parallel to serial in consequence. <br>
> <br>
> i.e. multiple io-worker threads carrying out IO write fops are
<br>
> blocked in posix_writev to execute final write fop pwrite/pwritev
in<br>
> __posix_writev function ONE AFTER ANOTHER. <br>
> <br>
> For example: <br>
> <br>
> thread1: iot_worker -> ... -> posix_writev() |
<br>
> thread2: iot_worker -> ... -> posix_writev() |
<br>
> thread3: iot_worker -> ... -> posix_writev() ->
__posix_writev() <br>
> thread4: iot_worker -> ... -> posix_writev() |
<br>
> <br>
> there are 4 iot_worker thread doing the 128KB IO write fops as <br>
> above, but only one can execute __posix_writev function and the <br>
> others have to wait. <br>
> <br>
> however, if the afr volume is configured on with storage.linux-aio
<br>
> which is off in default, the iot_worker will use posix_aio_writev
<br>
> instead of posix_writev to write data. <br>
> the posix_aio_writev function won't be affected by <br>
> GLUSTERFS_WRITE_IS_APPEND, and the AFR volume write performance goes
up. </font></tt>
<br><tt><font size=2>> I think this is a bug :-(.</font></tt>
<br>
<br><tt><font size=2>Yeah, I agree with you. I suppose the GLUSTERFS_WRITE_IS_APPEND
is a misuse in afr_writev.</font></tt>
<br><tt><font size=2>I checked the original intent of GLUSTERS_WRITE_IS_APPEND
change at review website:</font></tt>
<br><a href=http://review.gluster.org/#/c/5501/><font size=3 color=blue><u>http://review.gluster.org/#/c/5501/</u></font></a><font size=3>
</font>
<br>
<br><tt><font size=2>The initial purpose seems to avoid an unnecessary
</font></tt><tt><font size=2 color=#2f2f2f>fsync()</font></tt><font size=3>
</font><tt><font size=2>in </font></tt>
<br><tt><font size=2>afr_changelog_post_op_safe function if the writing
data position </font></tt>
<br><tt><font size=2>was currently at the end of the file, detected by
</font></tt>
<br><tt><font size=2>(preop.ia_size == offset || (fd->flags & O_APPEND))
in posix_writev.</font></tt>
<br>
<br><tt><font size=2>In </font></tt><tt><font size=2>comparison with the
afr write performance loss, I think </font></tt>
<br><tt><font size=2>it costs too much. </font></tt>
<br>
<br><tt><font size=2>I suggest to make the </font></tt><tt><font size=2>GLUSTERS_WRITE_IS_APPEND
</font></tt><tt><font size=2>setting configurable</font></tt>
<br><tt><font size=2>just as ensure-durability in afr.</font></tt>
<br>
<br><tt><font size=2>> <br>
> So, my question is whether AFR volume could work fine with <br>
> storage.linux-aio configuration which bypass the <br>
> GLUSTERFS_WRITE_IS_APPEND setting in afr_writev, <br>
> and why glusterfs keeps posix_aio_writev different from posix_writev
? <br>
> <br>
> Any replies to clear my confusion would be grateful, and thanks in
advance.</font></tt>
<br><tt><font size=2>> What is the workload you have? multiple writers
on same file workloads?</font></tt>
<br>
<br><tt><font size=2>I test the afr gluster volume by fio like this:</font></tt>
<br><tt><font size=2>fio --filename=/mnt/afr/20G.dat --direct=1 --rw=write
--bs=128k --size=20G --numjobs=8 </font></tt>
<br><tt><font size=2>--runtime=60 --group_reporting --name=afr_test --iodepth=1
--ioengine=libaio</font></tt>
<br>
<br><tt><font size=2>The Glusterfs BRICKS are two IBM X3550 M3. </font></tt>
<br>
<br><tt><font size=2>The local disk direct write performance of 128KB IO
req block size is about 18MB/s </font></tt>
<br><tt><font size=2>in single thread and 80MB/s in 8 multi-threads.</font></tt>
<br>
<br><tt><font size=2>If the GLUSTERS_WRITE_IS_APPEND is configed, the afr
gluster volume write performance is 18MB/s</font></tt>
<br><tt><font size=2>as the single thread, and if not, the performance
is nearby 75MB/s.(network bandwith is enough)</font></tt>
<br><tt><font size=2><br>
> <br>
> Pranith</font></tt>
<br><tt><font size=2>> <br>
> <br>
> --------------------------------------------------------<br>
> ZTE Information Security Notice: The information contained in this
<br>
> mail (and any attachment transmitted herewith) is privileged and <br>
> confidential and is intended for the exclusive use of the addressee<br>
> (s). If you are not an intended recipient, any disclosure, <br>
> reproduction, distribution or other dissemination or use of the <br>
> information contained is strictly prohibited. If you have received
<br>
> this mail in error, please delete it and notify us immediately.<br>
> <br>
</font></tt>
<br><tt><font size=2>> <br>
> <br>
</font></tt>
<br><tt><font size=2>> _______________________________________________<br>
> Gluster-devel mailing list<br>
> Gluster-devel@gluster.org<br>
> </font></tt><a href="http://www.gluster.org/mailman/listinfo/gluster-devel"><tt><font size=2>http://www.gluster.org/mailman/listinfo/gluster-devel</font></tt></a>
<br><pre><font color="blue">
--------------------------------------------------------
ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s). If you are not an intended recipient, any disclosure, reproduction, distribution or other dissemination or use of the information contained is strictly prohibited. If you have received this mail in error, please delete it and notify us immediately.
</font></pre><br>