<HTML><HEAD>
<STYLE id=eMClientCss>blockquote.cite { margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px; border-left: 1px solid #cccccc }
blockquote.cite2 {margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px; border-left: 1px solid #cccccc; margin-top: 3px; padding-top: 0px; }
.plain pre, .plain tt { font-family: monospace; font-size: 100%; font-weight: normal; font-style: normal; white-space: pre-wrap; }
a img { border: 0px; }body {font-family: Times New Roman;font-size: 12pt;}
.plain pre, .plain tt {font-family: Times New Roman;font-size: 12pt;}
</STYLE>
<STYLE>#x2c6bddfe6a90449b8a2838ae2a9ba8cc BLOCKQUOTE.cite
{PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px}
#x2c6bddfe6a90449b8a2838ae2a9ba8cc BLOCKQUOTE.cite2
{MARGIN-TOP: 3px; PADDING-TOP: 0px; PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px}
#x2c6bddfe6a90449b8a2838ae2a9ba8cc .plain PRE, #x2c6bddfe6a90449b8a2838ae2a9ba8cc .plain TT
{FONT-SIZE: 100%; FONT-FAMILY: monospace; WHITE-SPACE: pre-wrap; FONT-WEIGHT: normal; FONT-STYLE: normal}
#x2c6bddfe6a90449b8a2838ae2a9ba8cc A IMG
{BORDER-TOP: 0px; BORDER-RIGHT: 0px; BORDER-BOTTOM: 0px; BORDER-LEFT: 0px}
#x2c6bddfe6a90449b8a2838ae2a9ba8cc #x6c555ae0819a4526b0bbad1bc21c76d6, #x2c6bddfe6a90449b8a2838ae2a9ba8cc #xa2e15f847053498c9f73f30e6507a826, #x2c6bddfe6a90449b8a2838ae2a9ba8cc .plain PRE, #x2c6bddfe6a90449b8a2838ae2a9ba8cc .plain TT, #x2c6bddfe6a90449b8a2838ae2a9ba8cc
{FONT-SIZE: 12pt; FONT-FAMILY: Times New Roman}
</STYLE>
</HEAD>
<BODY scroll=auto class>
<DIV>Resent to gluster-devel and gluster-user. Anyone have any ideas what might be causing this issue?</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>------ Forwarded Message ------</DIV>
<DIV>From: "David Robinson" <<A href="mailto:drobinson@corvidtec.com">drobinson@corvidtec.com</A>></DIV>
<DIV>To: "Shyam" <<A href="mailto:srangana@redhat.com">srangana@redhat.com</A>></DIV>
<DIV>Cc: "Patrick Glomski" <<A href="mailto:patrick.glomski@corvidtec.com">patrick.glomski@corvidtec.com</A>></DIV>
<DIV>Sent: 8/17/2015 6:09:07 PM</DIV>
<DIV>Subject: Re[2]: [Red Hat Bugzilla] Your Outstanding Requests</DIV>
<DIV> </DIV>
<DIV id=x2c6bddfe6a90449b8a2838ae2a9ba8cc class=plain>
<DIV>Shyam,</DIV>
<DIV> </DIV>
<DIV>I am revisiting this with some additional testing. I think that a lot of the "small file performance" issues that people bring up could be related to this bug. I know that it is causing me major issues on my production system as it continues to slow down as files are added. We currently have 200TB on that system and we are getting to a point that the system is becoming fairly unusable due to the increasing slowdowns we are witnessing. e.g. 'ls -R boost' has gone from 5-seconds to over 3-minutes </DIV>
<DIV> </DIV>
<DIV>The test is repeatable and it does not appear to be related to XFS or the system hardware or configuration. <SPAN id=xa2e15f847053498c9f73f30e6507a826 class=plain>Let me know if you are okay with me posting this to the gluster-user and gluster-devel message boards. I would like to know if anyone else is seeing the same behavior. </SPAN></DIV>
<DIV> </DIV>
<DIV><SPAN id=x6c555ae0819a4526b0bbad1bc21c76d6 class=plain>
<DIV>The steps I took are:</DIV>
<OL>
<LI>Run speedtest1 (see attached)
<OL>
<LI>Creates a gluster volume (testbrick01) and mounts the volume </LI>
<LI>Run timing of operations (tar is done 10x to show slowdown; slowdown is directly proportional to the number of files on the volume) </LI>
<LI>Repeat timing of the operations directly on the XFS partition (/data/brick01) </LI></OL></LI>
<LI><EM><STRONG>Reboot the machine</STRONG></EM> and run speedtest2 (see attached)
<OL>
<LI>repeat the timings from speedtest1 to show the slowdown after the reboot
<OL>
<LI>*** After the reboot, all of the operations are significantly slowed down. The slowdown isn't just in the creation of the symbolic links in .glusterfs as previously thought. </LI>
<LI>*** The timings done after the reboot directly on the XFS partition (/data/brick01) do not show any slowdown. This indicates that it isn't an XFS issue, right?</LI></OL></LI></OL></LI>
<LI>Run speedtest3 (see attached)
<OL>
<LI>Create a new gluster volume (testbrick02), mount, and repeat the above steps to show that everything is working fine.</LI></OL></LI></OL></SPAN></DIV>
<DIV> </DIV>
<DIV>After a reboot, the gluster volume slows down significantly and the amount of the slowdown is proportional to the number of files (not the amount of space used). After only 10-extractions, there are roughly 7.5-million files in the volume and the extraction time has gone from 1.5 minutes to 5-minutes. The "du -h" went from 8-seconds to 30-seconds. Note that these timings for speedtest2 will continually get worse if you add more files to the gluster volume. Summary of timing results:</DIV>
<DIV> </DIV>
<DIV>
<TABLE style="HEIGHT: 116px; WIDTH: 913px" cellSpacing=1 cols=4 cellPadding=1 width=913 border=2>
<CAPTION>Gluster Timing</CAPTION>
<TBODY>
<TR>
<TD>
<DIV align=center> Test </DIV></TD>
<TD>
<DIV align=center>speedtest1</DIV></TD>
<TD>
<DIV align=center>speedtest2</DIV></TD>
<TD>
<DIV align=center>speedtest3</DIV></TD></TR>
<TR>
<TD>
<DIV align=center>gluster: tar -xPf</DIV></TD>
<TD>
<DIV align=center>1m30s</DIV></TD>
<TD>
<DIV align=center>3.5-8m</DIV></TD>
<TD>
<DIV align=center>1m30s</DIV></TD></TR>
<TR>
<TD>
<DIV align=center>gluster: du -h -s</DIV></TD>
<TD>
<DIV align=center>8s</DIV></TD>
<TD>
<DIV align=center>30s</DIV></TD>
<TD>
<DIV align=center>7s</DIV></TD></TR>
<TR>
<TD>
<DIV align=center>gluster: find</DIV></TD>
<TD>
<DIV align=center>8s</DIV></TD>
<TD>
<DIV align=center>27s</DIV></TD>
<TD>
<DIV align=center>7s</DIV></TD></TR>
<TR>
<TD>
<DIV align=center>gluster: ls -R</DIV></TD>
<TD>
<DIV align=center>8s</DIV></TD>
<TD>
<DIV align=center>18s</DIV></TD>
<TD>
<DIV align=center>7s</DIV></TD></TR>
<TR>
<TD>
<DIV align=center>xfs: tar -xPf</DIV></TD>
<TD>
<DIV align=center>2s</DIV></TD>
<TD>
<DIV align=center>2s</DIV></TD>
<TD>
<DIV align=center>2s</DIV></TD></TR>
<TR>
<TD>
<DIV align=center>xfs: du -h -s</DIV></TD>
<TD>
<DIV align=center>.07s</DIV></TD>
<TD>
<DIV align=center>.07s</DIV></TD>
<TD>
<DIV align=center>.07s</DIV></TD></TR>
<TR>
<TD>
<DIV align=center>xfs: find</DIV></TD>
<TD>
<DIV align=center>.09s</DIV></TD>
<TD>
<DIV align=center>.09s</DIV></TD>
<TD>
<DIV align=center>.09s</DIV></TD></TR>
<TR>
<TD>
<DIV align=center>xfs: ls -R</DIV></TD>
<TD>
<DIV align=center>.13s</DIV></TD>
<TD>
<DIV align=center>.13s</DIV></TD>
<TD>
<DIV align=center>.13s</DIV></TD></TR></TBODY></TABLE></DIV>
<DIV> </DIV>
<DIV>Note that the good news is that there is absolutely no slowdown in the system if you do not do a reboot, so gluster is performing extremely well up to the point of a reboot. I created a test volume and extracted the boost directory 100-times to create 70-million files. There was no slowdown detected even with this extremely large number of files. However, after rebooting the system the tar extraction went to 20+ minutes and an 'ls -R' went to almost 3-minutes.</DIV>
<DIV> </DIV>
<DIV>Are the system startup options in gluster different after a reboot when compared to when the volume is initially created?</DIV>
<DIV>Are the mount options different during system startup when compared to simply using a mount command? </DIV>
<DIV>Do you have any other ideas what could be different from a reboot? Keep in mind that after reboot, I can create a new gluster volume and extract the files many, many times with no slowdown. The slowdown only occurs to an existing gluster volume after a machine reboot. Even after a reboot, if I create a clean volume, there is no slowdown until the next machine reboot.</DIV>
<DIV> </DIV>
<DIV>Note: The log files attached have the "No data available" messages parsed out to reduce the file size. There were an enormous amount of these. One of my colleagues submitted something to the message board about these errors in 3.7.3.</DIV>
<BLOCKQUOTE style="MARGIN-RIGHT: 0px" dir=ltr>
<DIV><FONT size=2><EM>[2015-08-17 17:03:37.270219] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 6643: REMOVEXATTR() /boost_1_57_0/boost/accumulators/accumulators.hpp => -1 (No data available)<BR>[2015-08-17 17:03:37.271004] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 6646: REMOVEXATTR() /boost_1_57_0/boost/accumulators/accumulators.hpp => -1 (No data available)<BR>[2015-08-17 17:03:37.271663] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 6648: REMOVEXATTR() /boost_1_57_0/boost/accumulators/accumulators.hpp => -1 (No data available)<BR>[2015-08-17 17:03:37.274273] W [fuse-bridge.c:1230:fuse_err_cbk] 0-glusterfs-fuse: 6662: REMOVEXATTR() /boost_1_57_0/boost/accumulators/accumulators_fwd.hpp => -1 (No data available)</EM></FONT></DIV></BLOCKQUOTE>
<DIV> </DIV>
<UL>
<LI>System details: Scientific Linux 7.1, XFS, the latest gluster (3.7.3), and the simplest volume possible (single brick without replication, distribution, etc):</LI></UL>
<BLOCKQUOTE style="MARGIN-RIGHT: 0px" dir=ltr>
<BLOCKQUOTE style="MARGIN-RIGHT: 0px" dir=ltr>
<DIV><EM><FONT size=2>[root@gfstest ~]# cat /etc/redhat-release <BR>Scientific Linux release 7.1 (Nitrogen)<BR>[root@gfstest ~]# uname -a<BR>Linux gfstest.corvidtec.com 3.10.0-229.11.1.el7.x86_64 #1 SMP Wed Aug 5 14:37:37 CDT 2015 x86_64 x86_64 x86_64 GNU/Linux<BR>[root@gfstest ~]# gluster volume info testbrick01<BR> <BR>Volume Name: testbrick01<BR>Type: Distribute<BR>Volume ID: 4a003c5c-caef-4838-b62e-ac5d574aadcf<BR>Status: Started<BR>Number of Bricks: 1<BR>Transport-type: tcp<BR>Bricks:<BR>Brick1: gfstest.corvidtec.com:/data/brick01/testbrick01<BR>Options Reconfigured:<BR>performance.readdir-ahead: on</FONT></EM></DIV></BLOCKQUOTE></BLOCKQUOTE>
<BLOCKQUOTE style="MARGIN-RIGHT: 0px" dir=ltr>
<BLOCKQUOTE style="MARGIN-RIGHT: 0px" dir=ltr>
<DIV><FONT size=2><EM>[root@gfstest ~]# xfs_info /data/brick01<BR>meta-data=/dev/sdb1 isize=512 agcount=22, agsize=268435455 blks<BR> = sectsz=512 attr=2, projid32bit=1<BR> = crc=0 finobt=0<BR>data = bsize=4096 blocks=5859966208, imaxpct=5<BR> = sunit=0 swidth=0 blks<BR>naming =version 2 bsize=4096 ascii-ci=0 ftype=0<BR>log =internal bsize=4096 blocks=521728, version=2<BR> = sectsz=512 sunit=0 blks, lazy-count=1<BR>realtime =none extsz=4096 blocks=0, rtextents=0 </EM></FONT></DIV><EM><FONT size=2></FONT></EM>
<DIV><BR></DIV></BLOCKQUOTE></BLOCKQUOTE>
<DIV> </DIV>
<DIV>David</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>------ Original Message ------</DIV>
<DIV>From: "Shyam" <<A href="mailto:srangana@redhat.com">srangana@redhat.com</A>></DIV>
<DIV>To: "David Robinson" <<A href="mailto:drobinson@corvidtec.com">drobinson@corvidtec.com</A>></DIV>
<DIV>Sent: 8/3/2015 5:30:15 PM</DIV>
<DIV>Subject: Re: [Red Hat Bugzilla] Your Outstanding Requests</DIV>
<DIV> </DIV>
<UL class=cite2>
<UL class=cite>
<LI>Hi David, </LI>
<LI> </LI>
<LI>Not much. The perf results, on viewing in our local machines still had some symbols missing. </LI>
<LI> </LI>
<LI>We also had a release in between that got over last week. So things are a bit calmer now. </LI>
<LI> </LI>
<LI>I am thinking of running this test again internally as there was a point where I was able to see this issue in house, so that I can show it first hand to the file system folks. I have talked to the perf. engineer regarding the same, and we are working towards a free slot when we can test this, should be towards the end of this week or beginning of the next, as the systems are tied up in some benchmarking runs for the release. </LI>
<LI> </LI>
<LI>Regards, </LI>
<LI>Shyam </LI>
<LI></LI>
<LI></LI></UL></UL></DIV></BODY></HTML>