<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
/bin/ls just asks the filesystem for a list of files in this
directory.<br>
<br>
/bin/ls -F or --color or -l or anything else that requires some sort
of decoration will do a fstat for each file in order to find out the
mode, ownership, type, etc. in order to determine how to decorate
that filename. Each fstat starts with a lookup() and that lookup is
sent to each replica and must wait for a response. That response
includes data that is used to determine if a file needs healed,
insuring that you are getting accurate data for that fstat (lookup()
is also called for an open so this also will ensure you don't get
stale data). This is something you don't need to do with
non-replicated-over-network block devices.<br>
<br>
Because of the advantage you get replicating across systems,
glusterfs is going to be slower than nfs or anything that doesn't do
that. It's a cost that's paid for resiliency. It's up to you to
engineer your system to balance your needs with the tools that are
available.<br>
<br>
<div class="moz-cite-prefix">On 04/27/2015 03:17 PM, David Robinson
wrote:<br>
</div>
<blockquote
cite="mid:em624d8b07-28ab-49f5-8361-1e6f975f9501@dfrobins-vaio"
type="cite">
<style id="eMClientCss">
blockquote.cite { margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px; border-left: 1px solid #cccccc }
blockquote.cite2 {margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px; border-left: 1px solid #cccccc; margin-top: 3px; padding-top: 0px; }
.plain pre, .plain tt { font-family: monospace; font-size: 100%; font-weight: normal; font-style: normal; white-space: pre-wrap; }
a img { border: 0px; }body {font-family: Times New Roman;font-size: 12pt;}
.plain pre, .plain tt {font-family: Times New Roman;font-size: 12pt;}</style>
<div>Do you think this issue is related to the one seen when you
have 'ls' aliased to 'ls -F' or 'ls --color=auto'? </div>
<div>I included a snippet from a previous email that I had sent to
the gluster devels (see below).</div>
<div> </div>
<div>David</div>
<div> </div>
<div> </div>
<div><em><font size="2">> My code developers were moved over to
the gluster 3.6.1 system and were<br>
> struggling to use it due to extremely poor performance.
The issue was<br>
> when you went into a directory for the first time, the
system would hang<br>
> for 5-10 seconds before letting you list the contents
of the<br>
> directories. This was worse for directories with larger
numbers of<br>
> files (approx. 200 files). I noticed that this only
happened for<br>
> certain users and ended up tracing it out to alias
settings for the "ls"<br>
> command.<br>
><br>
> 'ls -F'<br>
> or<br>
> alias ls='ls --color=auto' #.... Default alias setting
for bash<br>
><br>
> Without these settings, the ls on a new directory takes
less than a<br>
> second. With either of these alias settings, it can
take 5-10 seconds<br>
> which makes code development extremely difficult. Note
that after you<br>
> do an ls in a directory once, you can repeat it without
the severe lag.<br>
> I assume it is caching this information.<br>
<br>
Ahhh yeah. I'm pretty sure this one is a known problem, with
the root cause<br>
being that some options for ls cause it to do a stat against
every file in<br>
the directory (which then has to reach out to every server,
for each file,<br>
to find out which one has the latest info to report back).</font></em></div>
<div> </div>
<div>------ Original Message ------</div>
<div>From: "Ben Turner" <<a moz-do-not-send="true"
href="mailto:bturner@redhat.com">bturner@redhat.com</a>></div>
<div>To: "David Robinson" <<a moz-do-not-send="true"
href="mailto:drobinson@corvidtec.com">drobinson@corvidtec.com</a>></div>
<div>Cc: "Ernie Dunbar" <<a moz-do-not-send="true"
href="mailto:maillist@lightspeed.ca">maillist@lightspeed.ca</a>>;
"Gluster Users" <<a moz-do-not-send="true"
href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>></div>
<div>Sent: 4/27/2015 5:56:02 PM</div>
<div>Subject: Re: Re[2]: [Gluster-users] Disastrous performance
with rsync to mounted Gluster volume.</div>
<div> </div>
<div id="xa442b4ecd81243f8ba9a2c8331f55ca6">
<blockquote class="cite2" type="cite">
<div>----- Original Message -----</div>
<blockquote class="cite" type="cite">
<div> From: "David Robinson" <<a moz-do-not-send="true"
href="mailto:david.robinson@corvidtec.com">david.robinson@corvidtec.com</a>></div>
<div> To: "Ben Turner" <<a moz-do-not-send="true"
href="mailto:bturner@redhat.com">bturner@redhat.com</a>>,
"Ernie Dunbar" <<a moz-do-not-send="true"
href="mailto:maillist@lightspeed.ca">maillist@lightspeed.ca</a>></div>
<div> Cc: "Gluster Users" <<a moz-do-not-send="true"
href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>></div>
<div> Sent: Monday, April 27, 2015 5:21:08 PM</div>
<div> Subject: Re[2]: [Gluster-users] Disastrous performance
with rsync to mounted Gluster volume.</div>
<div> </div>
<div> I am also having a terrible time with rsync and
gluster. The vast</div>
<div> majority of my time is spent figuring out what to
sync... This sync</div>
<div> takes 17-hours even though very little data is being
transferred.</div>
<div> </div>
<div> sent 120,523 bytes received 74,485,191,265 bytes
1,210,720.02</div>
<div> bytes/sec</div>
<div> total size is 27,589,660,889,910 speedup is 370.40</div>
<div> </div>
</blockquote>
<div> </div>
<div>Maybe we could try something to confirm / deny my theory.
What about asking rsync to ignore anything that could
differ between bricks in a replicated pair. A couple
options I see are:</div>
<div> </div>
<div>--size-only means that rsync will skip files that match
in size, even if the timestamps differ. This means it will
synchronise less files than the default behaviour. It will
miss any file with changes that don't affect the overall
file size.</div>
<div> </div>
<div>--ignore-times means that rsync will checksum every file,
even if the timestamps and file sizes match. This means it
will synchronise more files than the default behaviour. It
will include changes to files even where the file size is
the same and the modification date/time has been reset to
the original value (resetting the date/time is unlikely to
be done in practise, but it could happen).</div>
<div> </div>
<div>These may also help, but it looks more to be for
recovering from brick failures:</div>
<div> </div>
<div><a moz-do-not-send="true"
href="http://blog.gluster.org/category/rsync/">http://blog.gluster.org/category/rsync/</a></div>
<div><a moz-do-not-send="true"
href="https://mjanja.ch/2014/07/parallelizing-rsync/?utm_source=rss&utm_medium=rss&utm_campaign=parallelizing-rsync#sync_brick">https://mjanja.ch/2014/07/parallelizing-rsync/?utm_source=rss&utm_medium=rss&utm_campaign=parallelizing-rsync#sync_brick</a></div>
<div> </div>
<div>I'll try some stuff in the lab and see if I can come up
with RCA or something that helps.</div>
<div> </div>
<div>-b</div>
<div> </div>
<blockquote class="cite" type="cite">
<div> </div>
<div> ------ Original Message ------</div>
<div> From: "Ben Turner" <<a moz-do-not-send="true"
href="mailto:bturner@redhat.com">bturner@redhat.com</a>></div>
<div> To: "Ernie Dunbar" <<a moz-do-not-send="true"
href="mailto:maillist@lightspeed.ca">maillist@lightspeed.ca</a>></div>
<div> Cc: "Gluster Users" <<a moz-do-not-send="true"
href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>></div>
<div> Sent: 4/27/2015 4:52:35 PM</div>
<div> Subject: Re: [Gluster-users] Disastrous performance
with rsync to</div>
<div> mounted Gluster volume.</div>
<div> </div>
<div> >----- Original Message -----</div>
<div> >> From: "Ernie Dunbar" <<a
moz-do-not-send="true"
href="mailto:maillist@lightspeed.ca">maillist@lightspeed.ca</a>></div>
<div> >> To: "Gluster Users" <<a
moz-do-not-send="true"
href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>></div>
<div> >> Sent: Monday, April 27, 2015 4:24:56 PM</div>
<div> >> Subject: Re: [Gluster-users] Disastrous
performance with rsync to</div>
<div> >>mounted Gluster volume.</div>
<div> >></div>
<div> >> On 2015-04-24 11:43, Joe Julian wrote:</div>
<div> >></div>
<div> >> >> This should get you where you need
to be. Before you start to</div>
<div> >>migrate</div>
<div> >> >> the data maybe do a couple DDs and
send me the output so we can</div>
<div> >>get an</div>
<div> >> >> idea of how your cluster performs:</div>
<div> >> >></div>
<div> >> >> time `dd if=/dev/zero
of=<gluster-mount>/myfile bs=1024k</div>
<div> >>count=1000;</div>
<div> >> >> sync`</div>
<div> >> >> echo 3 >
/proc/sys/vm/drop_caches</div>
<div> >> >> dd if=<gluster mount>
of=/dev/null bs=1024k count=1000</div>
<div> >> >></div>
<div> >> >> If you are using gigabit and
glusterfs mounts with replica 2 you</div>
<div> >> >> should get ~55 MB / sec writes and
~110 MB / sec reads. With NFS</div>
<div> >>you</div>
<div> >> >> will take a bit of a hit since NFS
doesnt know where files live</div>
<div> >>like</div>
<div> >> >> glusterfs does.</div>
<div> >></div>
<div> >> After copying our data and doing a couple of
very slow rsyncs, I did</div>
<div> >> your speed test and came back with these
results:</div>
<div> >></div>
<div> >> 1048576 bytes (1.0 MB) copied, 0.0307951 s,
34.1 MB/s</div>
<div> >> root@backup:/home/webmailbak# dd
if=/dev/zero of=/mnt/testfile</div>
<div> >> count=1024 bs=1024; sync</div>
<div> >> 1024+0 records in</div>
<div> >> 1024+0 records out</div>
<div> >> 1048576 bytes (1.0 MB) copied, 0.0298592 s,
35.1 MB/s</div>
<div> >> root@backup:/home/webmailbak# dd
if=/dev/zero of=/mnt/testfile</div>
<div> >> count=1024 bs=1024; sync</div>
<div> >> 1024+0 records in</div>
<div> >> 1024+0 records out</div>
<div> >> 1048576 bytes (1.0 MB) copied, 0.0501495 s,
20.9 MB/s</div>
<div> >> root@backup:/home/webmailbak# echo 3 >
/proc/sys/vm/drop_caches</div>
<div> >> root@backup:/home/webmailbak# # dd
if=/mnt/testfile of=/dev/null</div>
<div> >> bs=1024k count=1000</div>
<div> >> 1+0 records in</div>
<div> >> 1+0 records out</div>
<div> >> 1048576 bytes (1.0 MB) copied, 0.0124498 s,
84.2 MB/s</div>
<div> >></div>
<div> >></div>
<div> >> Keep in mind that this is an NFS share over
the network.</div>
<div> >></div>
<div> >> I've also noticed that if I increase the
count of those writes, the</div>
<div> >> transfer speed increases as well:</div>
<div> >></div>
<div> >> 2097152 bytes (2.1 MB) copied, 0.036291 s,
57.8 MB/s</div>
<div> >> root@backup:/home/webmailbak# dd
if=/dev/zero of=/mnt/testfile</div>
<div> >> count=2048 bs=1024; sync</div>
<div> >> 2048+0 records in</div>
<div> >> 2048+0 records out</div>
<div> >> 2097152 bytes (2.1 MB) copied, 0.0362724 s,
57.8 MB/s</div>
<div> >> root@backup:/home/webmailbak# dd
if=/dev/zero of=/mnt/testfile</div>
<div> >> count=2048 bs=1024; sync</div>
<div> >> 2048+0 records in</div>
<div> >> 2048+0 records out</div>
<div> >> 2097152 bytes (2.1 MB) copied, 0.0360319 s,
58.2 MB/s</div>
<div> >> root@backup:/home/webmailbak# dd
if=/dev/zero of=/mnt/testfile</div>
<div> >> count=10240 bs=1024; sync</div>
<div> >> 10240+0 records in</div>
<div> >> 10240+0 records out</div>
<div> >> 10485760 bytes (10 MB) copied, 0.127219 s,
82.4 MB/s</div>
<div> >> root@backup:/home/webmailbak# dd
if=/dev/zero of=/mnt/testfile</div>
<div> >> count=10240 bs=1024; sync</div>
<div> >> 10240+0 records in</div>
<div> >> 10240+0 records out</div>
<div> >> 10485760 bytes (10 MB) copied, 0.128671 s,
81.5 MB/s</div>
<div> ></div>
<div> >This is correct, there is overhead that happens
with small files and</div>
<div> >the smaller the file the less throughput you get.
That said, since</div>
<div> >files are smaller you should get more files /
second but less MB /</div>
<div> >second. I have found that when you go under 16k
changing files size</div>
<div> >doesn't matter, you will get the same number of
16k files / second as</div>
<div> >you do 1 k files.</div>
<div> ></div>
<div> >></div>
<div> >></div>
<div> >> However, the biggest stumbling block for
rsync seems to be changes to</div>
<div> >> directories. I'm unsure about what exactly
it's doing (probably</div>
<div> >>changing</div>
<div> >> last access times?) but these minor writes
seem to take a very long</div>
<div> >>time</div>
<div> >> when normally they would not. Actual file
copies (as in the very</div>
<div> >>files</div>
<div> >> that are actually new within those same
directories) appear to take</div>
<div> >> quite a lot less time than the directory
updates.</div>
<div> ></div>
<div> >Dragons be here! Access time is not kept in sync
across the</div>
<div> >replicas(IIRC, someone correct me if I am wrong!)
and each time a dir</div>
<div> >is read from a different brick I bet the access
time is different.</div>
<div> ></div>
<div> >></div>
<div> >> For example:</div>
<div> >></div>
<div> >> # time rsync -av --inplace --whole-file
--ignore-existing</div>
<div> >>--delete-after</div>
<div> >> gromm/* /mnt/gromm/</div>
<div> >> building file list ... done</div>
<div> >> Maildir/ ## This part
takes a long time.</div>
<div> >> Maildir/.INBOX.Trash/</div>
<div> >> Maildir/.INBOX.Trash/cur/</div>
<div> >></div>
<div> >>Maildir/.INBOX.Trash/cur/1429836077.H817602P21531.pop.lightspeed.ca:2,S</div>
<div> >> Maildir/.INBOX.Trash/tmp/ ## The
previous three lines took</div>
<div> >>nearly</div>
<div> >> no time at all.</div>
<div> >> Maildir/cur/ ## This
takes a long time.</div>
<div> >>
Maildir/cur/1430160436.H952679P13870.pop.lightspeed.ca:2,S</div>
<div> >> Maildir/new/</div>
<div> >> Maildir/tmp/ ## The
previous lines again take no</div>
<div> >>time</div>
<div> >> at all.</div>
<div> >> deleting
Maildir/cur/1429836077.H817602P21531.pop.lightspeed.ca:2,S</div>
<div> >> ## This delete did take a while.</div>
<div> >> sent 1327634 bytes received 75 bytes
59009.29 bytes/sec</div>
<div> >> total size is 624491648 speedup is 470.35</div>
<div> >></div>
<div> >> real 0m26.110s</div>
<div> >> user 0m0.140s</div>
<div> >> sys 0m1.596s</div>
<div> >></div>
<div> >></div>
<div> >> So, rsync reports that it wrote 1327634
bytes at 59 kBytes/sec, and</div>
<div> >>the</div>
<div> >> whole operation took 26 seconds. To write 2
files that were around</div>
<div> >>20-30</div>
<div> >> kBytes each and delete 1.</div>
<div> >></div>
<div> >> The last rsync took around 56 minutes, when
normally such an rsync</div>
<div> >>would</div>
<div> >> have taken 5-10 minutes, writing over the
network via ssh.</div>
<div> ></div>
<div> >It may have something to do with the access times
not being in sync</div>
<div> >across replicated pairs. Maybe some has
experience with this / could</div>
<div> >this be tripping up rsync?</div>
<div> ></div>
<div> >-b</div>
<div> ></div>
<div> >>
_______________________________________________</div>
<div> >> Gluster-users mailing list</div>
<div> >> <a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a></div>
<div> >> <a moz-do-not-send="true"
href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></div>
<div> >></div>
<div> >_______________________________________________</div>
<div> >Gluster-users mailing list</div>
<div> ><a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a></div>
<div> ><a moz-do-not-send="true"
href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></div>
<div> </div>
<div> </div>
</blockquote>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>