<html><body><div style="font-family: garamond,new york,times,serif; font-size: 12pt; color: #000000"><div><br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Kyle Harris" <kyle.harris98@gmail.com><br><b>To: </b>"Pranith Kumar Karampuri" <pkarampu@redhat.com><br><b>Cc: </b>"Lindsay Mathieson" <lindsay.mathieson@gmail.com>, "Krutika Dhananjay" <kdhananj@redhat.com>, gluster-users@gluster.org, "Paul Cuzner" <pcuzner@redhat.com><br><b>Sent: </b>Friday, January 15, 2016 4:46:43 AM<br><b>Subject: </b>Re: [Gluster-users] High I/O And Processor Utilization<br><div><br></div><div dir="ltr"><p class="MsoNormal">I thought I might take a minute to update everyone on this
situation. I rebuilt the glusterFS using
a shard size of 256MB and then imported all VMs back on to the cluster. I rebuilt it from scratch rather than just
doing an export/import on the data so I could start everything fresh. I wish now I would have used 512MB but
unfortunately I didn’t see that suggestion in time. Anyway, the
good news is the system load has greatly decreased. The systems are all now in a usable state.</p><p class="MsoNormal">The bad news is that I am still seeing a bunch of heals and
not sure why. Because of that, I am also
still seeing the drives slow down from over 110 MB/sec without Gluster running on
them to ~ 25 MB/sec with bricks running on the drives. So it seems to me there is still an issue
here.</p></div></blockquote><div><br></div><div>Kyle,<br></div><div>Could you share</div><div>1) the output of `gluster volume info <VOLNAME>`<br></div><div>2) Could you share the client/mount logs and also the glustershd.log files?<br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><p class="MsoNormal">Also as fate would have it, I am having issues due to a bug
in the Gluster NFS implementation on this version (3.7) that necessitates the
need to set nfs.acl to off so also hoping for a fix for that soon. I think this is the bug but not sure: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1238318" target="_blank"><a>https://bugzilla.redhat.com/show_bug.cgi?id=1238318</a></a></p></div></blockquote><div><br></div><div>CC'ing Niels and Soumya wrt the NFS related issue.<br></div><div><br></div><div>-Krutika<br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><p class="MsoNormal"><br data-mce-bogus="1"></p><p class="MsoNormal">So to sum up, things are working but the performance leaves
much to be desired due mostly I suspect due to all the heals taking place.</p><p class="MsoNormal"><br></p><p class="MsoNormal">- Kyle</p><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 11, 2016 at 9:32 PM, Pranith Kumar Karampuri <span dir="ltr"><<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br><br>
On 01/12/2016 08:52 AM, Lindsay Mathieson wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 11/01/16 15:37, Krutika Dhananjay wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Kyle,<br><br>
Based on the testing we have done from our end, we've found that 512MB is a good number that is neither too big nor too small,<br>
and provides good performance both on the IO side and with respect to self-heal.<br></blockquote><br><br>
Hi Krutika, I experimented a lot with different chunk sizes, didn't find all that much difference between 4MB and 1GB<br><br>
But benchmarks are tricky things - I used Crystal Diskmark inside a VM, which is probably not the best assessment. And two of the bricks on my replica 3 are very slow, just test drives, not production. So I guess that would effevt things :)<br><br>
These are my current setting - what do you use?<br><br>
Volume Name: datastore1<br>
Type: Replicate<br>
Volume ID: 1261175d-64e1-48b1-9158-c32802cc09f0<br>
Status: Started<br>
Number of Bricks: 1 x 3 = 3<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: vnb.proxmox.softlog:/vmdata/datastore1<br>
Brick2: vng.proxmox.softlog:/vmdata/datastore1<br>
Brick3: vna.proxmox.softlog:/vmdata/datastore1<br>
Options Reconfigured:<br>
network.remote-dio: enable<br>
cluster.eager-lock: enable<br>
performance.io-cache: off<br>
performance.read-ahead: off<br>
performance.quick-read: off<br>
performance.stat-prefetch: off<br>
performance.strict-write-ordering: on<br>
performance.write-behind: off<br>
nfs.enable-ino32: off<br>
nfs.addr-namelookup: off<br>
nfs.disable: on<br>
performance.cache-refresh-timeout: 4<br>
performance.io-thread-count: 32<br>
cluster.server-quorum-type: server<br>
cluster.quorum-type: auto<br>
client.event-threads: 4<br>
server.event-threads: 4<br>
cluster.self-heal-window-size: 256<br>
features.shard-block-size: 512MB<br>
features.shard: on<br>
performance.readdir-ahead: off<br><br><br></blockquote></div></div>
Most of these tests are done by Paul Cuzner (CCed).<span class="HOEnZb"><span style="color: #888888;" data-mce-style="color: #888888;" color="#888888"><br>
<br>
Pranith<br>
</span></span></blockquote></div><br><div class="gmail_signature"><div dir="ltr"><div><div><br></div></div></div></div></div></div></blockquote><div><br></div></div></body></html>