<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p><br>
</p>
<br>
<div class="moz-cite-prefix">On 09/03/2016 09:34 PM, Benjamin
Kingston wrote:<br>
</div>
<blockquote
cite="mid:CAHA1JWLuUJPN-5tEAgqj=NkLoPM4dx2cHLO-J-JZmop3Kgfk2Q@mail.gmail.com"
type="cite">
<div dir="ltr">Hello Rafi,
<div><br>
</div>
<div>Thanks for the reply please see my answers below</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sat, Sep 3, 2016 at 12:07 AM,
Mohammed Rafi K C <span dir="ltr"><<a
moz-do-not-send="true" href="mailto:rkavunga@redhat.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:rkavunga@redhat.com">rkavunga@redhat.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>Hi Benjamin,</p>
<p><br>
</p>
<p>Can you tell us more about your work-load like the
file size</p>
</div>
</blockquote>
<div>Files are a range from 10GB test file generated from
/dev/urandom, many 100MB folder separated files to smaller
images and text files</div>
<div>This is a lab so typically there is no load, in the
case of my testing this issue the only file being accessed
was the 10GB test file. </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>, size of both hot and cold storage</p>
</div>
</blockquote>
<div>Hot storage is a 315GB portion of a 512GB SSD</div>
<div>Cold storage is a replica made up of 3 bricks on two
nodes. totaling 17TB</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>how the files are created (after attaching the tier
or before attaching the tier)</p>
</div>
</blockquote>
<div>I've experienced performance issues with files created
before hot attachment, 10GB test file, and copying files
to the volume after attachment (100MB files)</div>
</div>
</div>
</div>
</blockquote>
<br>
Files created before attaching hot tier will be present on hot brick
until it gets heated and migrated completely. During this time
interval we won't get the benefit of hot storage since the files are
served from cold brick.<br>
<br>
<br>
For smallfiles (order of kbs), We are experiencing some performance
issue mostly with EC (disperse ) volume.<br>
<br>
Follow up questions,<br>
What is the volume type ? and version of glusterfs ?<br>
<br>
Does reread gives an equal performance when it hits the server ?<br>
<br>
Have you done any rename for those files ?<br>
<br>
Regards<br>
Rafi KC<br>
<br>
<br>
<blockquote
cite="mid:CAHA1JWLuUJPN-5tEAgqj=NkLoPM4dx2cHLO-J-JZmop3Kgfk2Q@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p>, how long have you been using the tier volume, etc.</p>
</div>
</blockquote>
<div>At the moment I am not using the hot tier after
discovering the performance loss. I originally turned on
after learning of the feature a month or so ago. I feel
like I may have actually had this issue since then because
I have been troubleshooting a network throughalput issue
since then that was contributing to a 4MB/s write to the
volume. I recently resolved that issue and the 4MB/s write
was still observed until I removed the hot tier.</div>
<div><br>
</div>
<div>Unfortunate timing on both issues.</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<p><br>
</p>
<p>Regards</p>
<p>Rafi KC<br>
</p>
<div>
<div class="h5">
<p><br>
</p>
<p><br>
</p>
<br>
<div>On 09/03/2016 09:16 AM, Benjamin Kingston
wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div class="h5">
<div dir="ltr">
<div class="gmail_quote">
<div dir="ltr">Hello all,
<div><br>
</div>
<div>I've discovered an issue in my lab that
went unnoticed until recently, or just
came about with the latest Centos release.</div>
<div><br>
</div>
<div>When the SSD hot tier is enabled read
from the volume is 2MB/s, after detaching
AND committing, read of the same file is
at 150MB/s to /dev/null</div>
<div><br>
</div>
<div>If I copy the file to the hot bricks
directly the write is 150MB/s and read is
500MB/s on the first read, and then 4GB/s
on the subsequent reads (filesystem RAM
caching) to /dev/null</div>
<div><br>
</div>
<div>Just enabling tier storage takes the
performance to ~10-20IOPS and 2-10MB/s
even for local node mounted volume.</div>
<div><br>
</div>
<div>I don't see any major log issues and I
detached and did a fix-layout, but it
persists when re-enabling the tier.</div>
<div><br>
</div>
</div>
</div>
<br>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>______________________________<wbr>_________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-users</a></pre>
</blockquote>
</div>
</blockquote></div>
</div></div>
</blockquote>
</body></html>