<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi,<div class=""><br class=""></div><div class="">Yes of course:</div><div class=""><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">[root@lucifer ~]# pdsh -w cl-storage[1,3] du -s /export/brick_home/brick*/amyloid_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">cl-storage1: 1608522280<span class="Apple-tab-span" style="white-space:pre"> </span>/export/brick_home/brick1/amyloid_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">cl-storage3: 1619630616<span class="Apple-tab-span" style="white-space:pre"> </span>/export/brick_home/brick1/amyloid_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">cl-storage1: 1614057836<span class="Apple-tab-span" style="white-space:pre"> </span>/export/brick_home/brick2/amyloid_team</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">cl-storage3: 1602653808<span class="Apple-tab-span" style="white-space:pre"> </span>/export/brick_home/brick2/amyloid_team</div><div class=""><br class=""></div><div class="">The sum is: 6444864540 (around 6.4-6.5TB) while the quota list displays 7.7TB.</div><div class="">So, the mistake is roughly 1.2-1.3TB, in other words around 16% -which is too huge, no?</div><div class=""><br class=""></div><div class="">In addition, since the quota is exceeded, i note a lot of files like following:</div><div class=""><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">[root@lucifer ~]# pdsh -w cl-storage[1,3] "cd /export/brick_home/brick2/amyloid_team/tarus/project/ab1-40-x1_sen304-x2_inh3-x2/remd_charmm22star_scripts/; ls -ail remd_100.sh 2> /dev/null" 2>/dev/null</div><div style="margin: 0px; font-size: 11px; font-family: Menlo; color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">cl-storage3: 133325688 ---------T 2 tarus amyloid_team 0 16 févr. 10:20 remd_100.sh</div></div><div class="">note the ’T’ at the end of perms and the file size to 0B.</div><div class=""><br class=""></div><div class="">And, yesterday, some files were duplicated but not anymore...</div><div class=""></div><div class=""><br class=""></div><div class="">The worst is, previously, all these files were OK. In other words, exceeding quota made file or content deletions or corruptions… What can I do to prevent to situation for the futur -because I guess i cannot do something to rollback this situation now, right?</div><div class=""><br class=""></div><div class="">Geoffrey</div><div class="">------------------------------------------------------</div><div apple-content-edited="true" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Geoffrey Letessier<br class="">Responsable informatique & ingénieur système<br class="">UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br class="">Institut de Biologie Physico-Chimique<br class="">13, rue Pierre et Marie Curie - 75005 Paris<br class="">Tel: 01 58 41 50 93 - eMail: <a href="mailto:geoffrey.letessier@ibpc.fr" class="">geoffrey.letessier@ibpc.fr</a></div>
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">Le 9 juin 2015 à 09:01, Vijaikumar M <<a href="mailto:vmallika@redhat.com" class="">vmallika@redhat.com</a>> a écrit :</div><br class="Apple-interchange-newline"><div class="">
<meta content="text/html; charset=windows-1252" http-equiv="Content-Type" class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On Monday 08 June 2015 07:11 PM,
Geoffrey Letessier wrote:<br class="">
</div>
<blockquote cite="mid:C9DA5829-C3CB-43FD-BF65-A60A6E1F14E6@cnrs.fr" type="cite" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
In addition, i notice a very big difference between the sum of DU
on each brick and « quota list » display, as you can read below:
<div class="">
<div style="margin: 0px; font-size: 11px; font-family: Menlo;
color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">[root@lucifer ~]# pdsh -w cl-storage[1,3] du -sh
/export/brick_home/brick*/amyloid_team</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;
color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">cl-storage1: 1,6T<span class="Apple-tab-span" style="white-space:pre"> </span>/export/brick_home/brick1/amyloid_team</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;
color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">cl-storage3: 1,6T<span class="Apple-tab-span" style="white-space:pre"> </span>/export/brick_home/brick1/amyloid_team</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;
color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">cl-storage1: 1,6T<span class="Apple-tab-span" style="white-space:pre"> </span>/export/brick_home/brick2/amyloid_team</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;
color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">cl-storage3: 1,6T<span class="Apple-tab-span" style="white-space:pre"> </span>/export/brick_home/brick2/amyloid_team</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;
color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">[root@lucifer ~]# gluster volume quota vol_home list
/amyloid_team</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;
color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class=""> Path Hard-limit
Soft-limit Used Available</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;
color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">--------------------------------------------------------------------------------</div>
<div style="margin: 0px; font-size: 11px; font-family: Menlo;
color: rgb(255, 255, 255); background-color: rgb(0, 0, 0);" class="">/amyloid_team 9.0TB
90% 7.8TB 1.2TB</div>
<div class=""><br class="">
</div>
<div class="">As you can notice, the sum of all bricks gives me
roughly 6.4TB and « quota list » around 7.8TB; so there is a
difference of 1.4TB i’m not able to explain… Do you have any
idea?</div>
<div class=""><br class="">
</div>
</div>
</blockquote>
<br class="">
<tt class="">There were few issues when quota accounting </tt><tt class="">the size,
we have fixed </tt><tt class="">some of these issues in 3.7</tt><tt class=""><br class="">
</tt><tt class="">'df -h</tt><tt class="">' will round off the values, can you please
provide the output of 'df' without -h option?</tt><tt class=""><br class="">
</tt><br class="">
<br class="">
<br class="">
<br class="">
<blockquote cite="mid:C9DA5829-C3CB-43FD-BF65-A60A6E1F14E6@cnrs.fr" type="cite" class="">
<div class="">
<div class="">Thanks,</div>
<div class="">Geoffrey</div>
<div apple-content-edited="true" class="">
<div style="letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">------------------------------------------------------<br class="">
Geoffrey Letessier<br class="">
Responsable informatique & ingénieur système<br class="">
UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br class="">
Institut de Biologie Physico-Chimique<br class="">
13, rue Pierre et Marie Curie - 75005 Paris<br class="">
Tel: 01 58 41 50 93 - eMail: <a moz-do-not-send="true" href="mailto:geoffrey.letessier@ibpc.fr" class="">geoffrey.letessier@ibpc.fr</a></div>
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">Le 8 juin 2015 à 14:30, Geoffrey Letessier
<<a moz-do-not-send="true" href="mailto:geoffrey.letessier@cnrs.fr" class="">geoffrey.letessier@cnrs.fr</a>>
a écrit :</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode:
space; -webkit-line-break: after-white-space;" class="">Hello,
<div class=""><br class="">
</div>
<div class="">Concerning the 3.5.3 version of GlusterFS,
I met this morning a strange issue writing file when
quota is exceeded. </div>
<div class=""><br class="">
</div>
<div class="">One person of my lab, whose her quota is
exceeded (but she didn’t know about) try to modify a
file but, because of exceeded quota, she was unable to
and decided to exit VI. Now, her file is empty/blank
as you can read below:</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<tt class="">we suspect 'vi' might have created tmp file before writing to a
file. We are working on re-creating this problem and will update
you on the same.<br class="">
<br class="">
<br class="">
</tt><tt class="">
</tt>
<blockquote cite="mid:C9DA5829-C3CB-43FD-BF65-A60A6E1F14E6@cnrs.fr" type="cite" class="">
<div class="">
<div class="">
<blockquote type="cite" class="">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode:
space; -webkit-line-break: after-white-space;" class="">
<div class="">
<div style="margin: 0px; font-size: 11px; font-family:
Menlo; color: rgb(255, 255, 255); background-color:
rgb(0, 0, 0);" class="">pdsh@lucifer: cl-storage3:
ssh exited with exit code 2</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo; color: rgb(255, 255, 255); background-color:
rgb(0, 0, 0);" class="">cl-storage1: ---------T 2
tarus amyloid_team 0 19 févr. 12:34
/export/brick_home/brick1/amyloid_team/tarus/project/ab1-40-x1_sen304-x2_inh3-x2/remd_charmm22star_scripts/remd_115.sh</div>
<div style="margin: 0px; font-size: 11px; font-family:
Menlo; color: rgb(255, 255, 255); background-color:
rgb(0, 0, 0);" class="">cl-storage1: -rwxrw-r-- 2
tarus amyloid_team 0 8 juin 12:38
/export/brick_home/brick2/amyloid_team/tarus/project/ab1-40-x1_sen304-x2_inh3-x2/remd_charmm22star_scripts/remd_115.sh</div>
<div class=""><br class="">
</div>
<div class="">In addition, i dont understand why, my
volume being a distributed volume inside replica
(cl-storage[1,3] is replicated only on
cl-storage[2,4]), i have 2 « same » files (complete
path) in 2 different bricks (as you can read above).</div>
<div class=""><br class="">
</div>
<div class="">Thanks by advance for your help and
clarification.</div>
<div class="">Geoffrey</div>
<div apple-content-edited="true" class="">
<div style="letter-spacing: normal; orphans: auto;
text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows:
auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px; word-wrap:
break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;" class="">------------------------------------------------------<br class="">
Geoffrey Letessier<br class="">
Responsable informatique & ingénieur système<br class="">
UPR 9080 - CNRS - Laboratoire de
Biochimie Théorique<br class="">
Institut de Biologie Physico-Chimique<br class="">
13, rue Pierre et Marie Curie - 75005 Paris<br class="">
Tel: 01 58 41 50 93 - eMail: <a moz-do-not-send="true" href="mailto:geoffrey.letessier@ibpc.fr" class="">geoffrey.letessier@ibpc.fr</a></div>
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">Le 2 juin 2015 à 23:45, Geoffrey
Letessier <<a moz-do-not-send="true" href="mailto:geoffrey.letessier@cnrs.fr" class="">geoffrey.letessier@cnrs.fr</a>> a
écrit :</div>
<br class="Apple-interchange-newline">
<div class="">
<meta http-equiv="Content-Type" content="text/html; charset=windows-1252" class="">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space; -webkit-line-break:
after-white-space;" class="">Hi Ben,
<div class=""><br class="">
</div>
<div class="">I just check my messages log
files, both on client and server, and I dont
find any hung task you notice on yours.. </div>
<div class=""><br class="">
</div>
<div class="">As you can read below, i dont
note the performance issue in a simple DD
but I think my issue is concerning a set of
small files (tens of thousands nay more)…</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">[root@nisus test]# ddt -t 10g
/mnt/test/</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">Writing to /mnt/test/ddt.8362
... syncing ... done.</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">sleeping 10 seconds ... done.</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">Reading from /mnt/test/ddt.8362
... done.</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">10240MiB KiB/s CPU%</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">Write 114770 4</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">Read 40675 4</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class=""><br class="">
</div>
</div>
<div class="">for info: /mnt/test concerns
the single v2 GlFS volume</div>
<div class=""><br class="">
</div>
<div apple-content-edited="true" class="">
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">[root@nisus test]# ddt -t 10g
/mnt/fhgfs/</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">Writing to /mnt/fhgfs/ddt.8380
... syncing ... done.</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">sleeping 10 seconds ... done.</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">Reading from
/mnt/fhgfs/ddt.8380 ... done.</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">10240MiB KiB/s CPU%</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">Write 102591 1</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">Read 98079 2</div>
<div class=""><br class="">
</div>
</div>
</div>
<div class="">Do you have a idea how to
tune/optimize performance settings? and/or
TCP settings (MTU, etc.)?</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">| | UNTAR | DU
| FIND | TAR | RM |</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">| single | ~3m45s |
~43s | ~47s | ~3m10s | ~3m15s |</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">| replicated | ~5m10s |
~59s | ~1m6s | ~1m19s | ~1m49s |</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">| distributed | ~4m18s |
~41s | ~57s | ~2m24s | ~1m38s |</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">| dist-repl | ~8m18s |
~1m4s | ~1m11s | ~1m24s | ~2m40s |</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">| native FS | ~11s |
~4s | ~2s | ~56s | ~10s |</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">| BeeGFS | ~3m43s |
~15s | ~3s | ~1m33s | ~46s |</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">| single (v2) | ~3m6s |
~14s | ~32s | ~1m2s | ~44s |</div>
<div style="margin: 0px; font-size: 10px;
font-family: Menlo; color: rgb(255, 255,
255); background-color: rgb(0, 0, 0);" class="">---------------------------------------------------------------</div>
</div>
<div class="">for info: </div>
<div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>-BeeGFS
is a distributed FS (4 bricks, 2 bricks
per server and 2 servers)</div>
<div class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>-
single (v2): simple gluster volume with
default settings</div>
</div>
<div class=""><br class="">
</div>
<div class="">I also note I obtain the same
tar/untar performance issue with
FhGFS/BeeGFS but the rest (DU, FIND, RM)
looks like to be OK.</div>
<div class=""><br class="">
</div>
<div class="">Thank you very much for your
reply and help.</div>
<div class="">Geoffrey<br class="">
<div apple-content-edited="true" class="">
<div style="letter-spacing: normal;
orphans: auto; text-align: start;
text-indent: 0px; text-transform: none;
white-space: normal; widows: auto;
word-spacing: 0px;
-webkit-text-stroke-width: 0px;
word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;" class="">-----------------------------------------------<br class="">
Geoffrey Letessier<br class="">
<br class="">
Responsable informatique & ingénieur
système<br class="">
CNRS - UPR 9080 - Laboratoire
de Biochimie Théorique<br class="">
Institut de Biologie Physico-Chimique<br class="">
13, rue Pierre et Marie Curie -
75005 Paris<br class="">
Tel: 01 58 41 50 93 - eMail: <a moz-do-not-send="true" href="mailto:geoffrey.letessier@cnrs.fr" class="">geoffrey.letessier@cnrs.fr</a></div>
</div>
<br class="">
<div class="">
<div class="">Le 2 juin 2015 à 21:53, Ben
Turner <<a moz-do-not-send="true" href="mailto:bturner@redhat.com" class="">bturner@redhat.com</a>> a
écrit :</div>
<br class="Apple-interchange-newline">
<blockquote type="cite" class="">
<div style="font-size: 12px; font-style:
normal; font-variant: normal;
font-weight: normal; letter-spacing:
normal; line-height: normal; orphans:
auto; text-align: start; text-indent:
0px; text-transform: none;
white-space: normal; widows: auto;
word-spacing: 0px;
-webkit-text-stroke-width: 0px;" class="">I am seeing problems on 3.7
as well. Can you check
/var/log/messages on both the clients
and servers for hung tasks like:<br class="">
<br class="">
Jun 2 15:23:14 gqac006 kernel: "echo
0 >
/proc/sys/kernel/hung_task_timeout_secs"
disables this message.<br class="">
Jun 2 15:23:14 gqac006 kernel: iozone
D 0000000000000001 0 21999
1 0x00000080<br class="">
Jun 2 15:23:14 gqac006 kernel:
ffff880611321cc8 0000000000000082
ffff880611321c18 ffffffffa027236e<br class="">
Jun 2 15:23:14 gqac006 kernel:
ffff880611321c48 ffffffffa0272c10
ffff88052bd1e040 ffff880611321c78<br class="">
Jun 2 15:23:14 gqac006 kernel:
ffff88052bd1e0f0 ffff88062080c7a0
ffff880625addaf8 ffff880611321fd8<br class="">
Jun 2 15:23:14 gqac006 kernel: Call
Trace:<br class="">
Jun 2 15:23:14 gqac006 kernel:
[<ffffffffa027236e>] ?
rpc_make_runnable+0x7e/0x80 [sunrpc]<br class="">
Jun 2 15:23:14 gqac006 kernel:
[<ffffffffa0272c10>] ?
rpc_execute+0x50/0xa0 [sunrpc]<br class="">
Jun 2 15:23:14 gqac006 kernel:
[<ffffffff810aaa21>] ?
ktime_get_ts+0xb1/0xf0<br class="">
Jun 2 15:23:14 gqac006 kernel:
[<ffffffff811242d0>] ?
sync_page+0x0/0x50<br class="">
Jun 2 15:23:14 gqac006 kernel:
[<ffffffff8152a1b3>]
io_schedule+0x73/0xc0<br class="">
Jun 2 15:23:14 gqac006 kernel:
[<ffffffff8112430d>]
sync_page+0x3d/0x50<br class="">
Jun 2 15:23:14 gqac006 kernel:
[<ffffffff8152ac7f>]
__wait_on_bit+0x5f/0x90<br class="">
Jun 2 15:23:14 gqac006 kernel:
[<ffffffff81124543>]
wait_on_page_bit+0x73/0x80<br class="">
Jun 2 15:23:14 gqac006 kernel:
[<ffffffff8109eb80>] ?
wake_bit_function+0x0/0x50<br class="">
Jun 2 15:23:14 gqac006 kernel:
[<ffffffff8113a525>] ?
pagevec_lookup_tag+0x25/0x40<br class="">
Jun 2 15:23:14 gqac006 kernel:
[<ffffffff8112496b>]
wait_on_page_writeback_range+0xfb/0x190<br class="">
Jun 2 15:23:14 gqac006 kernel:
[<ffffffff81124b38>]
filemap_write_and_wait_range+0x78/0x90<br class="">
Jun 2 15:23:14 gqac006 kernel:
[<ffffffff811c07ce>]
vfs_fsync_range+0x7e/0x100<br class="">
Jun 2 15:23:14 gqac006 kernel:
[<ffffffff811c08bd>]
vfs_fsync+0x1d/0x20<br class="">
Jun 2 15:23:14 gqac006 kernel:
[<ffffffff811c08fe>]
do_fsync+0x3e/0x60<br class="">
Jun 2 15:23:14 gqac006 kernel:
[<ffffffff811c0950>]
sys_fsync+0x10/0x20<br class="">
Jun 2 15:23:14 gqac006 kernel:
[<ffffffff8100b072>]
system_call_fastpath+0x16/0x1b<br class="">
<br class="">
Do you see a perf problem with just a
simple DD or do you need a more
complex workload to hit the issue? I
think I saw an issue with metadata
performance that I am trying to run
down, let me know if you can see the
problem with simple DD reads / writes
or if we need to do some sort of dir /
metadata access as well.<br class="">
<br class="">
-b<br class="">
<br class="">
----- Original Message -----<br class="">
<blockquote type="cite" class="">From:
"Geoffrey Letessier" <<a moz-do-not-send="true" href="mailto:geoffrey.letessier@cnrs.fr" class="">geoffrey.letessier@cnrs.fr</a>><br class="">
To: "Pranith Kumar Karampuri" <<a moz-do-not-send="true" href="mailto:pkarampu@redhat.com" class="">pkarampu@redhat.com</a>><br class="">
Cc:<span class="Apple-converted-space"> </span><a moz-do-not-send="true" href="mailto:gluster-users@gluster.org" class="">gluster-users@gluster.org</a><br class="">
Sent: Tuesday, June 2, 2015 8:09:04
AM<br class="">
Subject: Re: [Gluster-users]
GlusterFS 3.7 - slow/poor
performances<br class="">
<br class="">
Hi Pranith,<br class="">
<br class="">
I’m sorry but I cannot bring you any
comparison because comparison will
be<br class="">
distorted by the fact in my HPC
cluster in production the network
technology<br class="">
is InfiniBand QDR and my volumes are
quite different (brick in RAID6<br class="">
(12x2TB), 2 bricks per server and 4
servers into my pool)<br class="">
<br class="">
Concerning your demand, in
attachments you can find all
expected results<br class="">
hoping it can help you to solve this
serious performance issue (maybe I
need<br class="">
play with glusterfs parameters?).<br class="">
<br class="">
Thank you very much by advance,<br class="">
Geoffrey<br class="">
------------------------------------------------------<br class="">
Geoffrey Letessier<br class="">
Responsable informatique &
ingénieur système<br class="">
UPR 9080 - CNRS - Laboratoire de
Biochimie Théorique<br class="">
Institut de Biologie
Physico-Chimique<br class="">
13, rue Pierre et Marie Curie -
75005 Paris<br class="">
Tel: 01 58 41 50 93 - eMail: <a moz-do-not-send="true" href="mailto:geoffrey.letessier@ibpc.fr" class="">geoffrey.letessier@ibpc.fr</a><br class="">
<br class="">
<br class="">
<br class="">
<br class="">
Le 2 juin 2015 à 10:09, Pranith
Kumar Karampuri < <a moz-do-not-send="true" href="mailto:pkarampu@redhat.com" class="">pkarampu@redhat.com</a>
> a<br class="">
écrit :<br class="">
<br class="">
hi Geoffrey,<br class="">
Since you are saying it happens on
all types of volumes, lets do the<br class="">
following:<br class="">
1) Create a dist-repl volume<br class="">
2) Set the options etc you need.<br class="">
3) enable gluster volume profile
using "gluster volume profile
<volname><br class="">
start"<br class="">
4) run the work load<br class="">
5) give output of "gluster volume
profile <volname> info"<br class="">
<br class="">
Repeat the steps above on new and
old version you are comparing this
with.<br class="">
That should give us insight into
what could be causing the slowness.<br class="">
<br class="">
Pranith<br class="">
On 06/02/2015 03:22 AM, Geoffrey
Letessier wrote:<br class="">
<br class="">
<br class="">
Dear all,<br class="">
<br class="">
I have a crash test cluster where
i’ve tested the new version of
GlusterFS<br class="">
(v3.7) before upgrading my HPC
cluster in production.<br class="">
But… all my tests show me very very
low performances.<br class="">
<br class="">
For my benches, as you can read
below, I do some actions (untar, du,
find,<br class="">
tar, rm) with linux kernel sources,
dropping cache, each on distributed,<br class="">
replicated, distributed-replicated,
single (single brick) volumes and
the<br class="">
native FS of one brick.<br class="">
<br class="">
# time (echo 3 >
/proc/sys/vm/drop_caches; tar xJf
~/linux-4.1-rc5.tar.xz;<br class="">
sync; echo 3 >
/proc/sys/vm/drop_caches)<br class="">
# time (echo 3 >
/proc/sys/vm/drop_caches; du -sh
linux-4.1-rc5/; echo 3 ><br class="">
/proc/sys/vm/drop_caches)<br class="">
# time (echo 3 >
/proc/sys/vm/drop_caches; find
linux-4.1-rc5/|wc -l; echo 3<br class="">
<blockquote type="cite" class="">/proc/sys/vm/drop_caches)<br class="">
</blockquote>
# time (echo 3 >
/proc/sys/vm/drop_caches; tar czf
linux-4.1-rc5.tgz<br class="">
linux-4.1-rc5/; echo 3 >
/proc/sys/vm/drop_caches)<br class="">
# time (echo 3 >
/proc/sys/vm/drop_caches; rm -rf
linux-4.1-rc5.tgz<br class="">
linux-4.1-rc5/; echo 3 >
/proc/sys/vm/drop_caches)<br class="">
<br class="">
And here are the process times:<br class="">
<br class="">
---------------------------------------------------------------<br class="">
| | UNTAR | DU | FIND | TAR | RM |<br class="">
---------------------------------------------------------------<br class="">
| single | ~3m45s | ~43s | ~47s |
~3m10s | ~3m15s |<br class="">
---------------------------------------------------------------<br class="">
| replicated | ~5m10s | ~59s | ~1m6s
| ~1m19s | ~1m49s |<br class="">
---------------------------------------------------------------<br class="">
| distributed | ~4m18s | ~41s | ~57s
| ~2m24s | ~1m38s |<br class="">
---------------------------------------------------------------<br class="">
| dist-repl | ~8m18s | ~1m4s |
~1m11s | ~1m24s | ~2m40s |<br class="">
---------------------------------------------------------------<br class="">
| native FS | ~11s | ~4s | ~2s |
~56s | ~10s |<br class="">
---------------------------------------------------------------<br class="">
<br class="">
I get the same results, whether with
default configurations with custom<br class="">
configurations.<br class="">
<br class="">
if I look at the side of the ifstat
command, I can note my IO write
processes<br class="">
never exceed 3MBs...<br class="">
<br class="">
EXT4 native FS seems to be faster
(roughly 15-20% but no more) than
XFS one<br class="">
<br class="">
My [test] storage cluster config is
composed by 2 identical servers
(biCPU<br class="">
Intel Xeon X5355, 8GB of RAM, 2x2TB
HDD (no-RAID) and Gb ethernet)<br class="">
<br class="">
My volume settings:<br class="">
single: 1server 1 brick<br class="">
replicated: 2 servers 1 brick each<br class="">
distributed: 2 servers 2 bricks each<br class="">
dist-repl: 2 bricks in the same
server and replica 2<br class="">
<br class="">
All seems to be OK in gluster status
command line.<br class="">
<br class="">
Do you have an idea why I obtain so
bad results?<br class="">
Thanks in advance.<br class="">
Geoffrey<br class="">
-----------------------------------------------<br class="">
Geoffrey Letessier<br class="">
<br class="">
Responsable informatique &
ingénieur système<br class="">
CNRS - UPR 9080 - Laboratoire de
Biochimie Théorique<br class="">
Institut de Biologie
Physico-Chimique<br class="">
13, rue Pierre et Marie Curie -
75005 Paris<br class="">
Tel: 01 58 41 50 93 - eMail: <a moz-do-not-send="true" href="mailto:geoffrey.letessier@cnrs.fr" class="">geoffrey.letessier@cnrs.fr</a><br class="">
<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
Gluster-users mailing list <a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">
<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" class="">http://www.gluster.org/mailman/listinfo/gluster-users</a><br class="">
<br class="">
<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
Gluster-users mailing list<br class="">
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">
<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" class="">http://www.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre wrap="" class="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br class="">
</div>
</div></blockquote></div><br class=""></div></body></html>