<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<tt>Hi Geoffrey,</tt><tt><br>
</tt><tt><br>
</tt><tt>Sorry for the delayed response. We will look at the log
files you provided in your previous email and update you the
workaround as soon as possible.</tt><tt><br>
</tt><tt><br>
</tt><tt>Thanks,</tt><tt><br>
</tt><tt>Vijay</tt><br>
<br>
<br>
<div class="moz-cite-prefix">On Thursday 11 June 2015 05:43 PM,
Geoffrey Letessier wrote:<br>
</div>
<blockquote cite="mid:06A0EDE3-2CD8-45B1-95DF-4BD0AB7FBECA@cnrs.fr"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
Hi Vijay,
<div class=""><br class="">
</div>
<div class="">Could you take a time to take a look at this? I
found only one thing about my issues in Red Hat bugzilla (<a
moz-do-not-send="true"
href="https://bugzilla.redhat.com/show_bug.cgi?id=917901"
class="">https://bugzilla.redhat.com/show_bug.cgi?id=917901</a>)
But, my storage & computing clusters are still in production
now and I wonder if I should warn my community about of a needed
production break or can I apply a fix during production? (i.e.
without updating my GlusterFS version on my storage cluster).</div>
<div class=""><br class="">
</div>
<div class="">Thanks in advance,</div>
<div class="">Geoffrey<br class="">
<div apple-content-edited="true" class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal;
orphans: auto; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: auto;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
word-wrap: break-word; -webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;" class="">------------------------------------------------------<br
class="">
Geoffrey Letessier<br class="">
Responsable informatique & ingénieur système<br class="">
UPR 9080 - CNRS - Laboratoire de Biochimie Théorique<br
class="">
Institut de Biologie Physico-Chimique<br class="">
13, rue Pierre et Marie Curie - 75005 Paris<br class="">
Tel: 01 58 41 50 93 - eMail: <a moz-do-not-send="true"
href="mailto:geoffrey.letessier@ibpc.fr" class="">geoffrey.letessier@ibpc.fr</a></div>
</div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">Le 10 juin 2015 à 06:12, Vijaikumar M <<a
moz-do-not-send="true" href="mailto:vmallika@redhat.com"
class="">vmallika@redhat.com</a>> a écrit :</div>
<br class="Apple-interchange-newline">
<div class="">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type" class="">
<div text="#000000" bgcolor="#FFFFFF" class=""> <tt
class="">Hi Geoffrey,</tt><tt class=""><br class="">
</tt><tt class=""><br class="">
</tt><tt class="">grep for 'ERROR' from the log file,
and only these lines would be sufficient.</tt><tt
class=""><br class="">
</tt><tt class=""><br class="">
</tt><tt class="">Thanks,</tt><tt class=""><br class="">
</tt><tt class="">Vijay</tt><br class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On Wednesday 10 June 2015
04:38 AM, Geoffrey Letessier wrote:<br class="">
</div>
<blockquote
cite="mid:C6B2CB51-209E-454E-BF5A-08323449338A@cnrs.fr"
type="cite" class="">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252" class="">
Hello Vijay,
<div class=""><br class="">
</div>
<div class="">Quota-verify is still running since a
couple of hours (more than 10) and each output file
sizes (4 files because 4 bricks per replica) are
very huge: around 800MB per file in the first server
and 5GB per file in the second one. Do your still
want these? How can I send it to you?</div>
<div class=""><br class="">
</div>
<div class="">Nice night (in France)</div>
<div class="">Geoffrey<br class="">
<div apple-content-edited="true" class="">
------------------------------------------------------<br
class="">
Geoffrey Letessier<br class="">
Responsable informatique & ingénieur système<br
class="">
UPR 9080 - CNRS - Laboratoire de Biochimie
Théorique<br class="">
Institut de Biologie Physico-Chimique<br class="">
13, rue Pierre et Marie Curie - 75005 Paris<br
class="">
Tel: 01 58 41 50 93 - eMail: <a
moz-do-not-send="true"
href="mailto:geoffrey.letessier@ibpc.fr"
class="">geoffrey.letessier@ibpc.fr</a> </div>
<br class="">
<div class="">
<div class="">Le 9 juin 2015 à 12:46, Vijaikumar M
<<a moz-do-not-send="true"
href="mailto:vmallika@redhat.com" class="">vmallika@redhat.com</a>>
a écrit :</div>
<br class="Apple-interchange-newline">
<blockquote type="cite" class="">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type" class="">
<div bgcolor="#FFFFFF" text="#000000" class="">
<tt class="">Hi Geoffrey,</tt><tt class=""><br
class="">
</tt><tt class=""><br class="">
</tt><tt class="">The file content deletion is
because of 'vi editor' behaviour of
truncating the file when writing the updated
content.</tt><tt class=""><br class="">
</tt><tt class=""><br class="">
</tt><tt class="">Regarding quota size/usage
problem, can you please execute the script
attached on each brick and provide us the
output generated, this will help us analyse
why quota list is showing wrong-size.</tt><tt
class=""><br class="">
</tt>
<meta http-equiv="content-type"
content="text/html; charset=windows-1252"
class="">
<tt class="">The script basically crawls the
directory given as argument.</tt><tt
class=""><br class="">
</tt><tt class="">It collects quota "contri"
and "size" extended attribute and also
"block size" from stat call.</tt><tt
class=""><br class="">
</tt><tt class=""><br class="">
</tt><tt class="">Usage:</tt><tt class=""><br
class="">
</tt><tt class=""><br class="">
</tt><tt class="">./quota-verify -b
<brick_path> | tee brick_name.log</tt><tt
class=""><br class="">
</tt><tt class=""><br class="">
</tt><tt class=""><br class="">
</tt><tt class="">Thanks,</tt><tt class=""><br
class="">
</tt><tt class="">Vijay</tt><tt class=""><br
class="">
</tt><tt class=""><br class="">
</tt><br class="">
<br class="">
<div class="moz-cite-prefix">On Tuesday 09
June 2015 03:45 PM, Vijaikumar M wrote:<br
class="">
</div>
<blockquote
cite="mid:5576BCBB.7020502@redhat.com"
type="cite" class="">
<meta content="text/html;
charset=windows-1252"
http-equiv="Content-Type" class="">
<br class="">
<br class="">
<div class="moz-cite-prefix">On Tuesday 09
June 2015 03:40 PM, Geoffrey Letessier
wrote:<br class="">
</div>
<blockquote
cite="mid:EF13613C-0262-4E26-A146-920746BBB363@cnrs.fr"
type="cite" class="">
<meta http-equiv="Content-Type"
content="text/html;
charset=windows-1252" class="">
Hi Vijay,
<div class=""><br class="">
</div>
<div class="">Thanks for having replied.</div>
<div class=""><br class="">
</div>
<div class="">Unfortunately, i check each
bricks on my stockage pool and dont find
any backup file.. damage!</div>
</blockquote>
<tt class=""><br class="">
</tt><tt class="">Please check backup file
on client machine where the file was
edited and on the home dir of a user (this
is the user login </tt><tt class="">used
to edit a file</tt><tt class="">).</tt><tt
class=""><br class="">
</tt><tt class=""><br class="">
</tt><tt class="">Thanks,</tt><tt class=""><br
class="">
</tt><tt class="">Vijay</tt><br class="">
<br class="">
<br class="">
<blockquote
cite="mid:EF13613C-0262-4E26-A146-920746BBB363@cnrs.fr"
type="cite" class="">
<div class=""><br class="">
</div>
<div class="">Thank you again!</div>
<div class="">Good luck and see you,</div>
<div class="">Geoffrey<br class="">
<div apple-content-edited="true"
class="">
<div style="letter-spacing: normal;
orphans: auto; text-align: start;
text-indent: 0px; text-transform:
none; white-space: normal; widows:
auto; word-spacing: 0px;
-webkit-text-stroke-width: 0px;
word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break:
after-white-space;" class="">------------------------------------------------------<br
class="">
Geoffrey Letessier<br class="">
Responsable informatique &
ingénieur système<br class="">
UPR 9080 - CNRS - Laboratoire de
Biochimie Théorique<br class="">
Institut de Biologie
Physico-Chimique<br class="">
13, rue Pierre et Marie Curie -
75005 Paris<br class="">
Tel: 01 58 41 50 93 - eMail: <a
moz-do-not-send="true"
href="mailto:geoffrey.letessier@ibpc.fr"
class="">geoffrey.letessier@ibpc.fr</a></div>
</div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">Le 9 juin 2015 à
10:05, Vijaikumar M <<a
moz-do-not-send="true"
href="mailto:vmallika@redhat.com"
class="">vmallika@redhat.com</a>>
a écrit :</div>
<br
class="Apple-interchange-newline">
<div class="">
<meta content="text/html;
charset=windows-1252"
http-equiv="Content-Type"
class="">
<div bgcolor="#FFFFFF"
text="#000000" class=""> <br
class="">
<br class="">
<div class="moz-cite-prefix">On
Tuesday 09 June 2015 01:08 PM,
Geoffrey Letessier wrote:<br
class="">
</div>
<blockquote
cite="mid:A2852C15-BF63-421F-A730-EA03D983CFBE@cnrs.fr"
type="cite" class="">
<meta
http-equiv="Content-Type"
content="text/html;
charset=windows-1252"
class="">
Hi,
<div class=""><br class="">
</div>
<div class="">Yes of course:</div>
<div class="">
<div style="margin: 0px;
font-size: 11px;
font-family: Menlo; color:
rgb(255, 255, 255);
background-color: rgb(0,
0, 0);" class="">[root@lucifer
~]# pdsh -w
cl-storage[1,3] du -s
/export/brick_home/brick*/amyloid_team</div>
<div style="margin: 0px;
font-size: 11px;
font-family: Menlo; color:
rgb(255, 255, 255);
background-color: rgb(0,
0, 0);" class="">cl-storage1:
1608522280<span
class="Apple-tab-span"
style="white-space:pre">
</span>/export/brick_home/brick1/amyloid_team</div>
<div style="margin: 0px;
font-size: 11px;
font-family: Menlo; color:
rgb(255, 255, 255);
background-color: rgb(0,
0, 0);" class="">cl-storage3:
1619630616<span
class="Apple-tab-span"
style="white-space:pre">
</span>/export/brick_home/brick1/amyloid_team</div>
<div style="margin: 0px;
font-size: 11px;
font-family: Menlo; color:
rgb(255, 255, 255);
background-color: rgb(0,
0, 0);" class="">cl-storage1:
1614057836<span
class="Apple-tab-span"
style="white-space:pre">
</span>/export/brick_home/brick2/amyloid_team</div>
<div style="margin: 0px;
font-size: 11px;
font-family: Menlo; color:
rgb(255, 255, 255);
background-color: rgb(0,
0, 0);" class="">cl-storage3:
1602653808<span
class="Apple-tab-span"
style="white-space:pre">
</span>/export/brick_home/brick2/amyloid_team</div>
<div class=""><br class="">
</div>
<div class="">The sum
is: 6444864540 (around
6.4-6.5TB) while the quota
list displays 7.7TB.</div>
<div class="">So, the
mistake is roughly
1.2-1.3TB, in other words
around 16% -which is too
huge, no?</div>
<div class=""><br class="">
</div>
<div class="">In addition,
since the quota is
exceeded, i note a lot of
files like following:</div>
<div class="">
<div style="margin: 0px;
font-size: 11px;
font-family: Menlo;
color: rgb(255, 255,
255); background-color:
rgb(0, 0, 0);" class="">[root@lucifer
~]# pdsh -w
cl-storage[1,3] "cd
/export/brick_home/brick2/amyloid_team/tarus/project/ab1-40-x1_sen304-x2_inh3-x2/remd_charmm22star_scripts/;
ls -ail remd_100.sh
2> /dev/null"
2>/dev/null</div>
<div style="margin: 0px;
font-size: 11px;
font-family: Menlo;
color: rgb(255, 255,
255); background-color:
rgb(0, 0, 0);" class="">cl-storage3:
133325688 ---------T 2
tarus amyloid_team 0 16
févr. 10:20 remd_100.sh</div>
</div>
<div class="">note the ’T’
at the end of perms and
the file size to 0B.</div>
<div class=""><br class="">
</div>
<div class="">And,
yesterday, some files were
duplicated but not
anymore...</div>
<div class=""><br class="">
</div>
<div class="">The worst is,
previously, all these
files were OK. In other
words, exceeding quota
made file or content
deletions or corruptions…
What can I do to prevent
to situation for the futur
-because I guess i cannot
do something to rollback
this situation now, right?</div>
<div class=""><br class="">
</div>
</div>
</blockquote>
<tt class=""><br class="">
Hi </tt><tt class="">Geoffrey,</tt><tt
class=""><br class="">
</tt><tt class=""><br class="">
</tt><tt class="">I tried
re-creating the problem.<br
class="">
<br class="">
Here is the behaviour of vi
editor.</tt><tt class=""><br
class="">
</tt><tt class="">When a file is
saved in vi editor, it creates
a backup file under home dir
and opens the original file
with 'O_TRUNC' flag and hence
file was truncated.</tt><tt
class=""><br class="">
</tt><tt class=""><br class="">
<br class="">
Here is the strace of vi
editor when it gets 'EDQUOT'
error:</tt><tt class=""><br
class="">
</tt><tt class=""><br class="">
</tt><tt class="">open("hello",
O_WRONLY|O_CREAT|O_TRUNC,
0644) = 3</tt><tt class=""><br
class="">
</tt><tt class="">write(3, "line
one\nline two\n", 18) = 18</tt><tt
class=""><br class="">
</tt><tt class="">fsync(3)
= 0</tt><tt class=""><br
class="">
</tt><tt class="">close(3)
= -1 EDQUOT (Disk quota
exceeded)</tt><tt class=""><br
class="">
</tt><tt class="">chmod("hello",
0100644) = 0</tt><tt
class=""><br class="">
</tt><tt class="">open("/root/hello~",
O_RDONLY) = 3</tt><tt
class=""><br class="">
</tt><b class=""><tt class="">open("hello",
O_WRONLY|O_CREAT|O_TRUNC,
0644) = 7</tt></b><tt
class=""><br class="">
</tt><tt class="">read(3, "line
one\n", 256) = 9</tt><tt
class=""><br class="">
</tt><tt class="">write(7, "line
one\n", 9) = 9</tt><tt
class=""><br class="">
</tt><tt class="">read(3, "",
256) =
0</tt><tt class=""><br
class="">
</tt><tt class="">close(7)
= -1 EDQUOT (Disk quota
exceeded)</tt><tt class=""><br
class="">
</tt><tt class="">close(3)
= 0</tt><tt class=""><br
class="">
</tt><tt class=""><br class="">
</tt><tt class=""><br class="">
</tt><tt class="">To re-cover
the truncated file, please
find if there are any backup
file 'remd_115.sh~' under '~/'
or on the same dir where this
file exists.</tt><tt class="">
If exists you can copy this
file.<br class="">
</tt><tt class=""><br class="">
</tt><tt class="">Thanks,</tt><tt
class=""><br class="">
</tt><tt class="">Vijay</tt><br
class="">
<br class="">
<br class="">
<blockquote
cite="mid:A2852C15-BF63-421F-A730-EA03D983CFBE@cnrs.fr"
type="cite" class="">
<div class="">
<div class="">Geoffrey</div>
<div class="">------------------------------------------------------</div>
<div
apple-content-edited="true"
class="">
<div
style="letter-spacing:
normal; orphans: auto;
text-align: start;
text-indent: 0px;
text-transform: none;
white-space: normal;
widows: auto;
word-spacing: 0px;
-webkit-text-stroke-width:
0px; word-wrap:
break-word;
-webkit-nbsp-mode:
space;
-webkit-line-break:
after-white-space;"
class="">Geoffrey
Letessier<br class="">
Responsable
informatique &
ingénieur système<br
class="">
UPR 9080 - CNRS -
Laboratoire de
Biochimie Théorique<br
class="">
Institut de Biologie
Physico-Chimique<br
class="">
13, rue Pierre et Marie
Curie - 75005 Paris<br
class="">
Tel: 01 58 41 50 93 -
eMail: <a
moz-do-not-send="true"
href="mailto:geoffrey.letessier@ibpc.fr" class="">geoffrey.letessier@ibpc.fr</a></div>
</div>
<br class="">
<div class="">
<blockquote type="cite"
class="">
<div class="">Le 9 juin
2015 à 09:01,
Vijaikumar M <<a
moz-do-not-send="true"
href="mailto:vmallika@redhat.com" class="">vmallika@redhat.com</a>> a
écrit :</div>
<br
class="Apple-interchange-newline">
<div class="">
<meta
content="text/html;
charset=windows-1252" http-equiv="Content-Type" class="">
<div bgcolor="#FFFFFF"
text="#000000"
class=""> <br
class="">
<br class="">
<div
class="moz-cite-prefix">On
Monday 08 June
2015 07:11 PM,
Geoffrey Letessier
wrote:<br class="">
</div>
<blockquote
cite="mid:C9DA5829-C3CB-43FD-BF65-A60A6E1F14E6@cnrs.fr"
type="cite"
class="">
<meta
http-equiv="Content-Type"
content="text/html;
charset=windows-1252" class="">
In addition, i
notice a very big
difference between
the sum of DU on
each brick and
« quota list »
display, as you
can read below:
<div class="">
<div
style="margin:
0px;
font-size:
11px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">[root@lucifer
~]# pdsh -w
cl-storage[1,3]
du -sh
/export/brick_home/brick*/amyloid_team</div>
<div
style="margin:
0px;
font-size:
11px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">cl-storage1:
1,6T<span
class="Apple-tab-span"
style="white-space:pre"> </span>/export/brick_home/brick1/amyloid_team</div>
<div
style="margin:
0px;
font-size:
11px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">cl-storage3:
1,6T<span
class="Apple-tab-span"
style="white-space:pre"> </span>/export/brick_home/brick1/amyloid_team</div>
<div
style="margin:
0px;
font-size:
11px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">cl-storage1:
1,6T<span
class="Apple-tab-span"
style="white-space:pre"> </span>/export/brick_home/brick2/amyloid_team</div>
<div
style="margin:
0px;
font-size:
11px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">cl-storage3:
1,6T<span
class="Apple-tab-span"
style="white-space:pre"> </span>/export/brick_home/brick2/amyloid_team</div>
<div
style="margin:
0px;
font-size:
11px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">[root@lucifer
~]# gluster
volume quota
vol_home list
/amyloid_team</div>
<div
style="margin:
0px;
font-size:
11px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">
Path
Hard-limit
Soft-limit
Used
Available</div>
<div
style="margin:
0px;
font-size:
11px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">--------------------------------------------------------------------------------</div>
<div
style="margin:
0px;
font-size:
11px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">/amyloid_team
9.0TB
90%
7.8TB 1.2TB</div>
<div class=""><br
class="">
</div>
<div class="">As
you can
notice, the
sum of all
bricks gives
me roughly
6.4TB and
« quota list »
around 7.8TB;
so there is a
difference of
1.4TB i’m not
able to
explain… Do
you have any
idea?</div>
<div class=""><br
class="">
</div>
</div>
</blockquote>
<br class="">
<tt class="">There
were few issues
when quota
accounting </tt><tt
class="">the size,
we have fixed </tt><tt
class="">some of
these issues in
3.7</tt><tt
class=""><br
class="">
</tt><tt class="">'df
-h</tt><tt
class="">' will
round off the
values, can you
please provide the
output of 'df'
without -h option?</tt><tt
class=""><br
class="">
</tt><br class="">
<br class="">
<br class="">
<br class="">
<blockquote
cite="mid:C9DA5829-C3CB-43FD-BF65-A60A6E1F14E6@cnrs.fr"
type="cite"
class="">
<div class="">
<div class="">Thanks,</div>
<div class="">Geoffrey</div>
<div
apple-content-edited="true"
class="">
<div
style="letter-spacing:
normal;
orphans: auto;
text-align:
start;
text-indent:
0px;
text-transform:
none;
white-space:
normal;
widows: auto;
word-spacing:
0px;
-webkit-text-stroke-width:
0px;
word-wrap:
break-word;
-webkit-nbsp-mode:
space;
-webkit-line-break:
after-white-space;" class="">------------------------------------------------------<br
class="">
Geoffrey
Letessier<br
class="">
Responsable
informatique &
ingénieur
système<br
class="">
UPR 9080 -
CNRS -
Laboratoire de
Biochimie Théorique<br class="">
Institut de
Biologie
Physico-Chimique<br
class="">
13, rue Pierre
et Marie Curie
- 75005 Paris<br
class="">
Tel: 01 58 41
50 93 -
eMail: <a
moz-do-not-send="true"
href="mailto:geoffrey.letessier@ibpc.fr" class="">geoffrey.letessier@ibpc.fr</a></div>
</div>
<br class="">
<div class="">
<blockquote
type="cite"
class="">
<div class="">Le
8 juin 2015 à
14:30,
Geoffrey
Letessier <<a
moz-do-not-send="true" href="mailto:geoffrey.letessier@cnrs.fr" class="">geoffrey.letessier@cnrs.fr</a>>
a écrit :</div>
<br
class="Apple-interchange-newline">
<div class="">
<meta
http-equiv="Content-Type"
content="text/html;
charset=windows-1252" class="">
<div
style="word-wrap:
break-word;
-webkit-nbsp-mode:
space;
-webkit-line-break:
after-white-space;" class="">Hello,
<div class=""><br
class="">
</div>
<div class="">Concerning
the 3.5.3
version of
GlusterFS, I
met this
morning a
strange issue
writing file
when quota is
exceeded. </div>
<div class=""><br
class="">
</div>
<div class="">One
person of my
lab, whose her
quota is
exceeded (but
she didn’t
know about)
try to modify
a file but,
because of
exceeded
quota, she was
unable to and
decided to
exit VI. Now,
her file is
empty/blank as
you can read
below:</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<tt class="">we
suspect 'vi' might
have created tmp
file before
writing to a file.
We are working on
re-creating this
problem and will
update you on the
same.<br class="">
<br class="">
<br class="">
</tt><tt class=""> </tt>
<blockquote
cite="mid:C9DA5829-C3CB-43FD-BF65-A60A6E1F14E6@cnrs.fr"
type="cite"
class="">
<div class="">
<div class="">
<blockquote
type="cite"
class="">
<div class="">
<div
style="word-wrap:
break-word;
-webkit-nbsp-mode:
space;
-webkit-line-break:
after-white-space;" class="">
<div class="">
<div
style="margin:
0px;
font-size:
11px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">pdsh@lucifer:
cl-storage3:
ssh exited
with exit code
2</div>
<div
style="margin:
0px;
font-size:
11px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">cl-storage1:
---------T 2
tarus
amyloid_team 0
19 févr. 12:34
/export/brick_home/brick1/amyloid_team/tarus/project/ab1-40-x1_sen304-x2_inh3-x2/remd_charmm22star_scripts/remd_115.sh</div>
<div
style="margin:
0px;
font-size:
11px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">cl-storage1:
-rwxrw-r-- 2
tarus
amyloid_team
0 8 juin
12:38
/export/brick_home/brick2/amyloid_team/tarus/project/ab1-40-x1_sen304-x2_inh3-x2/remd_charmm22star_scripts/remd_115.sh</div>
<div class=""><br
class="">
</div>
<div class="">In
addition, i
dont
understand
why, my volume
being a
distributed
volume inside
replica
(cl-storage[1,3]
is replicated
only on
cl-storage[2,4]),
i have 2
« same » files
(complete
path) in 2
different
bricks (as you
can read
above).</div>
<div class=""><br
class="">
</div>
<div class="">Thanks
by advance for
your help and
clarification.</div>
<div class="">Geoffrey</div>
<div
apple-content-edited="true"
class="">
<div
style="letter-spacing:
normal;
orphans: auto;
text-align:
start;
text-indent:
0px;
text-transform:
none;
white-space:
normal;
widows: auto;
word-spacing:
0px;
-webkit-text-stroke-width:
0px;
word-wrap:
break-word;
-webkit-nbsp-mode:
space;
-webkit-line-break:
after-white-space;" class="">------------------------------------------------------<br
class="">
Geoffrey
Letessier<br
class="">
Responsable
informatique &
ingénieur
système<br
class="">
UPR 9080 -
CNRS -
Laboratoire de
Biochimie Théorique<br class="">
Institut de
Biologie
Physico-Chimique<br
class="">
13, rue Pierre
et Marie Curie
- 75005 Paris<br
class="">
Tel: 01 58 41
50 93 -
eMail: <a
moz-do-not-send="true"
href="mailto:geoffrey.letessier@ibpc.fr" class="">geoffrey.letessier@ibpc.fr</a></div>
</div>
<br class="">
<div class="">
<blockquote
type="cite"
class="">
<div class="">Le
2 juin 2015 à
23:45,
Geoffrey
Letessier <<a
moz-do-not-send="true" href="mailto:geoffrey.letessier@cnrs.fr" class="">geoffrey.letessier@cnrs.fr</a>>
a écrit :</div>
<br
class="Apple-interchange-newline">
<div class="">
<meta
http-equiv="Content-Type"
content="text/html;
charset=windows-1252" class="">
<div
style="word-wrap:
break-word;
-webkit-nbsp-mode:
space;
-webkit-line-break:
after-white-space;" class="">Hi Ben,
<div class=""><br
class="">
</div>
<div class="">I
just check my
messages log
files, both on
client and
server, and I
dont find any
hung task you
notice on
yours.. </div>
<div class=""><br
class="">
</div>
<div class="">As
you can read
below, i dont
note the
performance
issue in a
simple DD but
I think my
issue is
concerning a
set of small
files (tens of
thousands nay
more)…</div>
<div class=""><br
class="">
</div>
<div class="">
<div class="">
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">[root@nisus
test]# ddt -t
10g /mnt/test/</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">Writing
to
/mnt/test/ddt.8362
... syncing
... done.</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">sleeping
10 seconds ...
done.</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">Reading
from
/mnt/test/ddt.8362
... done.</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">10240MiB
KiB/s CPU%</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">Write
114770
4</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">Read
40675
4</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class=""><br
class="">
</div>
</div>
<div class="">for
info:
/mnt/test
concerns the
single v2 GlFS
volume</div>
<div class=""><br
class="">
</div>
<div
apple-content-edited="true"
class="">
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">[root@nisus
test]# ddt -t
10g
/mnt/fhgfs/</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">Writing
to
/mnt/fhgfs/ddt.8380
... syncing
... done.</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">sleeping
10 seconds ...
done.</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">Reading
from
/mnt/fhgfs/ddt.8380
... done.</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">10240MiB
KiB/s CPU%</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">Write
102591
1</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">Read
98079
2</div>
<div class=""><br
class="">
</div>
</div>
</div>
<div class="">Do
you have a
idea how to
tune/optimize
performance
settings?
and/or TCP
settings (MTU,
etc.)?</div>
<div class=""><br
class="">
</div>
<div class="">
<div class="">
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">---------------------------------------------------------------</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">|
|
UNTAR | DU
| FIND |
TAR | RM
|</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">---------------------------------------------------------------</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">|
single |
~3m45s |
~43s |
~47s |
~3m10s |
~3m15s |</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">---------------------------------------------------------------</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">|
replicated |
~5m10s |
~59s |
~1m6s |
~1m19s |
~1m49s |</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">---------------------------------------------------------------</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">|
distributed |
~4m18s |
~41s |
~57s |
~2m24s |
~1m38s |</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">---------------------------------------------------------------</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">|
dist-repl |
~8m18s |
~1m4s
| ~1m11s |
~1m24s |
~2m40s |</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">---------------------------------------------------------------</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">|
native FS |
~11s |
~4s |
~2s |
~56s | ~10s
|</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">---------------------------------------------------------------</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">|
BeeGFS |
~3m43s |
~15s |
~3s |
~1m33s |
~46s |</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">---------------------------------------------------------------</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">|
single (v2) |
~3m6s |
~14s |
~32s |
~1m2s | ~44s
|</div>
<div
style="margin:
0px;
font-size:
10px;
font-family:
Menlo; color:
rgb(255, 255,
255);
background-color:
rgb(0, 0, 0);"
class="">---------------------------------------------------------------</div>
</div>
<div class="">for
info: </div>
<div class=""><span
class="Apple-tab-span" style="white-space: pre;"> </span>-BeeGFS is a
distributed FS
(4 bricks, 2
bricks per
server and 2
servers)</div>
<div class=""><span
class="Apple-tab-span" style="white-space: pre;"> </span>- single (v2):
simple gluster
volume with
default
settings</div>
</div>
<div class=""><br
class="">
</div>
<div class="">I
also note I
obtain the
same tar/untar
performance
issue with
FhGFS/BeeGFS
but the rest
(DU, FIND, RM)
looks like to
be OK.</div>
<div class=""><br
class="">
</div>
<div class="">Thank
you very much
for your reply
and help.</div>
<div class="">Geoffrey<br
class="">
<div
apple-content-edited="true"
class="">
<div
style="letter-spacing:
normal;
orphans: auto;
text-align:
start;
text-indent:
0px;
text-transform:
none;
white-space:
normal;
widows: auto;
word-spacing:
0px;
-webkit-text-stroke-width:
0px;
word-wrap:
break-word;
-webkit-nbsp-mode:
space;
-webkit-line-break:
after-white-space;" class="">-----------------------------------------------<br
class="">
Geoffrey
Letessier<br
class="">
<br class="">
Responsable
informatique
&
ingénieur
système<br
class="">
CNRS - UPR
9080 -
Laboratoire
de Biochimie
Théorique<br
class="">
Institut de
Biologie
Physico-Chimique<br
class="">
13, rue Pierre
et Marie Curie
- 75005 Paris<br
class="">
Tel: 01 58 41
50 93 -
eMail: <a
moz-do-not-send="true"
href="mailto:geoffrey.letessier@cnrs.fr" class="">geoffrey.letessier@cnrs.fr</a></div>
</div>
<br class="">
<div class="">
<div class="">Le
2 juin 2015 à
21:53, Ben
Turner <<a
moz-do-not-send="true" href="mailto:bturner@redhat.com" class="">bturner@redhat.com</a>>
a écrit :</div>
<br
class="Apple-interchange-newline">
<blockquote
type="cite"
class="">
<div
style="font-size:
12px;
font-style:
normal;
font-variant:
normal;
font-weight:
normal;
letter-spacing:
normal;
line-height:
normal;
orphans: auto;
text-align:
start;
text-indent:
0px;
text-transform:
none;
white-space:
normal;
widows: auto;
word-spacing:
0px;
-webkit-text-stroke-width:
0px;" class="">I
am seeing
problems on
3.7 as well.
Can you check
/var/log/messages
on both the
clients and
servers for
hung tasks
like:<br
class="">
<br class="">
Jun 2
15:23:14
gqac006
kernel: "echo
0 >
/proc/sys/kernel/hung_task_timeout_secs"
disables this
message.<br
class="">
Jun 2
15:23:14
gqac006
kernel: iozone
D
0000000000000001
0 21999
1
0x00000080<br
class="">
Jun 2
15:23:14
gqac006
kernel:
ffff880611321cc8
0000000000000082
ffff880611321c18
ffffffffa027236e<br class="">
Jun 2
15:23:14
gqac006
kernel:
ffff880611321c48
ffffffffa0272c10
ffff88052bd1e040
ffff880611321c78<br class="">
Jun 2
15:23:14
gqac006
kernel:
ffff88052bd1e0f0
ffff88062080c7a0
ffff880625addaf8
ffff880611321fd8<br class="">
Jun 2
15:23:14
gqac006
kernel: Call
Trace:<br
class="">
Jun 2
15:23:14
gqac006
kernel:
[<ffffffffa027236e>]
?
rpc_make_runnable+0x7e/0x80
[sunrpc]<br
class="">
Jun 2
15:23:14
gqac006
kernel:
[<ffffffffa0272c10>]
?
rpc_execute+0x50/0xa0
[sunrpc]<br
class="">
Jun 2
15:23:14
gqac006
kernel:
[<ffffffff810aaa21>]
?
ktime_get_ts+0xb1/0xf0<br
class="">
Jun 2
15:23:14
gqac006
kernel:
[<ffffffff811242d0>]
?
sync_page+0x0/0x50<br
class="">
Jun 2
15:23:14
gqac006
kernel:
[<ffffffff8152a1b3>]
io_schedule+0x73/0xc0<br class="">
Jun 2
15:23:14
gqac006
kernel:
[<ffffffff8112430d>]
sync_page+0x3d/0x50<br class="">
Jun 2
15:23:14
gqac006
kernel:
[<ffffffff8152ac7f>]
__wait_on_bit+0x5f/0x90<br class="">
Jun 2
15:23:14
gqac006
kernel:
[<ffffffff81124543>]
wait_on_page_bit+0x73/0x80<br class="">
Jun 2
15:23:14
gqac006
kernel:
[<ffffffff8109eb80>]
?
wake_bit_function+0x0/0x50<br
class="">
Jun 2
15:23:14
gqac006
kernel:
[<ffffffff8113a525>]
?
pagevec_lookup_tag+0x25/0x40<br
class="">
Jun 2
15:23:14
gqac006
kernel:
[<ffffffff8112496b>]
wait_on_page_writeback_range+0xfb/0x190<br class="">
Jun 2
15:23:14
gqac006
kernel:
[<ffffffff81124b38>]
filemap_write_and_wait_range+0x78/0x90<br class="">
Jun 2
15:23:14
gqac006
kernel:
[<ffffffff811c07ce>]
vfs_fsync_range+0x7e/0x100<br class="">
Jun 2
15:23:14
gqac006
kernel:
[<ffffffff811c08bd>]
vfs_fsync+0x1d/0x20<br class="">
Jun 2
15:23:14
gqac006
kernel:
[<ffffffff811c08fe>]
do_fsync+0x3e/0x60<br class="">
Jun 2
15:23:14
gqac006
kernel:
[<ffffffff811c0950>]
sys_fsync+0x10/0x20<br class="">
Jun 2
15:23:14
gqac006
kernel:
[<ffffffff8100b072>]
system_call_fastpath+0x16/0x1b<br class="">
<br class="">
Do you see a
perf problem
with just a
simple DD or
do you need a
more complex
workload to
hit the issue?
I think I saw
an issue with
metadata
performance
that I am
trying to run
down, let me
know if you
can see the
problem with
simple DD
reads / writes
or if we need
to do some
sort of dir /
metadata
access as
well.<br
class="">
<br class="">
-b<br class="">
<br class="">
----- Original
Message -----<br
class="">
<blockquote
type="cite"
class="">From:
"Geoffrey
Letessier"
<<a
moz-do-not-send="true"
href="mailto:geoffrey.letessier@cnrs.fr" class="">geoffrey.letessier@cnrs.fr</a>><br
class="">
To: "Pranith
Kumar
Karampuri"
<<a
moz-do-not-send="true"
href="mailto:pkarampu@redhat.com" class="">pkarampu@redhat.com</a>><br
class="">
Cc:<span
class="Apple-converted-space"> </span><a
moz-do-not-send="true" href="mailto:gluster-users@gluster.org" class="">gluster-users@gluster.org</a><br
class="">
Sent: Tuesday,
June 2, 2015
8:09:04 AM<br
class="">
Subject: Re:
[Gluster-users]
GlusterFS 3.7
- slow/poor
performances<br
class="">
<br class="">
Hi Pranith,<br
class="">
<br class="">
I’m sorry but
I cannot bring
you any
comparison
because
comparison
will be<br
class="">
distorted by
the fact in my
HPC cluster in
production the
network
technology<br
class="">
is InfiniBand
QDR and my
volumes are
quite
different
(brick in
RAID6<br
class="">
(12x2TB), 2
bricks per
server and 4
servers into
my pool)<br
class="">
<br class="">
Concerning
your demand,
in attachments
you can find
all expected
results<br
class="">
hoping it can
help you to
solve this
serious
performance
issue (maybe I
need<br
class="">
play with
glusterfs
parameters?).<br
class="">
<br class="">
Thank you very
much by
advance,<br
class="">
Geoffrey<br
class="">
------------------------------------------------------<br class="">
Geoffrey
Letessier<br
class="">
Responsable
informatique
&
ingénieur
système<br
class="">
UPR 9080 -
CNRS -
Laboratoire de
Biochimie
Théorique<br
class="">
Institut de
Biologie
Physico-Chimique<br
class="">
13, rue Pierre
et Marie Curie
- 75005 Paris<br
class="">
Tel: 01 58 41
50 93 - eMail:
<a
moz-do-not-send="true"
href="mailto:geoffrey.letessier@ibpc.fr" class="">geoffrey.letessier@ibpc.fr</a><br
class="">
<br class="">
<br class="">
<br class="">
<br class="">
Le 2 juin 2015
à 10:09,
Pranith Kumar
Karampuri <
<a
moz-do-not-send="true"
href="mailto:pkarampu@redhat.com" class="">pkarampu@redhat.com</a> >
a<br class="">
écrit :<br
class="">
<br class="">
hi Geoffrey,<br
class="">
Since you are
saying it
happens on all
types of
volumes, lets
do the<br
class="">
following:<br
class="">
1) Create a
dist-repl
volume<br
class="">
2) Set the
options etc
you need.<br
class="">
3) enable
gluster volume
profile using
"gluster
volume profile
<volname><br class="">
start"<br
class="">
4) run the
work load<br
class="">
5) give output
of "gluster
volume profile
<volname>
info"<br
class="">
<br class="">
Repeat the
steps above on
new and old
version you
are comparing
this with.<br
class="">
That should
give us
insight into
what could be
causing the
slowness.<br
class="">
<br class="">
Pranith<br
class="">
On 06/02/2015
03:22 AM,
Geoffrey
Letessier
wrote:<br
class="">
<br class="">
<br class="">
Dear all,<br
class="">
<br class="">
I have a crash
test cluster
where i’ve
tested the new
version of
GlusterFS<br
class="">
(v3.7) before
upgrading my
HPC cluster in
production.<br
class="">
But… all my
tests show me
very very low
performances.<br
class="">
<br class="">
For my
benches, as
you can read
below, I do
some actions
(untar, du,
find,<br
class="">
tar, rm) with
linux kernel
sources,
dropping
cache, each on
distributed,<br
class="">
replicated,
distributed-replicated,
single (single
brick) volumes
and the<br
class="">
native FS of
one brick.<br
class="">
<br class="">
# time (echo 3
>
/proc/sys/vm/drop_caches;
tar xJf
~/linux-4.1-rc5.tar.xz;<br
class="">
sync; echo 3
>
/proc/sys/vm/drop_caches)<br
class="">
# time (echo 3
>
/proc/sys/vm/drop_caches;
du -sh
linux-4.1-rc5/;
echo 3 ><br
class="">
/proc/sys/vm/drop_caches)<br class="">
# time (echo 3
>
/proc/sys/vm/drop_caches;
find
linux-4.1-rc5/|wc
-l; echo 3<br
class="">
<blockquote
type="cite"
class="">/proc/sys/vm/drop_caches)<br
class="">
</blockquote>
# time (echo 3
>
/proc/sys/vm/drop_caches;
tar czf
linux-4.1-rc5.tgz<br
class="">
linux-4.1-rc5/;
echo 3 >
/proc/sys/vm/drop_caches)<br
class="">
# time (echo 3
>
/proc/sys/vm/drop_caches;
rm -rf
linux-4.1-rc5.tgz<br
class="">
linux-4.1-rc5/;
echo 3 >
/proc/sys/vm/drop_caches)<br
class="">
<br class="">
And here are
the process
times:<br
class="">
<br class="">
---------------------------------------------------------------<br
class="">
| | UNTAR | DU
| FIND | TAR |
RM |<br
class="">
---------------------------------------------------------------<br
class="">
| single |
~3m45s | ~43s
| ~47s |
~3m10s |
~3m15s |<br
class="">
---------------------------------------------------------------<br
class="">
| replicated |
~5m10s | ~59s
| ~1m6s |
~1m19s |
~1m49s |<br
class="">
---------------------------------------------------------------<br
class="">
| distributed
| ~4m18s |
~41s | ~57s |
~2m24s |
~1m38s |<br
class="">
---------------------------------------------------------------<br
class="">
| dist-repl |
~8m18s | ~1m4s
| ~1m11s |
~1m24s |
~2m40s |<br
class="">
---------------------------------------------------------------<br
class="">
| native FS |
~11s | ~4s |
~2s | ~56s |
~10s |<br
class="">
---------------------------------------------------------------<br
class="">
<br class="">
I get the same
results,
whether with
default
configurations
with custom<br
class="">
configurations.<br class="">
<br class="">
if I look at
the side of
the ifstat
command, I can
note my IO
write
processes<br
class="">
never exceed
3MBs...<br
class="">
<br class="">
EXT4 native FS
seems to be
faster
(roughly
15-20% but no
more) than XFS
one<br
class="">
<br class="">
My [test]
storage
cluster config
is composed by
2 identical
servers (biCPU<br
class="">
Intel Xeon
X5355, 8GB of
RAM, 2x2TB HDD
(no-RAID) and
Gb ethernet)<br
class="">
<br class="">
My volume
settings:<br
class="">
single:
1server 1
brick<br
class="">
replicated: 2
servers 1
brick each<br
class="">
distributed: 2
servers 2
bricks each<br
class="">
dist-repl: 2
bricks in the
same server
and replica 2<br
class="">
<br class="">
All seems to
be OK in
gluster status
command line.<br
class="">
<br class="">
Do you have an
idea why I
obtain so bad
results?<br
class="">
Thanks in
advance.<br
class="">
Geoffrey<br
class="">
-----------------------------------------------<br class="">
Geoffrey
Letessier<br
class="">
<br class="">
Responsable
informatique
&
ingénieur
système<br
class="">
CNRS - UPR
9080 -
Laboratoire de
Biochimie
Théorique<br
class="">
Institut de
Biologie
Physico-Chimique<br
class="">
13, rue Pierre
et Marie Curie
- 75005 Paris<br
class="">
Tel: 01 58 41
50 93 - eMail:
<a
moz-do-not-send="true"
href="mailto:geoffrey.letessier@cnrs.fr" class="">geoffrey.letessier@cnrs.fr</a><br
class="">
<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
Gluster-users
mailing list <a
moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br
class="">
<a
moz-do-not-send="true"
href="http://www.gluster.org/mailman/listinfo/gluster-users" class="">http://www.gluster.org/mailman/listinfo/gluster-users</a><br
class="">
<br class="">
<br class="">
<br class="">
<br class="">
_______________________________________________<br class="">
Gluster-users
mailing list<br
class="">
<a
moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br
class="">
<a
moz-do-not-send="true"
href="http://www.gluster.org/mailman/listinfo/gluster-users" class="">http://www.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
<br class="">
<fieldset
class="mimeAttachmentHeader"></fieldset>
<br class="">
<pre class="" wrap="">_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<br class="">
</blockquote>
<br class="">
</div>
<span class=""><quota-verify.gz></span></blockquote>
</div>
<br class="">
</div>
</blockquote>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
<br>
</body>
</html>