<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi Alessandro,<br>
what you describe here reminds me of this issue:<br>
<a class="moz-txt-link-freetext" href="http://www.spinics.net/lists/gluster-users/msg20144.html">http://www.spinics.net/lists/gluster-users/msg20144.html</a><br>
<br>
And now that you mention it, the mess on our cluster could indeed
have been triggered by an aborted rebalance.<br>
This is a very important clue, since apparently developers were
never able to reproduce the issue in the lab. I also tried to
reproduce the issue on a test cluster, but never succeeded.<br>
<br>
The example you describe below seems to me relatively easy to fix. A
<meta http-equiv="content-type" content="text/html;
charset=windows-1252">
<font color="#000000">rebalance fix-layout</font> would eventually
get rid of the sticky bit files (---------T) on your brick 5 and 6
and you could manually remove the files created on 10/03 as long as
you also remove the corresponding link file in the .glusterfs dir on
that brick.<br>
<br>
I whole heartedly agree with you that this needs urgent attention of
developers before they start working on new features. A mess like
this in a distributed file system makes the file system unusable for
production. This should never happen, never! And if it does a
rebalance should be able to detect and fix it... fast and
efficiently. I also agree that the status of a rebalance should be
more telling, giving a clear idea how long it would still take to
complete. On large clusters a rebalance often takes ages and makes
the entire cluster extremely vulnerable. (another scary operation is
a remove-brick operation, but this is another story)<br>
<br>
What I did in our case, maybe this could help you too as a quick fix
for the most critical directories, is to rsync to a different
storage (via a mount point). rsync only copies one file of
duplicated files and you could separately copy a good version (in
the case below e.g.: -rw-r--r-- 2 seviri users 68 May 26 2014
/data/glusterfs/home/brick1/seviri/.forward) of the problem files.
But probably, as soon as you remove the files created on 10/03
(incl. the gluster link file in .glusterfs), the listing via your
NFS mount will be restored. Try this out with a couple of files you
have back-upped to be sure.<br>
<br>
Hope this helps!<br>
<br>
Cheers,<br>
Olav<br>
<br>
<br>
<br>
<br>
<br>
<pre class="moz-signature" cols="72">
</pre>
<div class="moz-cite-prefix">On 20/03/15 12:22, Alessandro Ipe
wrote:<br>
</div>
<blockquote cite="mid:2839724.tvGUlBUEYC@snow.oma.be" type="cite">
<meta name="qrichtext" content="1">
<style type="text/css">
p, li { white-space: pre-wrap; }
</style>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">Hi,</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">After lauching a "rebalance" on an idle
gluster system one week ago, its status told me it has scanned</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> more than 23 millions files on each of my 6
bricks. However, without knowing at least the total files to</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> be scanned, this status is USELESS from an
end-user perspective, because it does not allow you to</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> know WHEN the rebalance could eventually
complete (one day, one week, one year or never). From</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> my point of view, the total files per bricks
could be obtained and maintained when activating quota,</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> since the whole filesystem has to be
crawled...</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">After one week being offline and still no
clue when the rebalance would complete, I decided to stop it...</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> Enormous mistake... It seems that rebalance
cannot manage to not screw some files. Example, on</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> the only client mounting the gluster system,
"ls -la /home/seviri" returns</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">ls: cannot access /home/seviri/.forward:
Stale NFS file handle</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">ls: cannot access /home/seviri/.forward:
Stale NFS file handle</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">-????????? ? ? ? ? ? .forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">-????????? ? ? ? ? ? .forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">while this file could perfectly be accessed
before (being rebalanced) and has not been modifed for at</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> least 3 years.</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">Getting the extended attributes on the
various bricks 3, 4, 5, 6 (3-4 replicate, 5-6 replicate)</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"><span style=" font-weight:600;">Brick 3:</span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">ls -l
/data/glusterfs/home/brick?/seviri/.forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">-rw-r--r-- 2 seviri users 68 May 26 2014
/data/glusterfs/home/brick1/seviri/.forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">-rw-r--r-- 2 seviri users 68 Mar 10 10:22
/data/glusterfs/home/brick2/seviri/.forward</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">getfattr -d -m . -e hex
/data/glusterfs/home/brick?/seviri/.forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"># file:
data/glusterfs/home/brick1/seviri/.forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.afr.home-client-8=0x000000000000000000000000</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.afr.home-client-9=0x000000000000000000000000</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.gfid=0xc1d268beb17443a39d914de917de123a</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"># file:
data/glusterfs/home/brick2/seviri/.forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.afr.home-client-10=0x000000000000000000000000</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.afr.home-client-11=0x000000000000000000000000</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.gfid=0x14a1c10eb1474ef2bf72f4c6c64a90ce</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.glusterfs.quota.4138a9fa-a453-4b8e-905a-e02cce07d717.contri=0x0000000000000200</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.pgfid.4138a9fa-a453-4b8e-905a-e02cce07d717=0x00000001</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"><span style=" font-weight:600;">Brick 4:</span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">ls -l
/data/glusterfs/home/brick?/seviri/.forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">-rw-r--r-- 2 seviri users 68 May 26 2014
/data/glusterfs/home/brick1/seviri/.forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">-rw-r--r-- 2 seviri users 68 Mar 10 10:22
/data/glusterfs/home/brick2/seviri/.forward</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">getfattr -d -m . -e hex
/data/glusterfs/home/brick?/seviri/.forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"># file:
data/glusterfs/home/brick1/seviri/.forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.afr.home-client-8=0x000000000000000000000000</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.afr.home-client-9=0x000000000000000000000000</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.gfid=0xc1d268beb17443a39d914de917de123a</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"># file:
data/glusterfs/home/brick2/seviri/.forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.afr.home-client-10=0x000000000000000000000000</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.afr.home-client-11=0x000000000000000000000000</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.gfid=0x14a1c10eb1474ef2bf72f4c6c64a90ce</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.glusterfs.quota.4138a9fa-a453-4b8e-905a-e02cce07d717.contri=0x0000000000000200</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.pgfid.4138a9fa-a453-4b8e-905a-e02cce07d717=0x00000001</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"><span style=" font-weight:600;">Brick 5:</span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">ls -l
/data/glusterfs/home/brick?/seviri/.forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">---------T 2 root root 0 Mar 18 08:19
/data/glusterfs/home/brick2/seviri/.forward</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">getfattr -d -m . -e hex
/data/glusterfs/home/brick?/seviri/.forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"># file:
data/glusterfs/home/brick2/seviri/.forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.gfid=0x14a1c10eb1474ef2bf72f4c6c64a90ce</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.glusterfs.dht.linkto=0x686f6d652d7265706c69636174652d3400</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"><span style=" font-weight:600;">Brick 6:</span></p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">ls -l
/data/glusterfs/home/brick?/seviri/.forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">---------T 2 root root 0 Mar 18 08:19
/data/glusterfs/home/brick2/seviri/.forward</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">getfattr -d -m . -e hex
/data/glusterfs/home/brick?/seviri/.forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"># file:
data/glusterfs/home/brick2/seviri/.forward</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.gfid=0x14a1c10eb1474ef2bf72f4c6c64a90ce</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">trusted.glusterfs.dht.linkto=0x686f6d652d7265706c69636174652d3400</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">Looking at the results from bricks 3 & 4
shows something weird. The file exists on 2 sub-bricks</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> storage directories, while it should only be
found once on each brick server. Or is the issue lying in the</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> results of bricks 5 & 6 ? <span style="
font-weight:600;">How can I fix this, please</span> ? By the
way, the split-brain tutorial only covers</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> BASIC split-brain conditions and not complex
(real life) cases like this one. It would definitely benefit if</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> enriched by this one.</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">More generally, I think the concept of
gluster is promising, but if basic commands (rebalance,</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> absolutely needed after adding more storage)
from its own cli allows to put the system into an</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> unstable state, I am really starting to
question its ability to be used in a production environment. And</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> from an end-user perspective, I do not care
about new features added, no matter how appealing they</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> could be, if the basic ones are not almost
totally reliable. Finally, testing gluster under high load on
the</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> brick servers (real world conditions) would
certainly gives insight to the developpers on what it failing</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> and what needs therefore to be fixed to
mitigate this and improve gluster reliability.</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">Forgive my harsh words/criticisms, but having
to struggle with gluster issues for two weeks now is</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> getting on my nerves since my colleagues can
not use the data stored on it and I do not see any time</p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;"> from now when it will be back online.</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">Regards,</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<p style=" margin-top:0px; margin-bottom:0px; margin-left:0px;
margin-right:0px; -qt-block-indent:0; text-indent:0px;
-qt-user-state:0;">Alessandro.</p>
<p style="-qt-paragraph-type:empty; margin-top:0px;
margin-bottom:0px; margin-left:0px; margin-right:0px;
-qt-block-indent:0; text-indent:0px; "> </p>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>