<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 13, 2016 at 12:11 PM, Dmitry Melekhov <span dir="ltr"><<a href="mailto:dm@belkam.com" target="_blank">dm@belkam.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>13.07.2016 10:24, Pranith Kumar
Karampuri пишет:<br>
</div><div><div class="h5">
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Jul 13, 2016 at 11:49 AM,
Dmitry Melekhov <span dir="ltr"><<a href="mailto:dm@belkam.com" target="_blank"></a><a href="mailto:dm@belkam.com" target="_blank">dm@belkam.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>13.07.2016 10:10, Pranith Kumar Karampuri пишет:<br>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Jul 13, 2016
at 11:27 AM, Dmitry Melekhov <span dir="ltr"><<a href="mailto:dm@belkam.com" target="_blank">dm@belkam.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>13.07.2016 09:50, Pranith Kumar
Karampuri пишет:<br>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Wed, Jul 13, 2016 at 11:11
AM, Dmitry Melekhov <span dir="ltr"><<a href="mailto:dm@belkam.com" target="_blank"></a><a href="mailto:dm@belkam.com" target="_blank">dm@belkam.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>13.07.2016 09:36,
Pranith Kumar
Karampuri пишет:<br>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Wed, Jul 13,
2016 at 10:58
AM, Dmitry
Melekhov <span dir="ltr"><<a href="mailto:dm@belkam.com" target="_blank"></a><a href="mailto:dm@belkam.com" target="_blank">dm@belkam.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>13.07.2016
09:26, Pranith
Kumar
Karampuri
пишет:<br>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Wed, Jul 13,
2016 at 10:50
AM, Dmitry
Melekhov <span dir="ltr"><<a href="mailto:dm@belkam.com" target="_blank"></a><a href="mailto:dm@belkam.com" target="_blank">dm@belkam.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>13.07.2016
09:16, Pranith
Kumar
Karampuri
пишет:<br>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Wed, Jul 13,
2016 at 10:38
AM, Dmitry
Melekhov <span dir="ltr"><<a href="mailto:dm@belkam.com" target="_blank"></a><a href="mailto:dm@belkam.com" target="_blank">dm@belkam.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>13.07.2016
09:04, Pranith
Kumar
Karampuri
пишет:<br>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Wed, Jul 13,
2016 at 10:29
AM, Dmitry
Melekhov <span dir="ltr"><<a href="mailto:dm@belkam.com" target="_blank"></a><a href="mailto:dm@belkam.com" target="_blank">dm@belkam.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>13.07.2016
08:56, Pranith
Kumar
Karampuri
пишет:<br>
</div>
<div>
<div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Wed, Jul 13,
2016 at 10:23
AM, Dmitry
Melekhov <span dir="ltr"><<a href="mailto:dm@belkam.com" target="_blank"></a><a href="mailto:dm@belkam.com" target="_blank">dm@belkam.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>13.07.2016
08:46, Pranith
Kumar
Karampuri
пишет:<br>
</div>
<span>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Wed, Jul 13,
2016 at 10:10
AM, Dmitry
Melekhov <span dir="ltr"><<a href="mailto:dm@belkam.com" target="_blank"></a><a href="mailto:dm@belkam.com" target="_blank">dm@belkam.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>13.07.2016
08:36, Pranith
Kumar
Karampuri
пишет:<br>
</div>
<span>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On
Wed, Jul 13,
2016 at 9:35
AM, Dmitry
Melekhov <span dir="ltr"><<a href="mailto:dm@belkam.com" target="_blank"></a><a href="mailto:dm@belkam.com" target="_blank">dm@belkam.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">13.07.2016
01:52,
Anuradha Talur
пишет:<span><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
----- Original
Message -----<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
From: "Dmitry
Melekhov" <<a href="mailto:dm@belkam.com" target="_blank"></a><a href="mailto:dm@belkam.com" target="_blank">dm@belkam.com</a>><br>
To: "Pranith
Kumar
Karampuri"
<<a href="mailto:pkarampu@redhat.com" target="_blank"></a><a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>><br>
Cc:
"gluster-users"
<<a href="mailto:gluster-users@gluster.org" target="_blank"></a><a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br>
Sent: Tuesday,
July 12, 2016
9:27:17 PM<br>
Subject: Re:
[Gluster-users]
3.7.13, index
healing
broken?<br>
<br>
<br>
<br>
12.07.2016
17:39, Pranith
Kumar
Karampuri
пишет:<br>
<br>
<br>
<br>
Wow, what are
the steps to
recreate the
problem?<br>
<br>
just set file
length to
zero, always
reproducible.<br>
<br>
</blockquote>
If you are
setting the
file length to
0 on one of
the bricks
(looks like<br>
that is the
case), it is
not a bug.<br>
<br>
Index heal
relies on
failures seen
from the mount
point(s)<br>
to identify
the files that
need heal. It
won't be able
to recognize
any file<br>
modification
done directly
on bricks.
Same goes for
heal info
command which<br>
is the reason
heal info also
shows 0
entries.<br>
</blockquote>
<br>
</span> Well,
this makes
self-heal
useless then-
if any file is
accidently
corrupted or
deleted (yes!
if file is
deleted
directly from
brick this is
no recognized
by idex heal
too), then it
will not be
self-healed,
because
self-heal uses
index heal.<span><br>
</span></blockquote>
<div><br>
</div>
<div>It is
better to look
into bit-rot
feature if you
want to guard
against these
kinds of
problems.<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</span> Bit
rot detects
bit problems,
not missing
files or their
wrong length,
i.e. this is
overhead for
such simple
task.<br>
</div>
</blockquote>
<div><br>
</div>
<div>It
detects wrong
length.
Because
checksum won't
match anymore.<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</span> Yes,
sure. I guess
that it will
detect missed
files too. But
it needs far
more
resources,
then just
comparing
directories in
bricks?<span><br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>What
use-case you
are trying out
is leading to
changing
things
directly on
the brick?<br>
</div>
</div>
</div>
</div>
</blockquote>
</span> I'm
trying to test
gluster
failure
tolerance and
right now I'm
not happy with
it...<br>
</div>
</blockquote>
<div><br>
</div>
<div>Which
cases of fault
tolerance are
you not happy
with? Making
changes
directly on
the brick or
anything else
as well?<br>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
</div>
I'll repeat:<span><br>
As I already
said- if I for
some reason (
real case can
be only by
accident )
will delete
file this will
not be
detected by
self-heal
daemon, and,
thus, will
lead to lower
replication
level, i.e.
lower failure
tolerance. <br>
</span></div>
</blockquote>
<div><br>
To prevent
such accidents
you need to
set selinux
policies so
that files
under the
brick are not
modified by
accident by
any user. At
least that is
the solution I
remember when
this was
discussed 3-4
years back.<br>
</div>
</div>
<br>
</div>
</div>
</blockquote>
</div>
</div>
So only
supported
platfrom is
linux? Or, may
be, it is
better to
improve
self-healing
to detect
missing or
wrong length
files, I guess
this is very
low cost in
terms of host
resources
operation.<br>
Just a
suggestion,
may be we need
to look to
alternatives
in near
future....<br>
<br>
</div>
</blockquote>
</div>
This is a
corner case,
from design
perspective it
is generally
not a good
idea to
optimize for
the corner
case. It is
better to
protect
ourselves from
the corner
case (SElinux
etc) or you
can also use
snapshots to
protect
against these
kind of
mishaps.<br>
</div>
<div class="gmail_extra"><br>
</div>
</div>
</blockquote>
</div>
</div>
Sorry, I'm not
agree. <br>
As you know
if on access
missed or
wrong lenghted
file from fuse
client it is
restored
(healed), i.e.
gluster
recognizes
file is wrong
and heal it ,
so I do not
see any reason
to provide
this such
function as
self-healing.<br>
Thank you!<br>
<br>
</div>
</blockquote>
</div>
Ah! Now how do
you suggest we
keep track of
which of 10s
of millions of
files the user
accidentally
deleted from
the brick
without
gluster's
knowledge?
Once it comes
to gluster's
knowledge we
can do
something. But
how does
gluster become
aware of
something it
is not keeping
track of? At
the time you
access it
gluster knows
something went
wrong so it
restores it.
If you change
something on
the bricks
even by
accident all
the data
gluster keeps
(similar to
journal) is a
waste. Even
the disk
filesystems
will ask you
to do fsck if
something
unexpected
happens so
full self-heal
is similar
operation.<br>
</div>
</div>
</blockquote>
<br>
</div>
</div>
You are
absolutely
right-
question is
why gluster
does not
become aware
about such
problem is
case of
self-healing?<br>
</div>
</blockquote>
<div><br>
</div>
<div>Because
the operations
that are
performed
directly on
brick do not
go through
gluster stack.<br>
</div>
<div> </div>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
OK, I'll repeat- <br>
<span> As you know if
on access missed or
wrong lenghted file
from fuse client it is
restored (healed),
i.e. gluster
recognizes file is
wrong and heal it , so
I do not see any
reason to provide this
such function as
self-healing.<br>
</span></div>
</blockquote>
<div><br>
</div>
<div>For which you need
accessing the file. </div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
That's right.<span><br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>For which you need full
crawl. You can't detect the
modification which doesn't
go through the stack so this
is the only possibility. <br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</span> OK, then, if self-heal is really
useless and no possible way to get it
will be provided, I guess we'll use
external script to check bricks
directories consistency,<br>
don't think ls and diff will get much
resources.<br>
</div>
</blockquote>
<div><br>
</div>
<div>How is this different from full
self-heal?<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</div>
</div>
Self-heal does not detect deleted or wrong-length files
.<br>
</div>
</blockquote>
<div><br>
It detects when you do full crawl. Which essentially is ls
-laR kind of thing on the whole volume. You don't need any
external scripts, keep doing full crawl once in a while
may be? </div>
</div>
</div>
</div>
</blockquote>
<br></div></div>
You mean on fuse mount?<br>
<br>
It doesn't work:<br>
<br>
[root@father ~]# mount -t glusterfs localhost:/pool gluster <br>
<br>
[root@father ~]# <br>
<br>
then make it zero lengths in brick:<br>
<br>
[root@father gluster]# >
/wall/pool/brick/gstatus-0.64-3.el7.x86_64.rpm <br>
[root@father gluster]# <br>
<br>
<br>
[root@father gluster]# ls -laR /root/gluster/<br>
/root/gluster/:<br>
итого 122153384<br>
drwxr-xr-x 4 qemu qemu 4096 июл 11 13:36 .<br>
dr-xr-x---. 10 root root 4096 июл 11 12:26 ..<br>
-rw-r--r-- 1 root root 8589934592 май 11 09:14 csr1000v1.img<br>
-rw-r--r-- 1 root root 0 июл 13 10:34
gstatus-0.64-3.el7.x86_64.rpm<br>
<br>
<br>
As you can see gstatus-0.64-3.el7.x86_64.rpm has 0 length<br>
But:<br>
<br>
[root@father gluster]# touch
/root/gluster/gstatus-0.64-3.el7.x86_64.rpm<br>
[root@father gluster]# ls -laR /root/gluster/<br>
/root/gluster/:<br>
итого 122153436<br>
drwxr-xr-x 4 qemu qemu 4096 июл 11 13:36 .<br>
dr-xr-x---. 10 root root 4096 июл 11 12:26 ..<br>
-rw-r--r-- 1 root root 8589934592 май 11 09:14 csr1000v1.img<br>
-rw-r--r-- 1 root root 52268 июл 13 10:36
gstatus-0.64-3.el7.x86_64.rpm<br>
<br>
<br>
I.e. if I do some i.o. on file then it is back.<br>
<br>
<br>
By the way the same problem if I delete file directly in brick:<br>
<br>
[root@father gluster]# rm
/wall/pool/brick/gstatus-0.64-3.el7.x86_64.rpm <br>
rm: удалить обычный файл
«/wall/pool/brick/gstatus-0.64-3.el7.x86_64.rpm»? y<br>
[root@father gluster]# ls -laR /root/gluster/<br>
/root/gluster/:<br>
итого 122153384<br>
drwxr-xr-x 4 qemu qemu 4096 июл 13 10:38 .<br>
dr-xr-x---. 10 root root 4096 июл 11 12:26 ..<br>
-rw-r--r-- 1 root root 8589934592 май 11 09:14 csr1000v1.img<br>
-rw-r--r-- 1 qemu qemu 43692064768 июл 13 10:38 infimonitor.img<br>
<br>
<br>
I don't see it in directory in fuse mount at all till touch, which
restores file too.<span class=""><br>
<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>If you need any performance improvements here, we will
be happy to help. Please give us feedback. </div>
</div>
</div>
</div>
</blockquote>
<br></span>
You recipe doesn't work :-( If there is difference between bricks
directories due to direct brick manipulation it leads to problems.<span class=""><br></span></div></blockquote><div><br></div><div>You have to execute "gluster volume heal <volname> full" for triggering full heal.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><span class="">
<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>All I was saying is it is not possible to detect them
through index heal. Because for the index to be populated
you need the operations to go through gluster stack.<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Why it can't ? I
don't know, you just said it is impossible in gluster
because it can only track changes only made through
gluster, i.e. bricks can have different files sets and
it is not recognized (true) because , as I understand,
gluster's self-heal thinks that brick underlying
filesystem can't be corrupted by server admin (not
true, I can say this as almost 25 years experienced
engineer, i.e. I did this several times ;-) ).<span><br>
<br>
<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <br>
Thank you!<br>
<br>
p.s.<br>
still can't understand why it can't be
implemented in gluster... :-(<span><br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span> <br>
</span>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra"><br clear="all">
<span><font color="#888888">
<span><font color="#888888">
<br>
-- <br>
<div data-smartmail="gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</font></span></font></span></div>
<span><font color="#888888">
</font></span></div>
<span><font color="#888888">
</font></span></blockquote>
<span><font color="#888888">
<br>
</font></span></div>
<span><font color="#888888">
</font></span></blockquote>
<span><font color="#888888">
</font></span></div>
<span><font color="#888888"> <br>
<br clear="all">
<br>
-- <br>
<div data-smartmail="gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</font></span></div>
</div>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div data-smartmail="gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</span></div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div data-smartmail="gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</span></div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div data-smartmail="gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</span></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>