<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 1, 2016 at 12:50 PM, Xavier Hernandez <span dir="ltr"><<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<span class=""><br>
<br>
On 01/06/16 08:53, Raghavendra Gowdappa wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
----- Original Message -----<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
From: "Xavier Hernandez" <<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>><br>
To: "Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>>, "Raghavendra G" <<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>><br>
Cc: "Gluster Devel" <<a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a>><br>
Sent: Wednesday, June 1, 2016 11:57:12 AM<br>
Subject: Re: [Gluster-devel] dht mkdir preop check, afr and (non-)readable afr subvols<br>
<br>
Oops, you are right. For entry operations the current version of the<br>
parent directory is not checked, just to avoid this problem.<br>
<br>
This means that mkdir will be sent to all alive subvolumes. However it<br>
still selects the group of answers that have a minimum quorum equal or<br>
greater than #bricks - redundancy. So it should be still valid.<br>
</blockquote>
<br>
What if the quorum is met on "bad" subvolumes? and mkdir was successful on bad subvolumes? Do we consider mkdir as successful? If yes, even EC suffers from the problem described in bz <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1341429" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1341429</a>.<br>
</blockquote>
<br></span>
I don't understand the real problem. How a subvolume of EC could be in bad state from the point of view of DHT ?<br>
<br>
If you use xattrs to configure something in the parent directories, you should have needed to use setxattr or xattrop to do that. These operations do consider good/bad bricks because they touch inode metadata. This will only succeed if enough (quorum) bricks have successfully processed it. If quorum is met but for an error answer, an error will be reported to DHT and the majority of bricks will be left in the old state (these should be considered the good subvolumes). If some brick has succeeded, it will be considered bad and will be healed. If no quorum is met (even for an error answer), EIO will be returned and the state of the directory should be considered unknown/damaged.<br></blockquote><div><br></div><div style="">Yes. Ideally, dht should use a getxattr for the layout xattr. But, for performance reasons we thought of overloading mkdir by introducing pre-operations (done by bricks). With plain dht it is a simple comparison of xattrs passed as argument and xattrs stored on disk. But, I failed to include afr and EC in the picture. Hence this issue. How difficult for EC and AFR to bring this kind of check? Is it even possible for afr and EC to implement this kind of pre-op checks with reasonable complexity? </div><div style=""><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
If a later mkdir checks this value in storage/posix and succeeds in enough bricks, it necessarily means that is has succeeded in good bricks, because there cannot be enough bricks with the bad xattr value.<br>
<br>
Note that quorum is always > #bricks/2 so we cannot have a quorum with good and bad bricks at the same time.<br>
<br>
Xavi<div class="HOEnZb"><div class="h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Xavi<br>
<br>
On 01/06/16 06:51, Pranith Kumar Karampuri wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Xavi,<br>
But if we keep winding only to good subvolumes, there is a case<br>
where bad subvolumes will never catch up right? i.e. if we keep creating<br>
files in same directory and everytime self-heal completes there are more<br>
entries mounts would have created on the good subvolumes alone. I think<br>
I must have missed this in the reviews if this is the current behavior.<br>
It was not in the earlier releases. Right?<br>
<br>
Pranith<br>
<br>
On Tue, May 31, 2016 at 2:17 PM, Raghavendra G <<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a><br>
<mailto:<a href="mailto:raghavendra@gluster.com" target="_blank">raghavendra@gluster.com</a>>> wrote:<br>
<br>
<br>
<br>
On Tue, May 31, 2016 at 12:37 PM, Xavier Hernandez<br>
<<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a> <mailto:<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>>> wrote:<br>
<br>
Hi,<br>
<br>
On 31/05/16 07:05, Raghavendra Gowdappa wrote:<br>
<br>
+gluster-devel, +Xavi<br>
<br>
Hi all,<br>
<br>
The context is [1], where bricks do pre-operation checks<br>
before doing a fop and proceed with fop only if pre-op check<br>
is successful.<br>
<br>
@Xavi,<br>
<br>
We need your inputs on behavior of EC subvolumes as well.<br>
<br>
<br>
If I understand correctly, EC shouldn't have any problems here.<br>
<br>
EC sends the mkdir request to all subvolumes that are currently<br>
considered "good" and tries to combine the answers. Answers that<br>
match in return code, errno (if necessary) and xdata contents<br>
(except for some special xattrs that are ignored for combination<br>
purposes), are grouped.<br>
<br>
Then it takes the group with more members/answers. If that group<br>
has a minimum size of #bricks - redundancy, it is considered the<br>
good answer. Otherwise EIO is returned because bricks are in an<br>
inconsistent state.<br>
<br>
If there's any answer in another group, it's considered bad and<br>
gets marked so that self-heal will repair it using the good<br>
information from the majority of bricks.<br>
<br>
xdata is combined and returned even if return code is -1.<br>
<br>
Is that enough to cover the needed behavior ?<br>
<br>
<br>
Thanks Xavi. That's sufficient for the feature in question. One of<br>
the main cases I was interested in was what would be the behaviour<br>
if mkdir succeeds on "bad" subvolume and fails on "good" subvolume.<br>
Since you never wind mkdir to "bad" subvolume(s), this situation<br>
never arises.<br>
<br>
<br>
<br>
<br>
Xavi<br>
<br>
<br>
<br>
[1] <a href="http://review.gluster.org/13885" rel="noreferrer" target="_blank">http://review.gluster.org/13885</a><br>
<br>
regards,<br>
Raghavendra<br>
<br>
----- Original Message -----<br>
<br>
From: "Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a><br>
<mailto:<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>>><br>
To: "Raghavendra Gowdappa" <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a><br>
<mailto:<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>>><br>
Cc: "team-quine-afr" <<a href="mailto:team-quine-afr@redhat.com" target="_blank">team-quine-afr@redhat.com</a><br>
<mailto:<a href="mailto:team-quine-afr@redhat.com" target="_blank">team-quine-afr@redhat.com</a>>>, "rhs-zteam"<br>
<<a href="mailto:rhs-zteam@redhat.com" target="_blank">rhs-zteam@redhat.com</a> <mailto:<a href="mailto:rhs-zteam@redhat.com" target="_blank">rhs-zteam@redhat.com</a>>><br>
Sent: Tuesday, May 31, 2016 10:22:49 AM<br>
Subject: Re: dht mkdir preop check, afr and<br>
(non-)readable afr subvols<br>
<br>
I think you should start a discussion on gluster-devel<br>
so that Xavi gets a<br>
chance to respond on the mails as well.<br>
<br>
On Tue, May 31, 2016 at 10:21 AM, Raghavendra Gowdappa<br>
<<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a> <mailto:<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>>><br>
wrote:<br>
<br>
Also note that we've plans to extend this pre-op<br>
check to all dentry<br>
operations which also depend parent layout. So, the<br>
discussion need to<br>
cover all dentry operations like:<br>
<br>
1. create<br>
2. mkdir<br>
3. rmdir<br>
4. mknod<br>
5. symlink<br>
6. unlink<br>
7. rename<br>
<br>
We also plan to have similar checks in lock codepath<br>
for directories too<br>
(planning to use hashed-subvolume as lock-subvolume<br>
for directories). So,<br>
more fops :)<br>
8. lk (posix locks)<br>
9. inodelk<br>
10. entrylk<br>
<br>
regards,<br>
Raghavendra<br>
<br>
----- Original Message -----<br>
<br>
From: "Raghavendra Gowdappa"<br>
<<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a> <mailto:<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>>><br>
To: "team-quine-afr" <<a href="mailto:team-quine-afr@redhat.com" target="_blank">team-quine-afr@redhat.com</a><br>
<mailto:<a href="mailto:team-quine-afr@redhat.com" target="_blank">team-quine-afr@redhat.com</a>>><br>
Cc: "rhs-zteam" <<a href="mailto:rhs-zteam@redhat.com" target="_blank">rhs-zteam@redhat.com</a><br>
<mailto:<a href="mailto:rhs-zteam@redhat.com" target="_blank">rhs-zteam@redhat.com</a>>><br>
Sent: Tuesday, May 31, 2016 10:15:04 AM<br>
Subject: dht mkdir preop check, afr and<br>
(non-)readable afr subvols<br>
<br>
Hi all,<br>
<br>
I have some queries related to the behavior of<br>
afr_mkdir with respect to<br>
readable subvols.<br>
<br>
1. While winding mkdir to subvols does afr check<br>
whether the subvolume is<br>
good/readable? Or does it wind to all subvols<br>
irrespective of whether a<br>
subvol is good/bad? In the latter case, what if<br>
a. mkdir succeeds on non-readable subvolume<br>
b. fails on readable subvolume<br>
<br>
What is the result reported to higher layers<br>
in the above scenario? If<br>
mkdir is failed, is it cleaned up on<br>
non-readable subvolume where it<br>
failed?<br>
<br>
I am interested in this case as dht-preop check<br>
relies on layout xattrs<br>
<br>
and I<br>
<br>
assume layout xattrs in particular (and all<br>
xattrs in general) are<br>
guaranteed to be correct only on a readable<br>
subvolume of afr. So, in<br>
<br>
essence<br>
<br>
we shouldn't be winding down mkdir on<br>
non-readable subvols as whatever<br>
<br>
the<br>
<br>
decision brick makes as part of pre-op check is<br>
inherently flawed.<br>
<br>
regards,<br>
Raghavendra<br>
<br>
--<br>
Pranith<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a> <mailto:<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
<br>
<br>
<br>
<br>
--<br>
Raghavendra G<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a> <mailto:<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
<br>
<br>
<br>
<br>
--<br>
Pranith<br>
</blockquote>
<br>
</blockquote></blockquote>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Raghavendra G<br></div>
</div></div>