<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<br>
<div class="moz-cite-prefix">On 10/14/2015 11:08 PM, Игорь Бирюлин
wrote:<br>
</div>
<blockquote
cite="mid:CAEtWxpzUwFpkesWHj_OyxWPJODCz5zs+MWw7ncdNeaS-HOySxw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>
<div>Thanks for <span id="result_box" class="" lang="en"><span
class="">detailed description.<br>
</span></span></div>
<span id="result_box" class="" lang="en"><span class="">Do
you have a plans add resolution GFID split-brain by '</span></span>gluster
volume heal VOLNAME split-brain ...' ?<br>
</div>
</div>
</div>
</div>
</blockquote>
Not at the moment..<br>
<blockquote
cite="mid:CAEtWxpzUwFpkesWHj_OyxWPJODCz5zs+MWw7ncdNeaS-HOySxw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>What the main different between GFID split-brain and data
split brain? On nodes this file absolutely different by data
content and size or it isn't 'data' in glusterfs meaning?<br>
<br>
</div>
</div>
</div>
</blockquote>
GFID is unique to a file (something akin to an inode number) and is
assigned when a file is created. Data split-brain occurs when the
file with same gfid already exists on both bricks, but there's a
difference in the file's content. (eg. one write succeeded only on
brick1 and another write only on brick2). <br>
gfid split-brain occurs when a file creation happens twice (say an
application does an open() with O_CREAT) but succeeds only on one
brick each time.<br>
<blockquote
cite="mid:CAEtWxpzUwFpkesWHj_OyxWPJODCz5zs+MWw7ncdNeaS-HOySxw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>Best regards,<br>
</div>
Igor<br>
<br>
<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2015-10-14 20:13 GMT+03:00 Ravishankar
N <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span class=""> <br>
<br>
<div>On 10/14/2015 10:05 PM, Игорь Бирюлин wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Thanks for your replay.<br>
<br>
</div>
<div>If I do listing in mount point (/repo):<br>
# ls /repo/xxx/keyrings/debian-keyring.gpg<br>
ls: cannot access
/repo/xxx/keyrings/debian-keyring.gpg:
Input/output error<br>
# <br>
</div>
<div>In log /var/log/glusterfs/repo.log I see:<br>
[2015-10-14 16:27:36.006815] W [MSGID: 108008]
[afr-self-heal-name.c:359:afr_selfheal_name_gfid_mismatch_check]
0-repofiles-replicate-0: GFID mismatch for
<gfid:4a99bf9d-7423-47d9-a09d-fabaa333eccf>/debian-keyring.gpg
69aaeee6-624b-400a-aa46-b5c6166c014c on
repofiles-client-1 and
b95ad06e-786a-44e5-ba71-af661982071f on
repofiles-client-0<br>
</div>
</div>
</blockquote>
<br>
</span> So the file has ended up in GFID split-brain (The
trusted.gfid value is different in both bricks as seen in
your output below.), which cannot be handled by the
split-brain resolution commands. These commands can only
resolve data and metadata split-brain. I'm afraid you'll
manually need to delete one of the file and the .glusterfs
hardlink from the brick. Not sure why the parent-directory
was not listed in 'gluster v heal VOLNAME info
split-brain' output. <br>
<div>
<div class="h5"> <br>
<blockquote type="cite">
<div dir="ltr">
<div>[2015-10-14 16:27:36.008996] W
[fuse-bridge.c:451:fuse_entry_cbk]
0-glusterfs-fuse: 65961: LOOKUP()
/xxx/keyrings/debian-keyring.gpg => -1
(Input/output error)<br>
<br>
On first node getfattr return:<br>
# getfattr -d -m . -e hex
/storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg<br>
getfattr: Removing leading '/' from absolute
path names<br>
# file:
storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg<br>
trusted.afr.dirty=0x000000000000000000000000<br>
trusted.afr.repofiles-client-1=0x000000020000000100000000<br>
trusted.bit-rot.version=0x020000000000000055fdf0910003b37b<br>
trusted.gfid=0xb95ad06e786a44e5ba71af661982071f<br>
# ls -l
/storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg<br>
-rw-r--r-- 2 root root 3456271 Oct 13 19:00
/storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg<br>
#<br>
<br>
</div>
<div>On second node getfattr return:<br>
# getfattr -d -m . -e hex
/storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg<br>
getfattr: Removing leading '/' from absolute
path names<br>
# file:
storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg<br>
trusted.afr.dirty=0x000000000000000000000000<br>
trusted.afr.repofiles-client-0=0x000000000000000000000000<br>
trusted.bit-rot.version=0x020000000000000055f97b57000dc3c6<br>
trusted.gfid=0x69aaeee6624b400aaa46b5c6166c014c<br>
# ls -l
/storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg<br>
-rw-r--r-- 2 root root 3450346 Oct 9 16:22
/storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg<br>
</div>
<div>#<br>
<br>
</div>
<div>Best regards,<br>
</div>
<div>Igor<br>
</div>
<div><br>
<br>
<br>
<br>
<br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2015-10-14 19:14
GMT+03:00 Ravishankar N <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:ravishankar@redhat.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:ravishankar@redhat.com">ravishankar@redhat.com</a></a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span> <br>
<br>
<div>On 10/14/2015 07:02 PM, Игорь Бирюлин
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hello,<br>
today in my 2 nodes replica set I've
found split-brain. Command 'ls' start
told 'Input/output error'.<br>
</div>
</blockquote>
<font color="#000099"><br>
</font></span><font color="#3366ff"><font
color="#000099">What does the mount log
(/var/log/glusterfs/<path-to-mount>.log)
say when you get this error?<br>
<br>
Can you run getfattr as root for the
file from *both* bricks and share the
result?<br>
`getfattr -d -m . -e hex
/storage/gluster_brick_repofiles/xxx/keyrings/debian-keyring.gpg`<br>
<br>
Thanks.<br>
Ravi</font><br>
</font><br>
<br>
<blockquote type="cite">
<div>
<div>
<div dir="ltr">But command 'gluster v
heal VOLNAME info split-brain' does
not show problem files:<br>
# gluster v heal repofiles info
split-brain<br>
Brick
dist-int-master03.xxx:/storage/gluster_brick_repofiles<br>
Number of entries in split-brain: 0<br>
<br>
Brick
dist-int-master04.xxx:/storage/gluster_brick_repofiles<br>
Number of entries in split-brain: 0<br>
#<br>
In output of 'gluster v heal VOLNAME
info' I see problem files
(/xxx/keyrings/debian-keyring.gpg,
/repos.json), but without
split-brain markers:<br>
# gluster v heal repofiles info<br>
Brick
dist-int-master03.xxx:/storage/gluster_brick_repofiles<br>
/xxx/keyrings/debian-keyring.gpg<br>
<gfid:09ec49c9-911a-4b83-abe8-080fe79e7c69><br>
<gfid:35c51b11-a7fb-496d-9e88-6d5a54fda7da><br>
/repos.json<br>
<gfid:4f5cb2b5-30e2-43b0-a935-cfc42af883bf><br>
<gfid:9d2fc354-37c0-47a7-b9f3-379504cba797><br>
<gfid:cd86a246-9fc4-47d2-bb4d-67566677f77a><br>
<gfid:b932eed0-07e9-45c5-943e-7478e9f654b4><br>
<gfid:28bf2ffe-948c-4c7d-bce6-966242338581><br>
<gfid:ee5659ae-1335-42c5-a852-790387b4213b><br>
<gfid:fdfb6b8c-3c04-435a-b8d3-8d8341b66409><br>
Number of entries: 11<br>
<br>
Brick
dist-int-master04.xxx:/storage/gluster_brick_repofiles<br>
Number of entries: 0<br>
#<br>
<br>
I couldn't solve split-brain by new
standard command:<br>
# gluster v heal repofiles
split-brain bigger-file /repos.json<br>
Lookup failed on
/repos.json:Input/output error<br>
Volume heal failed.<br>
#<br>
<br>
Additional info:<br>
# gluster v info<br>
Volume Name: repofiles<br>
Type: Replicate<br>
Volume ID:
4b0e2a74-f1ca-4fe7-8518-23919e1b5fa0<br>
Status: Started<br>
Number of Bricks: 1 x 2 = 2<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1:
dist-int-master03.xxx:/storage/gluster_brick_repofiles<br>
Brick2:
dist-int-master04.xxx:/storage/gluster_brick_repofiles<br>
Options Reconfigured:<br>
performance.readdir-ahead: on<br>
client.event-threads: 4<br>
server.event-threads: 4<br>
cluster.lookup-optimize: on<br>
# cat /etc/issue<br>
Ubuntu 14.04.3 LTS \n \l<br>
# dpkg -l | grep glusterfs<br>
ii
glusterfs-client
3.7.5-ubuntu1~trusty1
amd64 clustered file-system
(client package)<br>
ii
glusterfs-common
3.7.5-ubuntu1~trusty1
amd64 GlusterFS common
libraries and translator modules<br>
ii
glusterfs-server
3.7.5-ubuntu1~trusty1
amd64 clustered file-system
(server package)<br>
#<br>
<br>
I have 2 questions:<br>
1. Why 'gluster v heal VOLNAME info
split-brain' doesn't show actual
split-brain? Why in 'gluster v heal
VOLNAME info' I doesn't see markers
like 'possible in split-brain'?<br>
How I can monitor my gluster
installation if these commands
doesn't show problems?<br>
2. Why 'gluster volume heal VOLNAME
split-brain bigger-file FILE'
doesn't solve split-brain? I
understand that I can solve
split-brain remove files from brick
but I thought to use this killer
feature.<br>
<br>
Best regards,<br>
Igor<br>
</div>
<br>
<fieldset></fieldset>
<br>
</div>
</div>
<pre>_______________________________________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>