<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Response inline<br>
<br>
<div class="moz-cite-prefix">On 01/28/2016 07:28 AM, Ronny Adsetts
wrote:<br>
</div>
<blockquote cite="mid:56AA3382.8030607@amazinginternet.com"
type="cite">
<pre wrap="">Hi all,
Have an issue I'm having trouble explaining or getting to the bottom of. I have a two node, two brick replicated 75G volume containing ~4661 files:
gotham:~# gluster volume status software
Status of volume: software
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick gotham.stor.graysofwestminster.co.uk:/data/gluste
rfs/software/brick1/brick 49152 Y 27296
Brick metropolis.stor.graysofwestminster.co.uk:/data/gl
usterfs/software/brick1/brick 49152 Y 30335
NFS Server on localhost 2049 Y 27309
Self-heal Daemon on localhost N/A Y 27316
NFS Server on metropolis.stor.graysofwestminster.co.uk 2049 Y 30348
Self-heal Daemon on metropolis.stor.graysofwestminster.
co.uk N/A Y 30355
Task Status of Volume software
------------------------------------------------------------------------------
There are no active volume tasks
gotham:~# gluster volume heal software info
Brick gotham.graysofwestminster.co.uk:/data/glusterfs/software/brick1/brick/
Number of entries: 0
Brick metropolis.graysofwestminster.co.uk:/data/glusterfs/software/brick1/brick/
Number of entries: 0
I have the volume mounted on both nodes at /stor/software. On one of the nodes, there are lots of files showing with a link count of 0 which is leading to the Windows software using the data having validation issues:
metropolis:~# ls -ltr /stor/software/win_patches/ | head
total 21585884
-rwxr--r-- 0 ainet Domain Admins 27816 Jan 4 2006 nullpatch.exe
-rwxrwxr-- 0 ronny Domain Admins 135477136 Oct 29 2006 W2KA_SP4_128_x86_ENU.exe
-rwxrwxr-- 0 ronny Domain Admins 432568 Oct 29 2006 MSXML26SP3.exe
-rwxrwxr-- 0 ronny Domain Admins 1070592 Oct 29 2006 msxml3sp7.msi
-rwxrwxr-- 0 ronny Domain Admins 711160 Oct 29 2006 Windows2000-KB842773.EXE
-rwxrwxr-- 0 ronny Domain Admins 760824 Oct 29 2006 Windows2000-KB842933.EXE
-rwxrwxr-- 0 ronny Domain Admins 2585872 Oct 29 2006 WindowsInstaller-KB893803v2.exe
-rwxrwxr-- 0 ronny Domain Admins 106240 Oct 29 2006 Windows-KB870669.exe
-rwxrwxr-- 0 ronny Domain Admins 1260024 Oct 29 2006 Windows2000-KB908506.EXE
>From the other node:
gotham:~# ls -ltr /stor/software/win_patches/ | head
total 21585884
-rwxr--r-- 1 1066 1016 27816 Jan 4 2006 nullpatch.exe
-rwxrwxr-- 1 1045 1016 135477136 Oct 29 2006 W2KA_SP4_128_x86_ENU.exe
-rwxrwxr-- 1 1045 1016 432568 Oct 29 2006 MSXML26SP3.exe
-rwxrwxr-- 1 1045 1016 1070592 Oct 29 2006 msxml3sp7.msi
-rwxrwxr-- 1 1045 1016 711160 Oct 29 2006 Windows2000-KB842773.EXE
-rwxrwxr-- 1 1045 1016 760824 Oct 29 2006 Windows2000-KB842933.EXE
-rwxrwxr-- 1 1045 1016 2585872 Oct 29 2006 WindowsInstaller-KB893803v2.exe
-rwxrwxr-- 1 1045 1016 106240 Oct 29 2006 Windows-KB870669.exe
-rwxrwxr-- 1 1045 1016 1260024 Oct 29 2006 Windows2000-KB908506.EXE
I'm seeing the following and lots more similar in the logs:
[2016-01-28 15:13:15.099478] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-software-client-1: remote operation failed: No such file or directory. Path: /win_patches/SkypeSetup71732106.msi (3489f648-9fde-4bcc-b906-6ef88ffcf90f)
</pre>
</blockquote>
<br>
client-1 is brick2, metropolis. That error says the file is missing,
but it also shows the gfid. Perhaps the gfid hardlink is missing
(the .glusterfs tree).<br>
<br>
Check the metadata for the file on both bricks:<br>
getfattr -m . -d -e hex
/data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi<br>
<br>
Ensure the gfid is the same on both. Check the number of links on
both, there should be 2. With a gifd of
489f648-9fde-4bcc-b906-6ef88ffcf90f, that file should be hard linked
to .glusterfs/48/9f/489f648-9fde-4bcc-b906-6ef88ffcf90f on both
bricks.<br>
<blockquote cite="mid:56AA3382.8030607@amazinginternet.com"
type="cite">
<pre wrap="">
File does exist on both bricks:
metropolis:~# ls -l /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
-rwxr--r-- 1 ainet Domain Admins 44380160 Dec 31 11:41 /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
gotham:~# ls -l /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
-rwxr--r-- 2 1066 1016 44380160 Dec 31 11:41 /data/glusterfs/software/brick1/brick/win_patches/SkypeSetup71732106.msi
Any pointers to get to the bottom of this would be greatly appreciated.</pre>
</blockquote>
<br>
Look for log entries in the brick logs that corrospond to the
timestamps on the client-side and see if there's some correlation.<br>
<br>
<blockquote cite="mid:56AA3382.8030607@amazinginternet.com"
type="cite">
<pre wrap="">
Thanks.
Ronny
</pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>