<html><body><div style="font-family: garamond,new york,times,serif; font-size: 12pt; color: #000000"><div><br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Andreas Tsaridas" <andreas.tsaridas@gmail.com><br><b>To: </b>"Krutika Dhananjay" <kdhananj@redhat.com><br><b>Cc: </b>"Pranith Kumar Karampuri" <pkarampu@redhat.com>, gluster-users@gluster.org<br><b>Sent: </b>Friday, January 8, 2016 11:08:58 PM<br><b>Subject: </b>Re: [Gluster-users] folder not being healed<br><div><br></div><div dir="ltr">Hello,<div><br></div><div>Tried doing the same on both bricks and it didn't help. Also tried stat on the folder.</div></div></blockquote><div><br></div><div>Andreas,<br></div><div>The commands I gave you are supposed to be executed on the mount point (the directory where the volume is mounted), and not at the bricks.<br></div><div>Here's what you can do: Create a temporary FUSE mount, and then execute the steps I asked you to execute in my previous response (cd, find, heal etc).<br></div><div><br></div><div>-Krutika<br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div>I don't understand why it shows that a folder has issues and needs heal and not the underlying files.</div><div><br></div><div>Thanks</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 7, 2016 at 10:36 AM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:garamond,new york,times,serif;font-size:12pt;color:#000000"><div>OK. Could you do the following:<br></div><div>1) cd into /media/ga/live/a from the mount point.<br></div><div>2) execute `find . | xargs stat`<br></div><div>3) execute `gluster volume heal <VOLNAME>`<br></div><div><br></div><div>and monitor the output of 'gluster volume heal <VOLNAME> info' to see if there is any progress?<br></div><div><br></div><div>-Krutika<br></div><hr><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><b>From: </b>"Andreas Tsaridas" <<a href="mailto:andreas.tsaridas@gmail.com" target="_blank">andreas.tsaridas@gmail.com</a>><br><b>To: </b>"Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>><span class=""><br><b>Cc: </b>"Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>>, <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br></span><b>Sent: </b>Wednesday, January 6, 2016 7:05:35 PM<div><div class="h5"><br><b>Subject: </b>Re: [Gluster-users] folder not being healed<br><div><br></div><div dir="ltr"><span style="font-size:12.8px">Hello Krutika,</span><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">I have never modified any extended attributes manually so I'm guessing its done by glusterfs.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">I checked all other glusterfs installations and they contain the same attributes. Don't know why you would think these are no normal.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Maybe you can provide some documentation for me to read or a way to tackle the issue ? I'm out of my waters when dealing with glusterfs extended attributes.</div><div><br></div><div>Thanks</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 6, 2016 at 6:08 AM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:garamond,new york,times,serif;font-size:12pt;color:#000000"><div>Andreas,<br></div><div><br></div><div>Gluster doesn't permit applications to set any extended attribute which starts with trusted.afr.* among other patterns.<br></div><div>It is not clear how trusted.afr.remote1/2 extended attributes are appearing in the getfattr output you shared.<br></div><div>Were these directly set from the backend (by backend, I mean the bricks) by any chance?<br></div><div><br></div><div>-Krutika<br></div><hr><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><b>From: </b>"Andreas Tsaridas" <<a href="mailto:andreas.tsaridas@gmail.com" target="_blank">andreas.tsaridas@gmail.com</a>><br><b>To: </b>"Pranith Kumar Karampuri" <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>><br><b>Cc: </b>"Krutika Dhananjay" <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>>, <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br><b>Sent: </b>Tuesday, January 5, 2016 12:27:41 AM<br><b>Subject: </b>Re: [Gluster-users] folder not being healed<div><div><br><div><br></div><div dir="ltr">Hi,<div><br></div><div>I don't understand the question. Should I sent you some kind of configuration ?</div><div><br></div><div>ps: tried looking for you on IRC</div><div><br></div><div>Thanks</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 4, 2016 at 5:20 PM, Pranith Kumar Karampuri <span dir="ltr"><<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><span><span> <br> <br></span></span><div>On 01/04/2016 09:14 PM, Andreas Tsaridas wrote:<br></div><blockquote><div dir="ltr">Hello,<div><br></div><div>Unfortunately I get :</div><div><br></div><div>-bash: /usr/bin/getfattr: Argument list too long</div><div><br></div><div>There are a lot of file in these directories and even ls takes a long time to show results.</div></div></blockquote>Kritika pointed out something important to me on IRC, Why does the volume have two sets of trusted.afr.* xattrs? i.e. trusted.afr.remote1/2 and trusted.afr.share-client-0/1.<span><span style="color:#888888"><br> <br> Pranith</span></span><div><div><br><blockquote><div dir="ltr"><div><br></div><div>How would I be able to keep the copy from web01 and discard the other ?</div><div><br></div><div>Thanks</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 4, 2016 at 3:59 PM, Pranith Kumar Karampuri <span dir="ltr"><<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>hi Andreas,<br> The directory is in split-brain. Do you have any files/directories, that are in split-brain in the directory 'media/ga/live/a' ??<br><br> Could you give output of<br> "getfattr -d -m. -e hex media/ga/live/a/*" on both the bricks?<span><span style="color:#888888"><br> <br> Pranith</span></span><div><div><br><br><div>On 01/04/2016 05:21 PM, Andreas Tsaridas wrote:<br></div><blockquote><div dir="ltr">Hello,<div><br></div><div>Please see below :</div><div>-----</div><div><br></div><div>web01 # getfattr -d -m . -e hex media/ga/live/a</div><div># file: media/ga/live/a</div><div>trusted.afr.dirty=0x000000000000000000000000</div><div>trusted.afr.remote1=0x000000000000000000000000</div><div>trusted.afr.remote2=0x000000000000000000000005</div><div>trusted.afr.share-client-0=0x000000000000000000000000</div><div>trusted.afr.share-client-1=0x0000000000000000000000ee</div><div>trusted.gfid=0xb13199a1464c44918464444b3f7eeee3</div><div>trusted.glusterfs.dht=0x000000010000000000000000ffffffff </div><div><br></div><div><br></div><div>------</div><div><br></div><div><div>web02 # getfattr -d -m . -e hex media/ga/live/a</div><div># file: media/ga/live/a</div><div>trusted.afr.dirty=0x000000000000000000000000</div><div>trusted.afr.remote1=0x000000000000000000000008</div><div>trusted.afr.remote2=0x000000000000000000000000</div><div>trusted.afr.share-client-0=0x000000000000000000000000</div><div>trusted.afr.share-client-1=0x000000000000000000000000</div><div>trusted.gfid=0xb13199a1464c44918464444b3f7eeee3</div><div>trusted.glusterfs.dht=0x000000010000000000000000ffffffff</div></div><div><br></div><div>------</div><div><br></div><div>Regards,</div><div>AT</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 4, 2016 at 12:44 PM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:garamond,new york,times,serif;font-size:12pt;color:#000000"><div>Hi,</div><div><br></div><div>Could you share the output of<br></div><div># getfattr -d -m . -e hex <abs-path-to-media/ga/live/a><br></div><div><br></div><div>from both the bricks?<br></div><div><br></div><div>-Krutika<br></div><hr><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><b>From: </b>"Andreas Tsaridas" <<a href="mailto:andreas.tsaridas@gmail.com" target="_blank">andreas.tsaridas@gmail.com</a>><br><b>To: </b><a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br><b>Sent: </b>Monday, January 4, 2016 5:10:58 PM<br><b>Subject: </b>[Gluster-users] folder not being healed<div><div><br><div><br></div><div dir="ltr">Hello,<div><br></div><div>I have a cluster of two replicated nodes in glusterfs 3.6.3 in RedHat 6.6. Problem is that a specific folder is always trying to be healed but never gets healed. This has been going on for 2 weeks now.</div><div><br></div><div>-----</div><div><br></div><div><div># gluster volume status</div><div>Status of volume: share</div><div>Gluster process<span style="white-space:pre-wrap"> </span>Port<span style="white-space:pre-wrap"> </span>Online<span style="white-space:pre-wrap"> </span>Pid</div><div>------------------------------------------------------------------------------</div><div>Brick 172.16.4.1:/srv/share/glusterfs<span style="white-space:pre-wrap"> </span>49152<span style="white-space:pre-wrap"> </span>Y<span style="white-space:pre-wrap"> </span>10416</div><div>Brick 172.16.4.2:/srv/share/glusterfs<span style="white-space:pre-wrap"> </span>49152<span style="white-space:pre-wrap"> </span>Y<span style="white-space:pre-wrap"> </span>19907</div><div>NFS Server on localhost<span style="white-space:pre-wrap"> </span>2049<span style="white-space:pre-wrap"> </span>Y<span style="white-space:pre-wrap"> </span>22664</div><div>Self-heal Daemon on localhost<span style="white-space:pre-wrap"> </span>N/A<span style="white-space:pre-wrap"> </span>Y<span style="white-space:pre-wrap"> </span>22676</div><div>NFS Server on 172.16.4.2<span style="white-space:pre-wrap"> </span>2049<span style="white-space:pre-wrap"> </span>Y<span style="white-space:pre-wrap"> </span>19923</div><div>Self-heal Daemon on 172.16.4.2<span style="white-space:pre-wrap"> </span>N/A<span style="white-space:pre-wrap"> </span>Y<span style="white-space:pre-wrap"> </span>19937</div><div><br></div><div>Task Status of Volume share</div><div>------------------------------------------------------------------------------</div><div>There are no active volume tasks</div></div><div><br></div><div>------</div><div><br></div><div><div># gluster volume info</div><div><br></div><div>Volume Name: share</div><div>Type: Replicate</div><div>Volume ID: 17224664-645c-48b7-bc3a-b8fc84c6ab30</div><div>Status: Started</div><div>Number of Bricks: 1 x 2 = 2</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: 172.16.4.1:/srv/share/glusterfs</div><div>Brick2: 172.16.4.2:/srv/share/glusterfs</div><div>Options Reconfigured:</div><div>cluster.background-self-heal-count: 20</div><div>cluster.heal-timeout: 2</div><div>performance.normal-prio-threads: 64</div><div>performance.high-prio-threads: 64</div><div>performance.least-prio-threads: 64</div><div>performance.low-prio-threads: 64</div><div>performance.flush-behind: off</div><div>performance.io-thread-count: 64</div></div><div><br></div><div>------</div><div><br></div><div><div># gluster volume heal share info</div><div>Brick web01.rsdc:/srv/share/glusterfs/</div><div>/media/ga/live/a - Possibly undergoing heal</div><div><br></div><div>Number of entries: 1</div><div><br></div><div>Brick web02.rsdc:/srv/share/glusterfs/</div><div>Number of entries: 0</div></div><div><br></div><div>-------</div><div><br></div><div><div># gluster volume heal share info split-brain</div><div>Gathering list of split brain entries on volume share has been successful</div><div><br></div><div>Brick 172.16.4.1:/srv/share/glusterfs</div><div>Number of entries: 0</div><div><br></div><div>Brick 172.16.4.2:/srv/share/glusterfs</div><div>Number of entries: 0</div></div><div><br></div><div>-------</div><div><br></div><div><div>==> /var/log/glusterfs/glustershd.log <==</div><div>[2016-01-04 11:35:33.004831] I [afr-self-heal-entry.c:554:afr_selfheal_entry_do] 0-share-replicate-0: performing entry selfheal on b13199a1-464c-4491-8464-444b3f7eeee3</div><div>[2016-01-04 11:36:07.449192] W [client-rpc-fops.c:2772:client3_3_lookup_cbk] 0-share-client-1: remote operation failed: No data available. Path: (null) (00000000-0000-0000-0000-000000000000)</div><div>[2016-01-04 11:36:07.449706] W [client-rpc-fops.c:240:client3_3_mknod_cbk] 0-share-client-1: remote operation failed: File exists. Path: (null)</div></div><div><br></div><div>Could you please advise ?</div><div><br></div><div>Kind regards,</div><div><br></div><div>AT</div></div><br></div></div>_______________________________________________<br> Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br><a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote><div><br></div></div></div></blockquote></div><br></div><br><br><pre>_______________________________________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></pre></blockquote><br></div></div></div></blockquote></div><br></div></blockquote><br></div></div></div></blockquote></div><br></div></div></div></blockquote><div><br></div></div></div></blockquote></div><br></div></div></div></blockquote><div><br></div></div></div></blockquote></div><br></div></blockquote><div><br></div></div></body></html>