<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 15, 2016 at 5:01 PM, Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><br>
<br>
On 06/15/2016 04:24 PM, B.K.Raghuram wrote:<br>
> Hi,<br>
><br>
> We're using gluster 3.6.1 and we periodically find that gluster commands<br>
> fail saying the it could not get the lock on one of the brick machines.<br>
> The logs on that machine then say something like :<br>
><br>
> [2016-06-15 08:17:03.076119] E<br>
> [glusterd-op-sm.c:3058:glusterd_op_ac_lock] 0-management: Unable to<br>
> acquire lock for vol2<br>
<br>
</span>This is a possible case if concurrent volume operations are run. Do you<br>
have any script which checks for volume status on an interval from all<br>
the nodes, if so then this is an expected behavior.<br></blockquote><div><br></div><div>Yes, I do have a couple of scripts that check on volume and quota status.. Given this, I do get a "Another transaction is in progress.." message which is ok. The problem is that sometimes I get the volume lock held message which never goes away. This sometimes results in glusterd consuming a lot of memory and CPU and the problem can only be fixed with a reboot. The log files are huge so I'm not sure if its ok to attach them to an email.<br><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span class="">><br>
> After sometime, glusterd then seems to give up and die..<br>
<br>
</span>Do you mean glusterd shuts down or segfaults, if so I am more interested<br>
in analyzing this part. Could you provide us the glusterd log,<br>
cmd_history log file along with core (in case of SEGV) from all the<br>
nodes for the further analysis?<br></blockquote><div><br></div><div>There is no segfault. glusterd just shuts down. As I said above, sometimes this happens and sometimes it just continues to hog a lot of memory and CPU.. <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span class=""><br>
><br>
> Interestingly, I also find the following line in the beginning of<br>
> etc-glusterfs-glusterd.vol.log and I dont know if this has any<br>
> significance to the issue :<br>
><br>
> [2016-06-14 06:48:57.282290] I<br>
> [glusterd-store.c:2063:glusterd_restore_op_version] 0-management:<br>
> Detected new install. Setting op-version to maximum : 30600<br>
></span><br></blockquote></div><br></div><div class="gmail_extra">What does this line signify?<br></div></div>