<div dir="ltr"><div>Hi,<br><br></div><div>Yeah, so the fuse mount log didn't convey much information.<br><br></div><div>So one of the reasons heal may have taken so long (and also consumed resources) is because of a bug in self-heal where it would do heal from both source bricks in 3-way replication. With such a bug, heal would take twice the amount of time and consume resources both the times by the same amount.<br><br></div><div>This issue is fixed at <a href="http://review.gluster.org/#/c/14008/" target="_blank">http://review.gluster.org/#/c/14008/</a> and will be available in 3.7.12.<br><br></div><div>The other thing you could do is to set cluster.data-self-heal-algorithm to 'full', for better heal performance and more regulated resource consumption by the same.<br></div><div> #gluster volume set <VOL> cluster.data-self-heal-algorithm full<br><br></div><div>As far as sharding is concerned, some critical caching issues were fixed in 3.7.7 and 3.7.8.<br></div><div>And my guess is that the vm crash/unbootable state could be because of this issue, which exists in 3.7.6.<br><br></div><div>3.7.10 saw the introduction of throttled client side heals which also moves such heals to the background, which is all the more helpful for preventing starvation of vms during client heal.<br></div><div><br></div><div>Considering these factors, I think it would be better if you upgraded your machines to 3.7.10.<br><br></div><div>Do let me know if migrating to 3.7.10 solves your issues.<br><br></div>-Krutika<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 18, 2016 at 12:40 PM, Kevin Lemonnier <span dir="ltr"><<a href="mailto:lemonnierk@ulrar.net" target="_blank">lemonnierk@ulrar.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Yes, but as I was saying I don't believe KVM is using a mount point, I think it uses<br>
the API (<a href="http://www.gluster.org/community/documentation/index.php/Libgfapi_with_qemu_libvirt" rel="noreferrer" target="_blank">http://www.gluster.org/community/documentation/index.php/Libgfapi_with_qemu_libvirt</a>).<br>
Might be mistaken ofcourse. Proxmox does have a mountpoint for conveniance, I'll attach those<br>
logs, hoping they contain the informations you need. They do seem to contain a lot of errors<br>
for the 15.<br>
For reference, there was a disconnect of the first brick (10.10.0.1) in the morning and then a successfull<br>
heal that caused about 40 minutes downtime of the VMs. Right after that heal finished (if my memory is<br>
correct it was about noon or close) the second brick (10.10.0.2) rebooted, and that's the one I disconnected<br>
to prevent the heal from causing another downtime.<br>
I reconnected it one at the end of the afternoon, hoping the heal would go well but everything went down<br>
like in the morning so I disconnected it again, and waited 11pm (23:00) to reconnect it and let it finish.<br>
<br>
Thanks for your help,<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
On Mon, Apr 18, 2016 at 12:28:28PM +0530, Krutika Dhananjay wrote:<br>
> Sorry, I was referring to the glusterfs client logs.<br>
><br>
> Assuming you are using FUSE mount, your log file will be in<br>
> /var/log/glusterfs/<hyphenated-mount-point-path>.log<br>
><br>
> -Krutika<br>
><br>
> On Sun, Apr 17, 2016 at 9:37 PM, Kevin Lemonnier <<a href="mailto:lemonnierk@ulrar.net">lemonnierk@ulrar.net</a>><br>
> wrote:<br>
><br>
> > I believe Proxmox is just an interface to KVM that uses the lib, so if I'm<br>
> > not mistaken there isn't client logs ?<br>
> ><br>
> > It's not the first time I have the issue, it happens on every heal on the<br>
> > 2 clusters I have.<br>
> ><br>
> > I did let the heal finish that night and the VMs are working now, but it<br>
> > is pretty scarry for future crashes or brick replacement.<br>
> > Should I maybe lower the shard size ? Won't solve the fact that 2 bricks<br>
> > on 3 aren't keeping the filesystem usable but might make the healing<br>
> > quicker right ?<br>
> ><br>
> > Thanks<br>
> ><br>
> > Le 17 avril 2016 17:56:37 GMT+02:00, Krutika Dhananjay <<br>
> > <a href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>> a écrit :<br>
> > >Could you share the client logs and information about the approx<br>
> > >time/day<br>
> > >when you saw this issue?<br>
> > ><br>
> > >-Krutika<br>
> > ><br>
> > >On Sat, Apr 16, 2016 at 12:57 AM, Kevin Lemonnier<br>
> > ><<a href="mailto:lemonnierk@ulrar.net">lemonnierk@ulrar.net</a>><br>
> > >wrote:<br>
> > ><br>
> > >> Hi,<br>
> > >><br>
> > >> We have a small glusterFS 3.7.6 cluster with 3 nodes running with<br>
> > >proxmox<br>
> > >> VM's on it. I did set up the different recommended option like the<br>
> > >virt<br>
> > >> group, but<br>
> > >> by hand since it's on debian. The shards are 256MB, if that matters.<br>
> > >><br>
> > >> This morning the second node crashed, and as it came back up started<br>
> > >a<br>
> > >> heal, but that basically froze all the VM's running on that volume.<br>
> > >Since<br>
> > >> we really really<br>
> > >> can't have 40 minutes down time in the middle of the day, I just<br>
> > >removed<br>
> > >> the node from the network and that stopped the heal, allowing the<br>
> > >VM's to<br>
> > >> access<br>
> > >> their disks again. The plan was to re-connecte the node in a couple<br>
> > >of<br>
> > >> hours to let it heal at night.<br>
> > >> But a VM crashed now, and it can't boot up again : seems to freez<br>
> > >trying<br>
> > >> to access the disks.<br>
> > >><br>
> > >> Looking at the heal info for the volume, it has gone way up since<br>
> > >this<br>
> > >> morning, it looks like the VM's aren't writing to both nodes, just<br>
> > >the one<br>
> > >> they are on.<br>
> > >> It seems pretty bad, we have 2 nodes on 3 up, I would expect the<br>
> > >volume to<br>
> > >> work just fine since it has quorum. What am I missing ?<br>
> > >><br>
> > >> It is still too early to start the heal, is there a way to start the<br>
> > >VM<br>
> > >> anyway right now ? I mean, it was running a moment ago so the data is<br>
> > >> there, it just needs<br>
> > >> to let the VM access it.<br>
> > >><br>
> > >><br>
> > >><br>
> > >> Volume Name: vm-storage<br>
> > >> Type: Replicate<br>
> > >> Volume ID: a5b19324-f032-4136-aaac-5e9a4c88aaef<br>
> > >> Status: Started<br>
> > >> Number of Bricks: 1 x 3 = 3<br>
> > >> Transport-type: tcp<br>
> > >> Bricks:<br>
> > >> Brick1: first_node:/mnt/vg1-storage<br>
> > >> Brick2: second_node:/mnt/vg1-storage<br>
> > >> Brick3: third_node:/mnt/vg1-storage<br>
> > >> Options Reconfigured:<br>
> > >> cluster.quorum-type: auto<br>
> > >> cluster.server-quorum-type: server<br>
> > >> network.remote-dio: enable<br>
> > >> cluster.eager-lock: enable<br>
> > >> performance.readdir-ahead: on<br>
> > >> performance.quick-read: off<br>
> > >> performance.read-ahead: off<br>
> > >> performance.io-cache: off<br>
> > >> performance.stat-prefetch: off<br>
> > >> features.shard: on<br>
> > >> features.shard-block-size: 256MB<br>
> > >> cluster.server-quorum-ratio: 51%<br>
> > >><br>
> > >><br>
> > >> Thanks for your help<br>
> > >><br>
> > >> --<br>
> > >> Kevin Lemonnier<br>
> > >> PGP Fingerprint : 89A5 2283 04A0 E6E9 0111<br>
> > >><br>
> > >> _______________________________________________<br>
> > >> Gluster-users mailing list<br>
> > >> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> > >> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> > >><br>
> ><br>
> > --<br>
> > Envoyé de mon appareil Android avec K-9 Mail. Veuillez excuser ma brièveté.<br>
> > _______________________________________________<br>
> > Gluster-users mailing list<br>
> > <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> ><br>
<br>
--<br>
Kevin Lemonnier<br>
PGP Fingerprint : 89A5 2283 04A0 E6E9 0111<br>
</div></div><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>