<div dir="ltr"><div><div>Sorry, I was referring to the glusterfs client logs.<br><br></div>Assuming you are using FUSE mount, your log file will be in /var/log/glusterfs/<hyphenated-mount-point-path>.log<br><br></div>-Krutika<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Apr 17, 2016 at 9:37 PM, Kevin Lemonnier <span dir="ltr"><<a href="mailto:lemonnierk@ulrar.net" target="_blank">lemonnierk@ulrar.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I believe Proxmox is just an interface to KVM that uses the lib, so if I'm not mistaken there isn't client logs ?<br>
<br>
It's not the first time I have the issue, it happens on every heal on the 2 clusters I have.<br>
<br>
I did let the heal finish that night and the VMs are working now, but it is pretty scarry for future crashes or brick replacement.<br>
Should I maybe lower the shard size ? Won't solve the fact that 2 bricks on 3 aren't keeping the filesystem usable but might make the healing quicker right ?<br>
<br>
Thanks<br>
<div class="HOEnZb"><div class="h5"><br>
Le 17 avril 2016 17:56:37 GMT+02:00, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>> a écrit :<br>
>Could you share the client logs and information about the approx<br>
>time/day<br>
>when you saw this issue?<br>
><br>
>-Krutika<br>
><br>
>On Sat, Apr 16, 2016 at 12:57 AM, Kevin Lemonnier<br>
><<a href="mailto:lemonnierk@ulrar.net">lemonnierk@ulrar.net</a>><br>
>wrote:<br>
><br>
>> Hi,<br>
>><br>
>> We have a small glusterFS 3.7.6 cluster with 3 nodes running with<br>
>proxmox<br>
>> VM's on it. I did set up the different recommended option like the<br>
>virt<br>
>> group, but<br>
>> by hand since it's on debian. The shards are 256MB, if that matters.<br>
>><br>
>> This morning the second node crashed, and as it came back up started<br>
>a<br>
>> heal, but that basically froze all the VM's running on that volume.<br>
>Since<br>
>> we really really<br>
>> can't have 40 minutes down time in the middle of the day, I just<br>
>removed<br>
>> the node from the network and that stopped the heal, allowing the<br>
>VM's to<br>
>> access<br>
>> their disks again. The plan was to re-connecte the node in a couple<br>
>of<br>
>> hours to let it heal at night.<br>
>> But a VM crashed now, and it can't boot up again : seems to freez<br>
>trying<br>
>> to access the disks.<br>
>><br>
>> Looking at the heal info for the volume, it has gone way up since<br>
>this<br>
>> morning, it looks like the VM's aren't writing to both nodes, just<br>
>the one<br>
>> they are on.<br>
>> It seems pretty bad, we have 2 nodes on 3 up, I would expect the<br>
>volume to<br>
>> work just fine since it has quorum. What am I missing ?<br>
>><br>
>> It is still too early to start the heal, is there a way to start the<br>
>VM<br>
>> anyway right now ? I mean, it was running a moment ago so the data is<br>
>> there, it just needs<br>
>> to let the VM access it.<br>
>><br>
>><br>
>><br>
>> Volume Name: vm-storage<br>
>> Type: Replicate<br>
>> Volume ID: a5b19324-f032-4136-aaac-5e9a4c88aaef<br>
>> Status: Started<br>
>> Number of Bricks: 1 x 3 = 3<br>
>> Transport-type: tcp<br>
>> Bricks:<br>
>> Brick1: first_node:/mnt/vg1-storage<br>
>> Brick2: second_node:/mnt/vg1-storage<br>
>> Brick3: third_node:/mnt/vg1-storage<br>
>> Options Reconfigured:<br>
>> cluster.quorum-type: auto<br>
>> cluster.server-quorum-type: server<br>
>> network.remote-dio: enable<br>
>> cluster.eager-lock: enable<br>
>> performance.readdir-ahead: on<br>
>> performance.quick-read: off<br>
>> performance.read-ahead: off<br>
>> performance.io-cache: off<br>
>> performance.stat-prefetch: off<br>
>> features.shard: on<br>
>> features.shard-block-size: 256MB<br>
>> cluster.server-quorum-ratio: 51%<br>
>><br>
>><br>
>> Thanks for your help<br>
>><br>
>> --<br>
>> Kevin Lemonnier<br>
>> PGP Fingerprint : 89A5 2283 04A0 E6E9 0111<br>
>><br>
>> _______________________________________________<br>
>> Gluster-users mailing list<br>
>> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
>> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
>><br>
<br>
</div></div><span class="HOEnZb"><font color="#888888">--<br>
Envoyé de mon appareil Android avec K-9 Mail. Veuillez excuser ma brièveté.<br>
</font></span><div class="HOEnZb"><div class="h5">_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></div></div></blockquote></div><br></div>