<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi Udo, thanks for posting your volume
info settings. Please note for the following, I am not one of the
devs, just a user, so unfortunately I have no authoritative
answers :(<br>
<br>
I am running a very similar setup - Proxmox 4.0, three nodes, but
using ceph for our production storage. Am heavily testing gluster
3.7 on the side. We find the performance of ceph slow on these
small setups and management of it a PITA.<br>
<br>
<br>
Some more questions<br>
<br>
- how are your VM images being accessed by Proxmox? gfapi?
(Proxmox Gluster storage type) or by using the fuse mount?<br>
<br>
- whats your underlying filesystem (ext4, zfs etc)<br>
<br>
- Are you using the HA/Watchdog system in Proxmox?<br>
<br>
<br>
<br>
On 07/12/15 21:03, Udo Giacomozzi wrote:<br>
</div>
<blockquote cite="mid:56656779.5000707@indunet.it" type="cite">esterday
I had a strange situation where Gluster healing corrupted <b
class="moz-txt-star"><span class="moz-txt-tag">*</span>all<span
class="moz-txt-tag">*</span></b> my VM images.
<br>
<br>
<br>
In detail:
<br>
I had about 15 VMs running (in Proxmox 4.0) totaling about 600 GB
of qcow2 images. Gluster is used as storage for those images in
replicate 3 setup (ie. 3 physical servers replicating all data).
<br>
All VMs were running on machine #1 - the two other machines (#2
and #3) were <b class="moz-txt-star"><span class="moz-txt-tag">*</span>idle<span
class="moz-txt-tag">*</span></b>.
<br>
Gluster was fully operating (no healing) when I rebooted machine
#2.
<br>
For other reasons I had to reboot machines #2 and #3 a few times,
but since all VMs were running on machine #1 and nothing on the
other machines was accessing Gluster files, I was confident that
this wouldn't disturb Gluster.
<br>
But anyway this means that I rebootet Gluster nodes during a
healing process.
<br>
<br>
After a few minutes, Gluster files began showing corruption - up
to the point that the qcow2 files became unreadable and all VMs
stopped working.
</blockquote>
<br>
<br>
:( sounds painful - my sympathies. <br>
<br>
You're running 3.5.2 - thats getting rather old. I use the gluster
debian repos:<br>
<br>
3.6.7 :
<a class="moz-txt-link-freetext" href="http://download.gluster.org/pub/gluster/glusterfs/3.6/LATEST/Debian/">http://download.gluster.org/pub/gluster/glusterfs/3.6/LATEST/Debian/</a><br>
3.7.6 :
<a class="moz-txt-link-freetext" href="http://download.gluster.org/pub/gluster/glusterfs/LATEST/Debian/jessie/">http://download.gluster.org/pub/gluster/glusterfs/LATEST/Debian/jessie/</a><br>
<br>
3.6.x is the latest stable, 3.7 is close to stable(?) 3.7 has some
nice new features such as sharding, which is very useful for VM
hosting - it enables much faster heal times.<br>
<br>
Regards what happened with your VM's, I'm not sure. Having two
servers down should have disabled the entire store making it not
readable or writable. I note that you are missing some settings that
need to be set for VM stores - there will be corruption problems if
you live migrate without them.<br>
<br>
<pre>quick-read=off
read-ahead=off
io-cache=off
stat-prefetch=off
eager-lock=enable
remote-dio=enable
quorum-type=auto
server-quorum-type=server
"stat-prefetch=off" is particularly important.
</pre>
<br>
<br>
<br>
<br>
<br>
</body>
</html>