<div dir="ltr">Hi,<div><br></div><div>Never tried something other than proxmox</div><div>Well, all the way around I was testing this issue I was watching installation logs and seen nothing but failures themselves.</div><div><br></div><div><a href="https://romanr.eu/owncloud/index.php/s/76Mf3Z1rIbvDgIO">https://romanr.eu/owncloud/index.php/s/76Mf3Z1rIbvDgIO</a><br></div><div>these are errors I've managed to get.</div><div>The only thing I understood during these test is that FS is being corrupted. Another guy who tested the same ended up with mount error during boot due to errors in FS and when it started to chkfs it, it reported that his installation suddenly runs FAT32 ... </div></div><div class="gmail_extra"><br><div class="gmail_quote">2015-08-06 5:26 GMT+03:00 Niels de Vos <span dir="ltr"><<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Tue, Aug 04, 2015 at 04:06:50PM +0300, Roman wrote:<br>
> Hi all,<br>
> I'm back and tested those things.<br>
> Michael was right. I've enabled read-ahead option and nothing changed.<br>
> So the thing causes the problem with libgfapi and d8 virtio drivers<br>
> is performance.write-behind. If it is off, everything works perfect. If I<br>
> set it On, different problems are as result (which were confirmed by other<br>
> user with same configuration).<br>
<br>
</span>Thanks for the follow up, Roman!<br>
<br>
We'll need to think about how to trace the I/O pattern that the<br>
installation does, and see if we can emulate that in a test-case. Once<br>
we can easily reproduce the problem, investigating the cause gets<br>
easier.<br>
<br>
I've got some more questions that should help us:<br>
- Do you know if any non-Proxmox environments had the same issue?<br>
- Is it possible to notice the failure in the guest and abort? When we<br>
track the I/O pattern, the failure would be at the end, and not<br>
somewhere in the trace (almost impossible to find).<br>
<br>
Thanks,<br>
Niels<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
> 2015-07-20 1:53 GMT+03:00 Roman <<a href="mailto:romeo.r@gmail.com">romeo.r@gmail.com</a>>:<br>
><br>
> > Thanks for your reply.<br>
> > I will test these options as soon as I'm back from ma vacation (2 weeks<br>
> > from now on). I'll be too far from servers to change something even on<br>
> > testing volume 8)<br>
> ><br>
> > 2015-07-19 23:47 GMT+03:00 Niels de Vos <<a href="mailto:ndevos@redhat.com">ndevos@redhat.com</a>>:<br>
> ><br>
> >> On Sat, Jul 18, 2015 at 03:56:37PM +0000, Michael Mol wrote:<br>
> >> > I think you'll find it's the write-behind that was killing you.<br>
> >> > Write-behind opens you up to a number of data consistency issues, and I<br>
> >> > strongly disrecommend it unless you have a rock-solid infrastructure<br>
> >> from<br>
> >> > the writer all the way to the disk the data ultimately sits on.<br>
> >><br>
> >> The suggestion to disable these two options was to change the access<br>
> >> pattern done by Qemu+libgfapi. Without looking into the source code, I<br>
> >> do not know how write-behind and read-ahead play together. Anything that<br>
> >> is "written behind" should get flushed when a "read ahead" operation<br>
> >> overlaps a "written behind" area. There have been issues with<br>
> >> write-behind before, where the flushing was not done in some corner<br>
> >> case (<a href="https://github.com/gluster/glusterfs/commit/b0515e2a" rel="noreferrer" target="_blank">https://github.com/gluster/glusterfs/commit/b0515e2a</a>). Maybe this<br>
> >> is something similar.<br>
> >><br>
> >> > I bet that if you re-enabled read-ahead, you won't see the problem. Just<br>
> >> > leave write-behind off.<br>
> >><br>
> >> Indeed, write-behind is most likely. Both results of one of the options<br>
> >> disabled at the time would be interesting to have.<br>
> >><br>
> >> Once it is clear which option causes the problem, we can analyze the<br>
> >> access pattern and hopefully fix the xlator.<br>
> >><br>
> >> Thanks,<br>
> >> Niels<br>
> >><br>
> >><br>
> >> > On Sat, Jul 18, 2015, 10:44 AM Roman <<a href="mailto:romeo.r@gmail.com">romeo.r@gmail.com</a>> wrote:<br>
> >> ><br>
> >> > solved after I've added (thanks to Niels de Vos) these options to the<br>
> >> > volumes:<br>
> >> ><br>
> >> > performance.read-ahead: off<br>
> >> ><br>
> >> > performance.write-behind: off<br>
> >> ><br>
> >> ><br>
> >> > 2015-07-15 17:23 GMT+03:00 Roman <<a href="mailto:romeo.r@gmail.com">romeo.r@gmail.com</a>>:<br>
> >> ><br>
> >> > hey,<br>
> >> ><br>
> >> > I've updated the bug, if some1 has some ideas - share plz.<br>
> >> ><br>
> >> > <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1242913" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1242913</a><br>
> >> ><br>
> >> ><br>
> >> > 2015-07-14 19:14 GMT+03:00 Kaushal M <<a href="mailto:kshlmster@gmail.com">kshlmster@gmail.com</a>>:<br>
> >> ><br>
> >> > Just a wild guess. What is the filesystem used for the debian 8<br>
> >> > installation? It could be the culprit.<br>
> >> ><br>
> >> > On Tue, Jul 14, 2015 at 7:27 PM, Roman <<a href="mailto:romeo.r@gmail.com">romeo.r@gmail.com</a>> wrote:<br>
> >> > > I've done this way: installed debian8 on local disks using netinstall<br>
> >> iso,<br>
> >> > > created a template of it and then cloned (full clone) it to glusterfs<br>
> >> > > storage backend. VM boots and runs fine... untill I start to install<br>
> >> > > something massive (DE ie). Last time it was mate failed to install<br>
> >> due to<br>
> >> > > python-gtk2 package problems (complaing that it could not compile it)<br>
> >> > ><br>
> >> ><br>
> >> > > 2015-07-14 16:37 GMT+03:00 Scott Harvanek <<a href="mailto:scott.harvanek@login.com">scott.harvanek@login.com</a>>:<br>
> >> > >><br>
> >> > >> What happens if you install from a full CD and not a net-install?<br>
> >> > >><br>
> >> > >> Limit the variables. Currently you are relying on remote mirrors and<br>
> >> > >> Internet connectivity.<br>
> >> > >><br>
> >> > >> It's either a Proxmox or Debian issue, I really don't think it's<br>
> >> Gluster.<br>
> >> > >> We have hundreds of Jessie installs running on GlusterFS backends.<br>
> >> > >><br>
> >> > >> --<br>
> >> > >> Scott H.<br>
> >> > >> Login, LLC.<br>
> >> > >><br>
> >> > >><br>
> >> > >><br>
> >> > >> Roman<br>
> >> > >> July 14, 2015 at 9:30 AM<br>
> >> > >> Hey,<br>
> >> > >><br>
> >> > >> thanks for reply.<br>
> >> > >> If it would be networking related, it would affect everything. But<br>
> >> it is<br>
> >> > >> only debian 8 which won't install.<br>
> >> > >> And yes, i did iperf test between gluster and proxmox nodes. Its ok.<br>
> >> > >> Installation fails on every node, where i try to install d8.<br>
> >> Sometimes it<br>
> >> > >> goes well (today 1 of 6 tries was fine). Other distros install fine.<br>
> >> > >> Sometimes installation process finishes, but VM won't start, just<br>
> >> hangs<br>
> >> > >> with errors like in this attached.<br>
> >> > >><br>
> >> > >><br>
> >> > >><br>
> >> > >><br>
> >> > >> --<br>
> >> > >> Best regards,<br>
> >> > >> Roman.<br>
> >> > >> Scott Harvanek<br>
> >> > >> July 14, 2015 at 9:17 AM<br>
> >> > >> We don't have this issue, I'll take a stab tho-<br>
> >> > >><br>
> >> > >> Have you confirmed everything is good on the network side of things?<br>
> >> > >> MTU/Loss/Errors?<br>
> >> > >><br>
> >> > >> Is your inconsistency linked to one specific brick? Have you tried<br>
> >> > running<br>
> >> > >> a replica instead of distributed?<br>
> >> > >><br>
> >> > >><br>
> >> > >><br>
> >> > >> _______________________________________________<br>
> >> > >> Gluster-users mailing list<br>
> >> ><br>
> >> > >> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> >> > >> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> >> > >> Roman<br>
> >> > >> July 14, 2015 at 6:38 AM<br>
> >> > >> here is one of the errors example. its like files that debian<br>
> >> installer<br>
> >> > >> copies to the virtual disk that is located on glusterfs storage are<br>
> >> > getting<br>
> >> > >> corrupted.<br>
> >> > >> in-target is /dev/vda1<br>
> >> > >><br>
> >> > >><br>
> >> > >><br>
> >> > >><br>
> >> > >><br>
> >> > >> --<br>
> >> > >> Best regards,<br>
> >> > >> Roman.<br>
> >> > >> _______________________________________________<br>
> >> > >> Gluster-users mailing list<br>
> >> > >> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> >> > >> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> >> > >> Roman<br>
> >> > >> July 14, 2015 at 4:50 AM<br>
> >> > >> Ubuntu 14.04 LTS base install and then mate install were fine!<br>
> >> > >><br>
> >> > >><br>
> >> > >><br>
> >> > >><br>
> >> > >> --<br>
> >> > >> Best regards,<br>
> >> > >> Roman.<br>
> >> > >> _______________________________________________<br>
> >> > >> Gluster-users mailing list<br>
> >> > >> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> >> > >> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> >> > >> Roman<br>
> >> > >> July 13, 2015 at 7:35 PM<br>
> >> > >> Bah... the randomness of this issue is killing me.<br>
> >> > >> Not only HA volumes are affected. Got an error during installation<br>
> >> of d8<br>
> >> > >> with mate (on python-gtk2 pkg) on Distributed volume also.<br>
> >> > >> I've checked the MD5SUM of installation iso, its ok.<br>
> >> > >><br>
> >> > >> Shortly after that on the same VE node I installed D7 with Gnome<br>
> >> without<br>
> >> > >> any problem on the HA glusterf volume.<br>
> >> > >><br>
> >> > >> And on the same VE node I've installed D8 with both Mate and Gnome<br>
> >> using<br>
> >> > >> local storage disks without problems. There is a bug somewhere in<br>
> >> > gluster or<br>
> >> > >> qemu... Proxmox uses RH kernel btw:<br>
> >> > >><br>
> >> > >> Linux services 2.6.32-37-pve<br>
> >> > >> QEMU emulator version 2.2.1<br>
> >> > >> glusterfs 3.6.4<br>
> >> > >><br>
> >> > >> any ideas?<br>
> >> > >> I'm ready to help to investigate this bug.<br>
> >> > >> When sun will shine, I'll try to install latest Ubuntu also. But now<br>
> >> I'm<br>
> >> > >> going to sleep.<br>
> >> > >><br>
> >> > >><br>
> >> > >><br>
> >> > >><br>
> >> > >> --<br>
> >> > >> Best regards,<br>
> >> > >> Roman.<br>
> >> > >> _______________________________________________<br>
> >> > >> Gluster-users mailing list<br>
> >> > >> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> >> > >> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> >> > >><br>
> >> > >><br>
> >> > ><br>
> >> > ><br>
> >> > ><br>
> >> > > --<br>
> >> > > Best regards,<br>
> >> > > Roman.<br>
> >> > ><br>
> >> > > _______________________________________________<br>
> >> > > Gluster-users mailing list<br>
> >> > > <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> >> > > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> >> ><br>
> >> ><br>
> >> ><br>
> >> > --<br>
> >> ><br>
> >> > Best regards,<br>
> >> > Roman.<br>
> >> ><br>
> >> ><br>
> >> ><br>
> >> > --<br>
> >> ><br>
> >> > Best regards,<br>
> >> > Roman.<br>
> >> ><br>
> >> > _______________________________________________<br>
> >> > Gluster-users mailing list<br>
> >> > <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> >> > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> >><br>
> >> > _______________________________________________<br>
> >> > Gluster-users mailing list<br>
> >> > <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> >> > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> >><br>
> >><br>
> ><br>
> ><br>
> > --<br>
> > Best regards,<br>
> > Roman.<br>
> ><br>
><br>
><br>
><br>
> --<br>
> Best regards,<br>
> Roman.<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">Best regards,<br>Roman.</div>
</div>