<div dir="ltr"><div><div>HI All,<br><br></div>Thank You for valuable feedback , I will test the suggested solutions, and update the thread.<br><br></div>Regards,<br>Chamara<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 20, 2015 at 4:17 PM, Deepak Shetty <span dir="ltr"><<a href="mailto:dpkshetty@gmail.com" target="_blank">dpkshetty@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>In addition, I would also like to add that i do suspect (just my hunch) that it could be related to multipath.<br></div>If you can try without multipath and if it doesn't re-create, i think that would be a good data point for kernel/OS vendor to debug further.<br><br></div>my 2 cents again :)<br><br>thanx,<br>deepak<br><br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 20, 2015 at 2:32 PM, Niels de Vos <span dir="ltr"><<a href="mailto:ndevos@redhat.com" target="_blank">ndevos@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Tue, Jan 20, 2015 at 11:55:40AM +0530, Deepak Shetty wrote:<br>
> What does "Controller" mean, the openstack controller node or somethign<br>
> else (like HBA ) ?<br>
> You picture says its SAN but the text says multi-path mount.. SAN would<br>
> mean block devices, so I am assuming you have redundant block devices on<br>
> the compute host, mkfs'ing it and then creating bricks for gluster ?<br>
><br>
><br>
> The stack trace looks like you hit a kernel bug and glusterfsd happens to<br>
> be running on the CPU at the time... my 2 cents<br>
<br>
</span>That definitely is a kernel issue. You should contact your OS support<br>
vendor about this.<br>
<br>
The bits you copy/pasted are not sufficient to see what caused it. The<br>
glusterfsd process is just a casualty of the kernel issue, and it is not<br>
likely this can be fixed in Gluster. I suspect you need a kernel<br>
patch/update.<br>
<br>
Niels<br>
<span><br>
><br>
> thanx,<br>
> deepak<br>
><br>
> On Tue, Jan 20, 2015 at 11:29 AM, chamara samarakoon <<a href="mailto:chthsa123@gmail.com" target="_blank">chthsa123@gmail.com</a>><br>
> wrote:<br>
><br>
> > Hi All,<br>
> ><br>
> ><br>
> > We have setup Openstack cloud as below. And the "/va/lib/nova/instances"<br>
> > is a Gluster volume.<br>
> ><br>
> > CentOS - 6.5<br>
> > Kernel - 2.6.32-431.29.2.el6.x86_64<br>
> > GlusterFS - glusterfs 3.5.2 built on Jul 31 2014 18:47:54<br>
> > OpenStack - RDO using Packstack<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > <br>
> ><br>
> ><br>
</span><span>> > Recently Controller node freezes with following error (Which required hard<br>
> > reboot), as a result Gluster volumes on compute node can not reach the<br>
> > controller and due to that all the instances on compute nodes become to<br>
> > read-only status which causes to restart all instances.<br>
> ><br>
> ><br>
> ><br>
> ><br>
</span>> > *BUG: scheduling while atomic : glusterfsd/42725/0xffffffff*<br>
> > *BUG: unable to handle kernel paging request at 0000000038a60d0a8*<br>
> > *IP: [<fffffffff81058e5d>] task_rq_lock+0x4d/0xa0*<br>
> > *PGD 1065525067 PUD 0*<br>
> > *Oops: 0000 [#1] SMP*<br>
> > *last sysfs file :<br>
> > /sys/device/pci0000:80/0000:80:02.0/0000:86:00.0/host2/port-2:0/end_device-2:0/target2:0:0/2:0:0:1/state*<br>
> > *CPU 0*<br>
> > *Modules linked in : xtconntrack iptable_filter ip_tables ipt_REDIRECT<br>
> > fuse ipv openvswitch vxlan iptable_mangle *<br>
<div><div>> ><br>
> > Please advice on above incident , also feedback on the Openstack +<br>
> > GlusterFS setup is appreciated.<br>
> ><br>
> > Thank You,<br>
> > Chamara<br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Gluster-users mailing list<br>
> > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
> ><br>
<br>
<br>
<br>
> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
<br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">chthsa</div>
</div>