<div dir="ltr"><div>A couple weeks ago while upgrading from 3.4 to 3.5 our 12 node cluster (brick on each node - replicated in pairs) one of the nodes spontaneously killed all userland processes either when I ran service glusterfs-server restart or the gluster volume status command immediately after it. (It was pasted as a one liner).</div><div><br></div><div>Today, I had the need to restart glusterfsd on a different node and so had killed glusterfsd and restarted glusterfs-server as a precaution caused the exact same issue - all user land processes were killed (including my ssh connection).</div><div><br></div><div>Has anyone else experience a similar issue or can point to anything to look at to figure out the cause?</div><div><br></div><div>At present nothing stands out in the logs themselves - glustershd.log records up to the event happening and then nothing. Other logs for the bricks and volumes look normal and only show one startup of when I did log back in and restart the services that had been killed.</div><div><br></div><div>System logs don't have any traces - just the notices that services are starting back up (eg. rsyslogd)</div><div><br></div><div>Servers: Ubuntu 14.04 using the GlusterFS ppa:</div><div><br></div><div><div>Volume Name: volumes</div><div>Type: Distributed-Replicate</div><div>Volume ID: 4dbcd901-af7a-437b-90c7-6b45def03748</div><div>Status: Started</div><div>Number of Bricks: 6 x 2 = 12</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: yyc-c01:/openstack_zfs/volumes/gluster</div><div>Brick2: yyc-c02:/openstack_zfs/volumes/gluster</div><div>Brick3: yyc-c03:/openstack_zfs/volumes/gluster</div><div>Brick4: yyc-c04:/openstack_zfs/volumes/gluster</div><div>Brick5: yyc-c05:/openstack_zfs/volumes/gluster</div><div>Brick6: yyc-c06:/openstack_zfs/volumes/gluster</div><div>Brick7: yyc-c07:/openstack_zfs/volumes/gluster</div><div>Brick8: yyc-c08:/openstack_zfs/volumes/gluster</div><div>Brick9: yyc-c09:/openstack_zfs/volumes/gluster</div><div>Brick10: yyc-c10:/openstack_zfs/volumes/gluster</div><div>Brick11: yyc-c11:/openstack_zfs/volumes/gluster</div><div>Brick12: yyc-c12:/openstack_zfs/volumes/gluster</div></div><div><br></div><div>Thanks,</div><div><br></div><div>-- Micheal</div>
</div>