<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hi,</div><div class=""><br class=""></div><div class="">It can be that LIO service starts before /mnt gets mounted. In absence of backend file LIO has created the new one on root filesystem (/mnt directory). Then gluster volume was mounted over, but as backend file was kept open by LIO - it still was used instead of the right one on gluster volume. Then, when you turn off the first node - active path for iSCSI disk switches to the second node (with empty file, placed on root filesystem).</div><br class=""><div><blockquote type="cite" class=""><div class="">18 нояб. 2016 г., в 19:21, Olivier Lambert <<a href="mailto:lambert.olivier@gmail.com" class="">lambert.olivier@gmail.com</a>> написал(а):</div><br class="Apple-interchange-newline"><div class="">After Node 1 is DOWN, LIO on Node2 (iSCSI target) is not writing<br class="">anymore in the local Gluster mount, but in the root partition.<br class=""><br class="">Despite "df -h" shows the Gluster brick mounted:<br class=""><br class="">/dev/mapper/centos-root 3,1G 3,1G 20K 100% /<br class="">...<br class="">/dev/xvdb 61G 61G 956M 99% /bricks/brick1<br class="">localhost:/gv0 61G 61G 956M 99% /mnt<br class=""><br class="">If I unmount it, I still see the "block.img" in /mnt which is filling<br class="">the root space. So it's like Fuse is messing with the local Gluster<br class="">mount, which could lead to the data corruption on the client level.<br class=""><br class="">It doesn't make sense for me... What am I missing?<br class=""><br class="">On Fri, Nov 18, 2016 at 5:00 PM, Olivier Lambert<br class=""><<a href="mailto:lambert.olivier@gmail.com" class="">lambert.olivier@gmail.com</a>> wrote:<br class=""><blockquote type="cite" class="">Yes, I did it only if I have the previous result of heal info ("number<br class="">of entries: 0"). But same result, as soon as the second Node is<br class="">offline (after they were both working/back online), everything is<br class="">corrupted.<br class=""><br class="">To recap:<br class=""><br class="">* Node 1 UP Node 2 UP -> OK<br class="">* Node 1 UP Node 2 DOWN -> OK (just a small lag for multipath to see<br class="">the path down and change if necessary)<br class="">* Node 1 UP Node 2 UP -> OK (and waiting to have no entries displayed<br class="">in heal command)<br class="">* Node 1 DOWN Node 2 UP -> NOT OK (data corruption)<br class=""><br class="">On Fri, Nov 18, 2016 at 3:39 PM, David Gossage<br class=""><<a href="mailto:dgossage@carouselchecks.com" class="">dgossage@carouselchecks.com</a>> wrote:<br class=""><blockquote type="cite" class="">On Fri, Nov 18, 2016 at 3:49 AM, Olivier Lambert <<a href="mailto:lambert.olivier@gmail.com" class="">lambert.olivier@gmail.com</a>><br class="">wrote:<br class=""><blockquote type="cite" class=""><br class="">Hi David,<br class=""><br class="">What are the exact commands to be sure it's fine?<br class=""><br class="">Right now I got:<br class=""><br class=""># gluster volume heal gv0 info<br class="">Brick 10.0.0.1:/bricks/brick1/gv0<br class="">Status: Connected<br class="">Number of entries: 0<br class=""><br class="">Brick 10.0.0.2:/bricks/brick1/gv0<br class="">Status: Connected<br class="">Number of entries: 0<br class=""><br class="">Brick 10.0.0.3:/bricks/brick1/gv0<br class="">Status: Connected<br class="">Number of entries: 0<br class=""><br class=""><br class=""></blockquote>Did you run this before taking down 2nd node to see if any heals were<br class="">ongoing?<br class=""><br class="">Also I see you have sharding enabled. Are your files being served sharded<br class="">already as well?<br class=""><br class=""><blockquote type="cite" class=""><br class="">Everything is online and working, but this command give a strange output:<br class=""><br class=""># gluster volume heal gv0 info heal-failed<br class="">Gathering list of heal failed entries on volume gv0 has been<br class="">unsuccessful on bricks that are down. Please check if all brick<br class="">processes are running.<br class=""><br class="">Is it normal?<br class=""></blockquote><br class=""><br class="">I don't think that is a valid command anymore as whern I run it I get same<br class="">message and this is in logs<br class=""> [2016-11-18 14:35:02.260503] I [MSGID: 106533]<br class="">[glusterd-volume-ops.c:878:__glusterd_handle_cli_heal_volume] 0-management:<br class="">Received heal vol req for volume GLUSTER1<br class="">[2016-11-18 14:35:02.263341] W [MSGID: 106530]<br class="">[glusterd-volume-ops.c:1882:glusterd_handle_heal_cmd] 0-management: Command<br class="">not supported. Please use "gluster volume heal GLUSTER1 info" and logs to<br class="">find the heal information.<br class="">[2016-11-18 14:35:02.263365] E [MSGID: 106301]<br class="">[glusterd-syncop.c:1297:gd_stage_op_phase] 0-management: Staging of<br class="">operation 'Volume Heal' failed on localhost : Command not supported. Please<br class="">use "gluster volume heal GLUSTER1 info" and logs to find the heal<br class="">information.<br class=""><br class=""><blockquote type="cite" class=""><br class="">On Fri, Nov 18, 2016 at 2:51 AM, David Gossage<br class=""><<a href="mailto:dgossage@carouselchecks.com" class="">dgossage@carouselchecks.com</a>> wrote:<br class=""><blockquote type="cite" class=""><br class="">On Thu, Nov 17, 2016 at 6:42 PM, Olivier Lambert<br class=""><<a href="mailto:lambert.olivier@gmail.com" class="">lambert.olivier@gmail.com</a>><br class="">wrote:<br class=""><blockquote type="cite" class=""><br class="">Okay, used the exact same config you provided, and adding an arbiter<br class="">node (node3)<br class=""><br class="">After halting node2, VM continues to work after a small "lag"/freeze.<br class="">I restarted node2 and it was back online: OK<br class=""><br class="">Then, after waiting few minutes, halting node1. And **just** at this<br class="">moment, the VM is corrupted (segmentation fault, /var/log folder empty<br class="">etc.)<br class=""><br class=""></blockquote>Other than waiting a few minutes did you make sure heals had completed?<br class=""><br class=""><blockquote type="cite" class=""><br class="">dmesg of the VM:<br class=""><br class="">[ 1645.852905] EXT4-fs error (device xvda1):<br class="">htree_dirblock_to_tree:988: inode #19: block 8286: comm bash: bad<br class="">entry in directory: rec_len is smaller than minimal - offset=0(0),<br class="">inode=0, rec_len=0, name_len=0<br class="">[ 1645.854509] Aborting journal on device xvda1-8.<br class="">[ 1645.855524] EXT4-fs (xvda1): Remounting filesystem read-only<br class=""><br class="">And got a lot of " comm bash: bad entry in directory" messages then...<br class=""><br class="">Here is the current config with all Node back online:<br class=""><br class=""># gluster volume info<br class=""><br class="">Volume Name: gv0<br class="">Type: Replicate<br class="">Volume ID: 5f15c919-57e3-4648-b20a-395d9fe3d7d6<br class="">Status: Started<br class="">Snapshot Count: 0<br class="">Number of Bricks: 1 x (2 + 1) = 3<br class="">Transport-type: tcp<br class="">Bricks:<br class="">Brick1: 10.0.0.1:/bricks/brick1/gv0<br class="">Brick2: 10.0.0.2:/bricks/brick1/gv0<br class="">Brick3: 10.0.0.3:/bricks/brick1/gv0 (arbiter)<br class="">Options Reconfigured:<br class="">nfs.disable: on<br class="">performance.readdir-ahead: on<br class="">transport.address-family: inet<br class="">features.shard: on<br class="">features.shard-block-size: 16MB<br class="">network.remote-dio: enable<br class="">cluster.eager-lock: enable<br class="">performance.io-cache: off<br class="">performance.read-ahead: off<br class="">performance.quick-read: off<br class="">performance.stat-prefetch: on<br class="">performance.strict-write-ordering: off<br class="">cluster.server-quorum-type: server<br class="">cluster.quorum-type: auto<br class="">cluster.data-self-heal: on<br class=""><br class=""><br class=""># gluster volume status<br class="">Status of volume: gv0<br class="">Gluster process TCP Port RDMA Port Online<br class="">Pid<br class=""><br class=""><br class="">------------------------------------------------------------------------------<br class="">Brick 10.0.0.1:/bricks/brick1/gv0 49152 0 Y<br class="">1331<br class="">Brick 10.0.0.2:/bricks/brick1/gv0 49152 0 Y<br class="">2274<br class="">Brick 10.0.0.3:/bricks/brick1/gv0 49152 0 Y<br class="">2355<br class="">Self-heal Daemon on localhost N/A N/A Y<br class="">2300<br class="">Self-heal Daemon on 10.0.0.3 N/A N/A Y<br class="">10530<br class="">Self-heal Daemon on 10.0.0.2 N/A N/A Y<br class="">2425<br class=""><br class="">Task Status of Volume gv0<br class=""><br class=""><br class="">------------------------------------------------------------------------------<br class="">There are no active volume tasks<br class=""><br class=""><br class=""><br class="">On Thu, Nov 17, 2016 at 11:35 PM, Olivier Lambert<br class=""><<a href="mailto:lambert.olivier@gmail.com" class="">lambert.olivier@gmail.com</a>> wrote:<br class=""><blockquote type="cite" class="">It's planned to have an arbiter soon :) It was just preliminary<br class="">tests.<br class=""><br class="">Thanks for the settings, I'll test this soon and I'll come back to<br class="">you!<br class=""><br class="">On Thu, Nov 17, 2016 at 11:29 PM, Lindsay Mathieson<br class=""><<a href="mailto:lindsay.mathieson@gmail.com" class="">lindsay.mathieson@gmail.com</a>> wrote:<br class=""><blockquote type="cite" class="">On 18/11/2016 8:17 AM, Olivier Lambert wrote:<br class=""><blockquote type="cite" class=""><br class="">gluster volume info gv0<br class=""><br class="">Volume Name: gv0<br class="">Type: Replicate<br class="">Volume ID: 2f8658ed-0d9d-4a6f-a00b-96e9d3470b53<br class="">Status: Started<br class="">Snapshot Count: 0<br class="">Number of Bricks: 1 x 2 = 2<br class="">Transport-type: tcp<br class="">Bricks:<br class="">Brick1: 10.0.0.1:/bricks/brick1/gv0<br class="">Brick2: 10.0.0.2:/bricks/brick1/gv0<br class="">Options Reconfigured:<br class="">nfs.disable: on<br class="">performance.readdir-ahead: on<br class="">transport.address-family: inet<br class="">features.shard: on<br class="">features.shard-block-size: 16MB<br class=""></blockquote><br class=""><br class=""><br class="">When hosting VM's its essential to set these options:<br class=""><br class="">network.remote-dio: enable<br class="">cluster.eager-lock: enable<br class="">performance.io-cache: off<br class="">performance.read-ahead: off<br class="">performance.quick-read: off<br class="">performance.stat-prefetch: on<br class="">performance.strict-write-ordering: off<br class="">cluster.server-quorum-type: server<br class="">cluster.quorum-type: auto<br class="">cluster.data-self-heal: on<br class=""><br class="">Also with replica two and quorum on (required) your volume will<br class="">become<br class="">read-only when one node goes down to prevent the possibility of<br class="">split-brain<br class="">- you *really* want to avoid that :)<br class=""><br class="">I'd recommend a replica 3 volume, that way 1 node can go down, but<br class="">the<br class="">other<br class="">two still form a quorum and will remain r/w.<br class=""><br class="">If the extra disks are not possible, then a Arbiter volume can be<br class="">setup<br class="">-<br class="">basically dummy files on the third node.<br class=""><br class=""><br class=""><br class="">--<br class="">Lindsay Mathieson<br class=""><br class="">_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">http://www.gluster.org/mailman/listinfo/gluster-users<br class=""></blockquote></blockquote>_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">http://www.gluster.org/mailman/listinfo/gluster-users<br class=""></blockquote><br class=""><br class=""></blockquote></blockquote><br class=""><br class=""></blockquote></blockquote>_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">http://www.gluster.org/mailman/listinfo/gluster-users<br class=""></div></blockquote></div><br class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">--</div><div class="">Дмитрий Глушенок</div><div class="">Инфосистемы Джет</div><div class="">+7-910-453-2568</div></div></div></div>
</div>
<br class=""></body></html>