<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Fri, Nov 18, 2016 at 3:49 AM, Olivier Lambert <span dir="ltr"><<a href="mailto:lambert.olivier@gmail.com" target="_blank">lambert.olivier@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi David,<br>
<br>
What are the exact commands to be sure it's fine?<br>
<br>
Right now I got:<br>
<br>
# gluster volume heal gv0 info<br>
Brick 10.0.0.1:/bricks/brick1/gv0<br>
Status: Connected<br>
Number of entries: 0<br>
<br>
Brick 10.0.0.2:/bricks/brick1/gv0<br>
Status: Connected<br>
Number of entries: 0<br>
<br>
Brick 10.0.0.3:/bricks/brick1/gv0<br>
Status: Connected<br>
Number of entries: 0<br>
<br>
<br></blockquote><div>Did you run this before taking down 2nd node to see if any heals were ongoing?</div><div><br></div><div>Also I see you have sharding enabled. Are your files being served sharded already as well?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Everything is online and working, but this command give a strange output:<br>
<br>
# gluster volume heal gv0 info heal-failed<br>
Gathering list of heal failed entries on volume gv0 has been<br>
unsuccessful on bricks that are down. Please check if all brick<br>
processes are running.<br>
<br>
Is it normal?<br></blockquote><div><br></div><div>I don't think that is a valid command anymore as whern I run it I get same message and this is in logs</div><div> [2016-11-18 14:35:02.260503] I [MSGID: 106533] [glusterd-volume-ops.c:878:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume GLUSTER1</div><div>[2016-11-18 14:35:02.263341] W [MSGID: 106530] [glusterd-volume-ops.c:1882:glusterd_handle_heal_cmd] 0-management: Command not supported. Please use "gluster volume heal GLUSTER1 info" and logs to find the heal information.</div><div>[2016-11-18 14:35:02.263365] E [MSGID: 106301] [glusterd-syncop.c:1297:gd_stage_op_phase] 0-management: Staging of operation 'Volume Heal' failed on localhost : Command not supported. Please use "gluster volume heal GLUSTER1 info" and logs to find the heal information.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
On Fri, Nov 18, 2016 at 2:51 AM, David Gossage<br>
<<a href="mailto:dgossage@carouselchecks.com">dgossage@carouselchecks.com</a>> wrote:<br>
><br>
> On Thu, Nov 17, 2016 at 6:42 PM, Olivier Lambert <<a href="mailto:lambert.olivier@gmail.com">lambert.olivier@gmail.com</a>><br>
> wrote:<br>
>><br>
>> Okay, used the exact same config you provided, and adding an arbiter<br>
>> node (node3)<br>
>><br>
>> After halting node2, VM continues to work after a small "lag"/freeze.<br>
>> I restarted node2 and it was back online: OK<br>
>><br>
>> Then, after waiting few minutes, halting node1. And **just** at this<br>
>> moment, the VM is corrupted (segmentation fault, /var/log folder empty<br>
>> etc.)<br>
>><br>
> Other than waiting a few minutes did you make sure heals had completed?<br>
><br>
>><br>
>> dmesg of the VM:<br>
>><br>
>> [ 1645.852905] EXT4-fs error (device xvda1):<br>
>> htree_dirblock_to_tree:988: inode #19: block 8286: comm bash: bad<br>
>> entry in directory: rec_len is smaller than minimal - offset=0(0),<br>
>> inode=0, rec_len=0, name_len=0<br>
>> [ 1645.854509] Aborting journal on device xvda1-8.<br>
>> [ 1645.855524] EXT4-fs (xvda1): Remounting filesystem read-only<br>
>><br>
>> And got a lot of " comm bash: bad entry in directory" messages then...<br>
>><br>
>> Here is the current config with all Node back online:<br>
>><br>
>> # gluster volume info<br>
>><br>
>> Volume Name: gv0<br>
>> Type: Replicate<br>
>> Volume ID: 5f15c919-57e3-4648-b20a-<wbr>395d9fe3d7d6<br>
>> Status: Started<br>
>> Snapshot Count: 0<br>
>> Number of Bricks: 1 x (2 + 1) = 3<br>
>> Transport-type: tcp<br>
>> Bricks:<br>
>> Brick1: 10.0.0.1:/bricks/brick1/gv0<br>
>> Brick2: 10.0.0.2:/bricks/brick1/gv0<br>
>> Brick3: 10.0.0.3:/bricks/brick1/gv0 (arbiter)<br>
>> Options Reconfigured:<br>
>> nfs.disable: on<br>
>> performance.readdir-ahead: on<br>
>> transport.address-family: inet<br>
>> features.shard: on<br>
>> features.shard-block-size: 16MB<br>
>> network.remote-dio: enable<br>
>> cluster.eager-lock: enable<br>
>> performance.io-cache: off<br>
>> performance.read-ahead: off<br>
>> performance.quick-read: off<br>
>> performance.stat-prefetch: on<br>
>> performance.strict-write-<wbr>ordering: off<br>
>> cluster.server-quorum-type: server<br>
>> cluster.quorum-type: auto<br>
>> cluster.data-self-heal: on<br>
>><br>
>><br>
>> # gluster volume status<br>
>> Status of volume: gv0<br>
>> Gluster process TCP Port RDMA Port Online<br>
>> Pid<br>
>><br>
>> ------------------------------<wbr>------------------------------<wbr>------------------<br>
>> Brick 10.0.0.1:/bricks/brick1/gv0 49152 0 Y<br>
>> 1331<br>
>> Brick 10.0.0.2:/bricks/brick1/gv0 49152 0 Y<br>
>> 2274<br>
>> Brick 10.0.0.3:/bricks/brick1/gv0 49152 0 Y<br>
>> 2355<br>
>> Self-heal Daemon on localhost N/A N/A Y<br>
>> 2300<br>
>> Self-heal Daemon on 10.0.0.3 N/A N/A Y<br>
>> 10530<br>
>> Self-heal Daemon on 10.0.0.2 N/A N/A Y<br>
>> 2425<br>
>><br>
>> Task Status of Volume gv0<br>
>><br>
>> ------------------------------<wbr>------------------------------<wbr>------------------<br>
>> There are no active volume tasks<br>
>><br>
>><br>
>><br>
>> On Thu, Nov 17, 2016 at 11:35 PM, Olivier Lambert<br>
>> <<a href="mailto:lambert.olivier@gmail.com">lambert.olivier@gmail.com</a>> wrote:<br>
>> > It's planned to have an arbiter soon :) It was just preliminary tests.<br>
>> ><br>
>> > Thanks for the settings, I'll test this soon and I'll come back to you!<br>
>> ><br>
>> > On Thu, Nov 17, 2016 at 11:29 PM, Lindsay Mathieson<br>
>> > <<a href="mailto:lindsay.mathieson@gmail.com">lindsay.mathieson@gmail.com</a>> wrote:<br>
>> >> On 18/11/2016 8:17 AM, Olivier Lambert wrote:<br>
>> >>><br>
>> >>> gluster volume info gv0<br>
>> >>><br>
>> >>> Volume Name: gv0<br>
>> >>> Type: Replicate<br>
>> >>> Volume ID: 2f8658ed-0d9d-4a6f-a00b-<wbr>96e9d3470b53<br>
>> >>> Status: Started<br>
>> >>> Snapshot Count: 0<br>
>> >>> Number of Bricks: 1 x 2 = 2<br>
>> >>> Transport-type: tcp<br>
>> >>> Bricks:<br>
>> >>> Brick1: 10.0.0.1:/bricks/brick1/gv0<br>
>> >>> Brick2: 10.0.0.2:/bricks/brick1/gv0<br>
>> >>> Options Reconfigured:<br>
>> >>> nfs.disable: on<br>
>> >>> performance.readdir-ahead: on<br>
>> >>> transport.address-family: inet<br>
>> >>> features.shard: on<br>
>> >>> features.shard-block-size: 16MB<br>
>> >><br>
>> >><br>
>> >><br>
>> >> When hosting VM's its essential to set these options:<br>
>> >><br>
>> >> network.remote-dio: enable<br>
>> >> cluster.eager-lock: enable<br>
>> >> performance.io-cache: off<br>
>> >> performance.read-ahead: off<br>
>> >> performance.quick-read: off<br>
>> >> performance.stat-prefetch: on<br>
>> >> performance.strict-write-<wbr>ordering: off<br>
>> >> cluster.server-quorum-type: server<br>
>> >> cluster.quorum-type: auto<br>
>> >> cluster.data-self-heal: on<br>
>> >><br>
>> >> Also with replica two and quorum on (required) your volume will become<br>
>> >> read-only when one node goes down to prevent the possibility of<br>
>> >> split-brain<br>
>> >> - you *really* want to avoid that :)<br>
>> >><br>
>> >> I'd recommend a replica 3 volume, that way 1 node can go down, but the<br>
>> >> other<br>
>> >> two still form a quorum and will remain r/w.<br>
>> >><br>
>> >> If the extra disks are not possible, then a Arbiter volume can be setup<br>
>> >> -<br>
>> >> basically dummy files on the third node.<br>
>> >><br>
>> >><br>
>> >><br>
>> >> --<br>
>> >> Lindsay Mathieson<br>
>> >><br>
>> >> ______________________________<wbr>_________________<br>
>> >> Gluster-users mailing list<br>
>> >> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
>> >> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
>> ______________________________<wbr>_________________<br>
>> Gluster-users mailing list<br>
>> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
>> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
><br>
><br>
</blockquote></div><br></div></div>