<div dir="ltr">So I've tried using a lot of your script, but I'm still unable to get past the "<span style="font-size:12.8px">Launching heal operation to perform full self heal on volume <volname> has been unsuccessful on bricks that are down. Please check if all brick processes are running." error message. Everything else seems like it's working, but the "gluster volume heal appian full" is never working.</span><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Is there any way to figure out what exactly happened that would cause this error message? The logs don't seem very useful in determining what exactly happened. It seems to just state that it can't seem to "Commit" with the other bricks.</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">When I restart the volume though, it sometimes fixes it, but not sure I want to run a script that constantly restarts the volume until "gluster volume heal appian full" is working.</span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 23, 2016 at 2:21 AM, Heiko L. <span dir="ltr"><<a href="mailto:heikol@fh-lausitz.de" target="_blank">heikol@fh-lausitz.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
hostname not needed<br>
<br>
# nodea=10.1.1.100;bricka=/mnt/sda6/brick4<br>
should be working<br>
<br>
but I prefer like to work with hostnames.<br>
<br>
<br>
regards heiko<br>
<br>
PS i forgot notes:<br>
- xfs,zfs (ext3 work, but partially bad performance (V3.4))<br>
- brickdir should not be topdir of fs<br>
/dev/sda6 /mnt/brick4, brick=/mnt/brick4 -> not recommended<br>
/dev/sda6 /mnt/sda6, brick=/mnt/sda6/brick4 better<br>
<div class="HOEnZb"><div class="h5"><br>
> Thank you for responding, Heiko. In the process of seeing the differences<br>
> between our two scripts. First thing I noticed was that the notes states "need<br>
> to be defined in the /etc/hosts". Would using the IP address directly be a<br>
> problem?<br>
><br>
> On Tue, Jun 21, 2016 at 2:10 PM, Heiko L. <<a href="mailto:heikol@fh-lausitz.de">heikol@fh-lausitz.de</a>> wrote:<br>
><br>
>> Am Di, 21.06.2016, 19:22 schrieb Danny Lee:<br>
>> > Hello,<br>
>> ><br>
>> ><br>
>> > We are currently figuring out how to add GlusterFS to our system to make<br>
>> > our systems highly available using scripts. We are using Gluster 3.7.11.<br>
>> ><br>
>> > Problem:<br>
>> > Trying to migrate to GlusterFS from a non-clustered system to a 3-node<br>
>> > glusterfs replicated cluster using scripts. Tried various things to<br>
>> make this work, but it sometimes causes us to be in an<br>
>> > indesirable state where if you call "gluster volume heal <volname><br>
>> full", we would get the error message, "Launching heal<br>
>> > operation to perform full self heal on volume <volname> has been<br>
>> unsuccessful on bricks that are down. Please check if<br>
>> > all brick processes are running." All the brick processes are running<br>
>> based on running the command, "gluster volume status<br>
>> > volname"<br>
>> ><br>
>> > Things we have tried:<br>
>> > Order of preference<br>
>> > 1. Create Volume with 3 Filesystems with the same data<br>
>> > 2. Create Volume with 2 Empty filesysytems and one with the data<br>
>> > 3. Create Volume with only one filesystem with data and then using<br>
>> > "add-brick" command to add the other two empty filesystems<br>
>> > 4. Create Volume with one empty filesystem, mounting it, and then copying<br>
>> > the data over to that one. And then finally, using "add-brick" command<br>
>> to add the other two empty filesystems<br>
>> - should be working<br>
>> - read each file on /mnt/gvol, to trigger replication [2]<br>
>><br>
>> > 5. Create Volume<br>
>> > with 3 empty filesystems, mounting it, and then copying the data over<br>
>> - my favorite<br>
>><br>
>> ><br>
>> > Other things to note:<br>
>> > A few minutes after the volume is created and started successfully, our<br>
>> > application server starts up against it, so reads and writes may happen<br>
>> pretty quickly after the volume has started. But there<br>
>> > is only about 50MB of data.<br>
>> ><br>
>> > Steps to reproduce (all in a script):<br>
>> > # This is run by the primary node with the IP Adress, <server-ip-1>, that<br>
>> > has data systemctl restart glusterd gluster peer probe <server-ip-2><br>
>> gluster peer probe <server-ip-3> Wait for "gluster peer<br>
>> > status" to all be in "Peer in Cluster" state gluster volume create<br>
>> <volname> replica 3 transport tcp ${BRICKS[0]} ${BRICKS[1]}<br>
>> > ${BRICKS[2]} force<br>
>> > gluster volume set <volname> nfs.disable true gluster volume start<br>
>> <volname> mkdir -p $MOUNT_POINT mount -t glusterfs<br>
>> > <server-ip-1>:/volname $MOUNT_POINT<br>
>> > find $MOUNT_POINT | xargs stat<br>
>><br>
>> I have written a script for 2 nodes. [1]<br>
>> but should be at least 3 nodes.<br>
>><br>
>><br>
>> I hope it helps you<br>
>> regards Heiko<br>
>><br>
>> ><br>
>> > Note that, when we added sleeps around the gluster commands, there was a<br>
>> > higher probability of success, but not 100%.<br>
>> ><br>
>> > # Once volume is started, all the the clients/servers will mount the<br>
>> > gluster filesystem by polling "mountpoint -q $MOUNT_POINT": mkdir -p<br>
>> $MOUNT_POINT mount -t glusterfs <server-ip-1>:/volname<br>
>> > $MOUNT_POINT<br>
>> ><br>
>> ><br>
>> > Logs:<br>
>> > *etc-glusterfs-glusterd.vol.log* in *server-ip-1*<br>
>> ><br>
>> ><br>
>> > [2016-06-21 14:10:38.285234] I [MSGID: 106533]<br>
>> > [glusterd-volume-ops.c:857:__glusterd_handle_cli_heal_volume]<br>
>> 0-management:<br>
>> > Received heal vol req for volume volname<br>
>> > [2016-06-21 14:10:38.296801] E [MSGID: 106153]<br>
>> > [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Commit failed on<br>
>> > <server-ip-2>. Please check log file for details.<br>
>> ><br>
>> ><br>
>> ><br>
>> > *usr-local-volname-data-mirrored-data.log* in *server-ip-1*<br>
>> ><br>
>> ><br>
>> > [2016-06-21 14:14:39.233366] E [MSGID: 114058]<br>
>> > [client-handshake.c:1524:client_query_portmap_cbk] 0-volname-client-0:<br>
>> > failed to get the port number for remote subvolume. Please run 'gluster<br>
>> volume status' on server to see if brick process is<br>
>> > running. *I think this is caused by the self heal daemon*<br>
>> ><br>
>> ><br>
>> > *cmd_history.log* in *server-ip-1*<br>
>> ><br>
>> ><br>
>> > [2016-06-21 14:10:38.298800] : volume heal volname full : FAILED :<br>
>> Commit<br>
>> > failed on <server-ip-2>. Please check log file for details.<br>
>> _______________________________________________<br>
>> > Gluster-users mailing list<br>
>> > <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
>> > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
>><br>
>> [1]<br>
>> <a href="http://www2.fh-lausitz.de/launic/comp/net/glusterfs/130620.glusterfs.create_brick_vol.howto.txt" rel="noreferrer" target="_blank">http://www2.fh-lausitz.de/launic/comp/net/glusterfs/130620.glusterfs.create_brick_vol.howto.txt</a><br>
>> - old, limit 2 nodes<br>
>><br>
>><br>
>> --<br>
>><br>
>><br>
>><br>
><br>
<br>
<br>
</div></div></blockquote></div><br></div>