<div dir="ltr"><div>Should be OK. But running the same version on both clients and servers is always the safest bet.<br><br></div>-Krutika<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 22, 2016 at 10:39 AM, qingwei wei <span dir="ltr"><<a href="mailto:tchengwee@gmail.com" target="_blank">tchengwee@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I updated my client to 3.7.14 and i no longer experience split-brain<br>
error. It seems like there is some changes on the client side that fix<br>
this error. Please note that, for this test, i still stick to 3.7.10<br>
server. Will it be an issue with client and server using different<br>
version?<br>
<br>
Cw<br>
<div class="HOEnZb"><div class="h5"><br>
On Tue, Aug 16, 2016 at 5:46 PM, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>> wrote:<br>
> 3.7.11 had quite a few bugs in afr and sharding+afr interop that were fixed<br>
> in 3.7.12.<br>
> Some of them were about files being reported as being in split-brain.<br>
> Chances are that some of them existed in 3.7.10 as well - which is what<br>
> you're using.<br>
><br>
> Do you mind trying the same test with 3.7.12 or a later version?<br>
><br>
> -Krutika<br>
><br>
> On Tue, Aug 16, 2016 at 2:46 PM, qingwei wei <<a href="mailto:tchengwee@gmail.com">tchengwee@gmail.com</a>> wrote:<br>
>><br>
>> Hi Niels,<br>
>><br>
>> My situation is that when i unplug the HDD physically, the FIO<br>
>> application exits with Input/Output error. However, when i do echo<br>
>> offline on the disk, the FIO application does freeze a bit but still<br>
>> manage to resume the IO workload after the freeze.<br>
>><br>
>> From what i can see from the client log, the error is split-brain<br>
>> which does not make sense as i still have 2 working replicas.<br>
>><br>
>> [2016-08-12 10:33:41.854283] E [MSGID:<br>
>> 108008][afr-transaction.c:<wbr>1989:afr_transaction]<br>
>> 0-ad17hwssd7-replicate-0:<br>
>> Failing WRITE on gfid 665a43df-1ece-4c9a-a6ee-<wbr>fcfa960d95bf:split-brain<br>
>> observed. [Input/output error]<br>
>><br>
>> So anyone can share their testing experience on this type disruptive<br>
>> test on shard volume? Thanks!<br>
>><br>
>> Regards,<br>
>><br>
>> Cheng Wee<br>
>><br>
>> On Tue, Aug 16, 2016 at 4:45 PM, Niels de Vos <<a href="mailto:ndevos@redhat.com">ndevos@redhat.com</a>> wrote:<br>
>> > On Tue, Aug 16, 2016 at 01:34:36PM +0800, qingwei wei wrote:<br>
>> >> Hi,<br>
>> >><br>
>> >> I am currently trying to test the distributed replica (3 replicas)<br>
>> >> reliability when 1 brick is down. I tried using both software unplug<br>
>> >> method by issuing the exho offline > /sys/block/sdx/device/state and<br>
>> >> also physically unplug the HDD and i encountered 2 different outcomes.<br>
>> >> For software unplug, the FIO workload continue to run but for<br>
>> >> physically unplug the HDD, FIO workload cannot continue with the<br>
>> >> following error:<br>
>> >><br>
>> >> [2016-08-12 10:33:41.854283] E [MSGID: 108008]<br>
>> >> [afr-transaction.c:1989:afr_<wbr>transaction] 0-ad17hwssd7-replicate-0:<br>
>> >> Failing WRITE on gfid 665a43df-1ece-4c9a-a6ee-<wbr>fcfa960d95bf:<br>
>> >> split-brain observed. [Input/output error]<br>
>> >><br>
>> >> From the server where i unplug the disk, i can see the following:<br>
>> >><br>
>> >> [2016-08-12 10:33:41.916456] D [MSGID: 0]<br>
>> >> [io-threads.c:351:iot_<wbr>schedule] 0-ad17hwssd7-io-threads: LOOKUP<br>
>> >> scheduled as fast fop<br>
>> >> [2016-08-12 10:33:41.916666] D [MSGID: 115050]<br>
>> >> [server-rpc-fops.c:179:server_<wbr>lookup_cbk] 0-ad17hwssd7-server: 8127:<br>
>> >> LOOKUP /.shard/150e99ee-ce3b-4b57-<wbr>8c40-99b4ecdf3822.90<br>
>> >><br>
>> >> (be318638-e8a0-4c6d-977d-<wbr>7a937aa84806/150e99ee-ce3b-<wbr>4b57-8c40-99b4ecdf3822.90)<br>
>> >> ==> (No such file or directory) [No such file or directory]<br>
>> >> [2016-08-12 10:33:41.916804] D [MSGID: 101171]<br>
>> >> [client_t.c:417:gf_client_<wbr>unref] 0-client_t:<br>
>> >><br>
>> >> hp.dctopenstack.org-25780-<wbr>2016/08/12-10:33:07:589960-<wbr>ad17hwssd7-client-0-0-0:<br>
>> >> ref-count 1<br>
>> >> [2016-08-12 10:33:41.917098] D [MSGID: 101171]<br>
>> >> [client_t.c:333:gf_client_ref] 0-client_t:<br>
>> >><br>
>> >> hp.dctopenstack.org-25780-<wbr>2016/08/12-10:33:07:589960-<wbr>ad17hwssd7-client-0-0-0:<br>
>> >> ref-count 2<br>
>> >> [2016-08-12 10:33:41.917145] W [MSGID: 115009]<br>
>> >> [server-resolve.c:571:server_<wbr>resolve] 0-ad17hwssd7-server: no<br>
>> >> resolution type for (null) (LOOKUP)<br>
>> >> [2016-08-12 10:33:41.917182] E [MSGID: 115050]<br>
>> >> [server-rpc-fops.c:179:server_<wbr>lookup_cbk] 0-ad17hwssd7-server: 8128:<br>
>> >> LOOKUP (null)<br>
>> >> (00000000-0000-0000-0000-<wbr>000000000000/150e99ee-ce3b-<wbr>4b57-8c40-99b4ecdf3822.90)<br>
>> >> ==> (Invalid argument) [Invalid argument]<br>
>> >><br>
>> >> I am using gluster 3.7.10 and the configuration is as follow:<br>
>> >><br>
>> >> diagnostics.brick-log-level: DEBUG<br>
>> >> diagnostics.client-log-level: DEBUG<br>
>> >> performance.io-thread-count: 16<br>
>> >> client.event-threads: 2<br>
>> >> server.event-threads: 2<br>
>> >> features.shard-block-size: 16MB<br>
>> >> features.shard: on<br>
>> >> server.allow-insecure: on<br>
>> >> storage.owner-uid: 165<br>
>> >> storage.owner-gid: 165<br>
>> >> nfs.disable: true<br>
>> >> performance.quick-read: off<br>
>> >> performance.io-cache: off<br>
>> >> performance.read-ahead: off<br>
>> >> performance.stat-prefetch: off<br>
>> >> cluster.lookup-optimize: on<br>
>> >> cluster.quorum-type: auto<br>
>> >> cluster.server-quorum-type: server<br>
>> >> transport.address-family: inet<br>
>> >> performance.readdir-ahead: on<br>
>> >><br>
>> >> This error only occur for sharding configuration. Do you guys perform<br>
>> >> this type of test before? Or do you think physically unplug the HDD is<br>
>> >> a valid test case?<br>
>> ><br>
>> > If you use replica-3, things should settle down again. The kernel and<br>
>> > teh brick process needs a little time to find out that the filesystem on<br>
>> > the disk that you pulled out is not responding anymore. The output og<br>
>> > "gluster volume status" should show that the brick process is offline.<br>
>> > As long as you have quorum, things should continue after a small delay<br>
>> > while waiting to mark the brick offline.<br>
>> ><br>
>> > People actually should test this scenario, it can be that power to disks<br>
>> > fail, or even (connections to) RAID-controllers. Hot-unplugging is<br>
>> > definitely a scenario that can emulate real-world problems.<br>
>> ><br>
>> > Niels<br>
>> ______________________________<wbr>_________________<br>
>> Gluster-devel mailing list<br>
>> <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
>> <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br>
><br>
><br>
</div></div></blockquote></div><br></div>