<div dir="ltr"><div><div>This should work without any issues. It is possible that the shard(s)<br>would get created with different gfids but the ones on the lagging brick<br>will eventually (by the time heal-info returns all zeroes) get replaced<br>with shards having the correct gfids.<br></div><div><br></div>Have you tried it yet? Did you face any issues?<br><br></div>-Krutika<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 27, 2016 at 3:48 PM, qingwei wei <span dir="ltr"><<a href="mailto:tchengwee@gmail.com" target="_blank">tchengwee@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
My final goal of the test is to see the impact of brick replacement<br>
while IO is till running.<br>
<br>
One scenario that i think of is as below:<br>
<br>
1. random read IO is performed on gluster volume (3 replicas)<br>
2. 1 brick down and IO still ongoing<br>
3. Perform brick replacement and IO still ongoing<br>
4. There will be a full heal on the new brick while IO still ongoing.<br>
<br>
Assume we have large number of files, reconstruct (recreate shard<br>
file) could take some time on this new brick. So will it be possible<br>
that some random read IO to yet created shard trigger the similar<br>
error?<br>
<br>
Thanks.<br>
<br>
Cwtan<br>
<div class="HOEnZb"><div class="h5"><br>
On Thu, Oct 27, 2016 at 4:26 PM, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>> wrote:<br>
> Found the RC. The problem seems to be that sharding translator attempts to<br>
> create<br>
> non-existent shards in read/write codepaths with a newly generated gfid<br>
> attached<br>
> to the create request in case the shard is absent. Replicate translator,<br>
> which sits below<br>
> sharding on the stack takes this request and plays it on all of its<br>
> replicas. On two of them it<br>
> fails with EEXIST, and on the one where the shards were removed from the<br>
> backend, the<br>
> shard path is created but with the newly generated gfid while the other two<br>
> replicas continue to<br>
> hold the original gfid (the one prior to rm -rf). Although this can be<br>
> fixed, it will require one<br>
> additional lookup for each shard for each read/write operation, causing the<br>
> latency of the read/write<br>
> response to the application to increase by a factor of 1 network call.<br>
><br>
> The test you're doing is partially (but not fully) manipulating and removing<br>
> data from the backend,<br>
> which is not recommended.<br>
><br>
> My question to you is this - what is the specific failure that you are<br>
> trying to simulate with removal of<br>
> contents of .shard? Normally, the `rm -rf on backend` type of tests are<br>
> performed to simulate disk<br>
> failure and its replacement with a brand new disk, in which case executing<br>
> the replace-brick/reset-brick<br>
> commands should be sufficient to recover all contents from the remaining two<br>
> replicas.<br>
><br>
> -Krutika<br>
><br>
> On Thu, Oct 27, 2016 at 12:49 PM, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>><br>
> wrote:<br>
>><br>
>> Now it's reproducible, thanks. :)<br>
>><br>
>> I think I know the RC. Let me confirm it through tests and report back.<br>
>><br>
>> -Krutika<br>
>><br>
>> On Thu, Oct 27, 2016 at 10:42 AM, qingwei wei <<a href="mailto:tchengwee@gmail.com">tchengwee@gmail.com</a>> wrote:<br>
>>><br>
>>> Hi,<br>
>>><br>
>>> I did few more test runs and it seems that it happens during this<br>
>>> sequence<br>
>>><br>
>>> 1.populate data using dd<br>
>>> 2. delete away ALL the shard files in one of the brick .shard folder<br>
>>> 3. Trying to access using dd, no error reported<br>
>>> 4. umount and mount.<br>
>>> 5. Trying to access using dd, no error reported<br>
>>> 6. umount and mount.<br>
>>> 7. Trying to access using dd and Input/Output error reported<br>
>>><br>
>>> during step 3 and 4, no file is created under the .shard directory<br>
>>> For step 7, a shard file is created with same file name but different<br>
>>> gfid compare to other good replicas.<br>
>>><br>
>>> Below is the client log and brick log with more details in the attached<br>
>>> log.<br>
>>><br>
>>> Client log<br>
>>><br>
>>> [2016-10-27 04:34:46.493281] D [MSGID: 0]<br>
>>> [shard.c:3138:shard_common_<wbr>mknod_cbk] 0-testHeal4-shard: mknod of<br>
>>> shard 1 failed: File exists<br>
>>> [2016-10-27 04:34:46.493351] D [MSGID: 0]<br>
>>> [dht-common.c:2633:dht_lookup] 0-testHeal4-dht: Calling fresh lookup<br>
>>> for /.shard/76bc4b0f-bb18-4736-<wbr>8327-99098cd0d7ce.1 on<br>
>>> testHeal4-replicate-0<br>
>>> [2016-10-27 04:34:46.494646] W [MSGID: 114031]<br>
>>> [client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal4-client-0:<br>
>>> remote operation failed. Path: (null)<br>
>>> (00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
>>> [2016-10-27 04:34:46.494673] D [MSGID: 0]<br>
>>> [client-rpc-fops.c:2989:<wbr>client3_3_lookup_cbk] 0-stack-trace:<br>
>>> stack-address: 0x7f9083edc1c8, testHeal4-client-0 returned -1 error:<br>
>>> Invalid argument [Invalid argument]<br>
>>> [2016-10-27 04:34:46.494705] W [MSGID: 114031]<br>
>>> [client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal4-client-1:<br>
>>> remote operation failed. Path: (null)<br>
>>> (00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
>>> [2016-10-27 04:34:46.494710] W [MSGID: 114031]<br>
>>> [client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal4-client-2:<br>
>>> remote operation failed. Path: (null)<br>
>>> (00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
>>> [2016-10-27 04:34:46.494730] D [MSGID: 0]<br>
>>> [client-rpc-fops.c:2989:<wbr>client3_3_lookup_cbk] 0-stack-trace:<br>
>>> stack-address: 0x7f9083edc1c8, testHeal4-client-1 returned -1 error:<br>
>>> Invalid argument [Invalid argument]<br>
>>> [2016-10-27 04:34:46.494751] D [MSGID: 0]<br>
>>> [client-rpc-fops.c:2989:<wbr>client3_3_lookup_cbk] 0-stack-trace:<br>
>>> stack-address: 0x7f9083edc1c8, testHeal4-client-2 returned -1 error:<br>
>>> Invalid argument [Invalid argument]<br>
>>> [2016-10-27 04:34:46.495339] D [MSGID: 0]<br>
>>> [afr-common.c:1986:afr_lookup_<wbr>done] 0-stack-trace: stack-address:<br>
>>> 0x7f9083edbb1c, testHeal4-replicate-0 returned -1 error: Input/output<br>
>>> error [Input/output error]<br>
>>> [2016-10-27 04:34:46.495364] D [MSGID: 0]<br>
>>> [dht-common.c:2220:dht_lookup_<wbr>cbk] 0-testHeal4-dht: fresh_lookup<br>
>>> returned for /.shard/76bc4b0f-bb18-4736-<wbr>8327-99098cd0d7ce.1 with<br>
>>> op_ret -1 [Input/output error]<br>
>>> [2016-10-27 04:34:46.495374] D [MSGID: 0]<br>
>>> [dht-common.c:2300:dht_lookup_<wbr>cbk] 0-testHeal4-dht: Lookup of<br>
>>> /.shard/76bc4b0f-bb18-4736-<wbr>8327-99098cd0d7ce.1 for subvolume<br>
>>> testHeal4-replicate-0 failed [Input/output error]<br>
>>> [2016-10-27 04:34:46.495384] D [MSGID: 0]<br>
>>> [dht-common.c:2363:dht_lookup_<wbr>cbk] 0-stack-trace: stack-address:<br>
>>> 0x7f9083edbb1c, testHeal4-dht returned -1 error: Input/output error<br>
>>> [Input/output error]<br>
>>> [2016-10-27 04:34:46.495395] E [MSGID: 133010]<br>
>>> [shard.c:1582:shard_common_<wbr>lookup_shards_cbk] 0-testHeal4-shard:<br>
>>> Lookup on shard 1 failed. Base file gfid =<br>
>>> 76bc4b0f-bb18-4736-8327-<wbr>99098cd0d7ce [Input/output error]<br>
>>> [2016-10-27 04:34:46.495406] D [MSGID: 0]<br>
>>> [shard.c:3086:shard_post_<wbr>lookup_shards_readv_handler] 0-stack-trace:<br>
>>> stack-address: 0x7f9083edbb1c, testHeal4-shard returned -1 error:<br>
>>> Input/output error [Input/output error]<br>
>>> [2016-10-27 04:34:46.495417] D [MSGID: 0]<br>
>>> [defaults.c:1010:default_<wbr>readv_cbk] 0-stack-trace: stack-address:<br>
>>> 0x7f9083edbb1c, testHeal4-write-behind returned -1 error: Input/output<br>
>>> error [Input/output error]<br>
>>> [2016-10-27 04:34:46.495428] D [MSGID: 0]<br>
>>> [read-ahead.c:462:ra_readv_<wbr>disabled_cbk] 0-stack-trace: stack-address:<br>
>>> 0x7f9083edbb1c, testHeal4-read-ahead returned -1 error: Input/output<br>
>>> error [Input/output error]<br>
>>><br>
>>> brick log<br>
>>><br>
>>> [2016-10-27 04:34:46.492055] D [MSGID: 0]<br>
>>> [io-threads.c:351:iot_<wbr>schedule] 0-testHeal4-io-threads: STATFS<br>
>>> scheduled as fast fop<br>
>>> [2016-10-27 04:34:46.492157] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_entrylk+0x93)<br>
>>> [0x7efebb37d633]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 3<br>
>>> [2016-10-27 04:34:46.492180] D [MSGID: 0]<br>
>>> [io-threads.c:351:iot_<wbr>schedule] 0-testHeal4-io-threads: ENTRYLK<br>
>>> scheduled as normal fop<br>
>>> [2016-10-27 04:34:46.492239] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_statfs_cbk+0x112)<br>
>>> [0x7efebb36f8e2]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.492271] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_entrylk_cbk+0xa6)<br>
>>> [0x7efebb3713a6]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.492535] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_mknod+0x80)<br>
>>> [0x7efebb37b690]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.492565] D [MSGID: 0]<br>
>>> [io-threads.c:351:iot_<wbr>schedule] 0-testHeal4-io-threads: MKNOD<br>
>>> scheduled as normal fop<br>
>>> [2016-10-27 04:34:46.492843] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_mknod_cbk+0x5ad)<br>
>>> [0x7efebb383c9d]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.492981] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_xattrop+0x86)<br>
>>> [0x7efebb3789d6]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.493056] D [MSGID: 0]<br>
>>> [io-threads.c:351:iot_<wbr>schedule] 0-testHeal4-io-threads: XATTROP<br>
>>> scheduled as slow fop<br>
>>> [2016-10-27 04:34:46.493128] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_entrylk+0x93)<br>
>>> [0x7efebb37d633]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 3<br>
>>> [2016-10-27 04:34:46.493148] D [MSGID: 0]<br>
>>> [io-threads.c:351:iot_<wbr>schedule] 0-testHeal4-io-threads: ENTRYLK<br>
>>> scheduled as normal fop<br>
>>> [2016-10-27 04:34:46.493214] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_xattrop_cbk+0xd9)<br>
>>> [0x7efebb370579]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.493239] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_entrylk_cbk+0xa6)<br>
>>> [0x7efebb3713a6]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.493490] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_lookup+0x8b)<br>
>>> [0x7efebb386beb]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.493514] D [MSGID: 0]<br>
>>> [io-threads.c:351:iot_<wbr>schedule] 0-testHeal4-io-threads: LOOKUP<br>
>>> scheduled as fast fop<br>
>>> [2016-10-27 04:34:46.493666] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_lookup_cbk+0x548)<br>
>>> [0x7efebb3864c8]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.493782] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_lookup+0x8b)<br>
>>> [0x7efebb386beb]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.493986] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_lookup_cbk+0x548)<br>
>>> [0x7efebb3864c8]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.494596] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_lookup+0x8b)<br>
>>> [0x7efebb386beb]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.494616] D [logging.c:1954:_gf_msg_<wbr>internal]<br>
>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.<br>
>>> About to flush least recently used log message to disk<br>
>>> [2016-10-27 04:34:46.493818] D [MSGID: 0]<br>
>>> [io-threads.c:351:iot_<wbr>schedule] 0-testHeal4-io-threads: LOOKUP<br>
>>> scheduled as fast fop<br>
>>> [2016-10-27 04:34:46.494616] W [MSGID: 115009]<br>
>>> [server-resolve.c:574:server_<wbr>resolve] 0-testHeal4-server: no<br>
>>> resolution type for (null) (LOOKUP)<br>
>>> [2016-10-27 04:34:46.494650] E [MSGID: 115050]<br>
>>> [server-rpc-fops.c:179:server_<wbr>lookup_cbk] 0-testHeal4-server: 29:<br>
>>> LOOKUP (null)<br>
>>> (00000000-0000-0000-0000-<wbr>000000000000/76bc4b0f-bb18-<wbr>4736-8327-99098cd0d7ce.1)<br>
>>> ==> (Invalid argument) [Invalid argument]<br>
>>> [2016-10-27 04:34:46.494720] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_lookup_cbk+0x548)<br>
>>> [0x7efebb3864c8]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.494936] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_lookup+0x8b)<br>
>>> [0x7efebb386beb]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.494967] D [MSGID: 0]<br>
>>> [io-threads.c:351:iot_<wbr>schedule] 0-testHeal4-io-threads: LOOKUP<br>
>>> scheduled as fast fop<br>
>>> [2016-10-27 04:34:46.495108] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_lookup_cbk+0x548)<br>
>>> [0x7efebb3864c8]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.595813] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_lookup+0x8b)<br>
>>> [0x7efebb386beb]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.595915] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_lookup+0x8b)<br>
>>> [0x7efebb386beb]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 3<br>
>>> [2016-10-27 04:34:46.596054] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_lookup_cbk+0x548)<br>
>>> [0x7efebb3864c8]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.596162] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_lookup_cbk+0x548)<br>
>>> [0x7efebb3864c8]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.596427] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_entrylk+0x93)<br>
>>> [0x7efebb37d633]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.596453] D [logging.c:1954:_gf_msg_<wbr>internal]<br>
>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.<br>
>>> About to flush least recently used log message to disk<br>
>>> The message "D [MSGID: 0] [io-threads.c:351:iot_<wbr>schedule]<br>
>>> 0-testHeal4-io-threads: LOOKUP scheduled as fast fop" repeated 2 times<br>
>>> between [2016-10-27 04:34:46.494967] and [2016-10-27 04:34:46.595944]<br>
>>> [2016-10-27 04:34:46.596453] D [MSGID: 0]<br>
>>> [io-threads.c:351:iot_<wbr>schedule] 0-testHeal4-io-threads: ENTRYLK<br>
>>> scheduled as normal fop<br>
>>> [2016-10-27 04:34:46.596551] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_entrylk+0x93)<br>
>>> [0x7efebb37d633]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 3<br>
>>> [2016-10-27 04:34:46.596603] D [logging.c:1954:_gf_msg_<wbr>internal]<br>
>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.<br>
>>> About to flush least recently used log message to disk<br>
>>> [2016-10-27 04:34:46.596611] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_entrylk_cbk+0xa6)<br>
>>> [0x7efebb3713a6]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.596570] D [MSGID: 0]<br>
>>> [io-threads.c:351:iot_<wbr>schedule] 0-testHeal4-io-threads: ENTRYLK<br>
>>> scheduled as normal fop<br>
>>> [2016-10-27 04:34:46.596602] D [MSGID: 0]<br>
>>> [entrylk.c:701:pl_common_<wbr>entrylk] 0-stack-trace: stack-address:<br>
>>> 0x7efecd3db738, testHeal4-locks returned -1 error: Resource<br>
>>> temporarily unavailable [Resource temporarily unavailable]<br>
>>> [2016-10-27 04:34:46.596650] D [MSGID: 0]<br>
>>> [defaults.c:1196:default_<wbr>entrylk_cbk] 0-stack-trace: stack-address:<br>
>>> 0x7efecd3db738, testHeal4-io-threads returned -1 error: Resource<br>
>>> temporarily unavailable [Resource temporarily unavailable]<br>
>>> [2016-10-27 04:34:46.596664] D [MSGID: 0]<br>
>>> [io-stats.c:1811:io_stats_<wbr>entrylk_cbk] 0-stack-trace: stack-address:<br>
>>> 0x7efecd3db738, /mnt/sdc_mssd/testHeal4 returned -1 error: Resource<br>
>>> temporarily unavailable [Resource temporarily unavailable]<br>
>>> [2016-10-27 04:34:46.596676] D [MSGID: 115054]<br>
>>> [server-rpc-fops.c:350:server_<wbr>entrylk_cbk] 0-testHeal4-server: 34:<br>
>>> ENTRYLK /.shard (be318638-e8a0-4c6d-977d-<wbr>7a937aa84806) ==> (Resource<br>
>>> temporarily unavailable) [Resource temporarily unavailable]<br>
>>> [2016-10-27 04:34:46.596764] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_entrylk_cbk+0xa6)<br>
>>> [0x7efebb3713a6]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.596791] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_mknod+0x80)<br>
>>> [0x7efebb37b690]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.596813] D [MSGID: 0]<br>
>>> [server-resolve.c:330:resolve_<wbr>entry_simple] 0-testHeal4-server: inode<br>
>>> (pointer: 0x7efea1aebaa0 gfid:041e3b34-14c2-4bb1-82e2-<wbr>db352232c3cf<br>
>>> found for path ((null)) while type is RESOLVE_NOT<br>
>>> [2016-10-27 04:34:46.596828] D [MSGID: 115057]<br>
>>> [server-rpc-fops.c:563:server_<wbr>mknod_cbk] 0-testHeal4-server: 35: MKNOD<br>
>>> (null)<br>
>>> (be318638-e8a0-4c6d-977d-<wbr>7a937aa84806/76bc4b0f-bb18-<wbr>4736-8327-99098cd0d7ce.1)<br>
>>> ==> (File exists) [File exists]<br>
>>> [2016-10-27 04:34:46.596896] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_mknod_cbk+0x5ad)<br>
>>> [0x7efebb383c9d]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.597174] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_entrylk+0x93)<br>
>>> [0x7efebb37d633]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.597199] D [MSGID: 0]<br>
>>> [io-threads.c:351:iot_<wbr>schedule] 0-testHeal4-io-threads: ENTRYLK<br>
>>> scheduled as normal fop<br>
>>> [2016-10-27 04:34:46.597289] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_entrylk_cbk+0xa6)<br>
>>> [0x7efebb3713a6]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.597396] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_entrylk+0x93)<br>
>>> [0x7efebb37d633]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.597571] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_entrylk_cbk+0xa6)<br>
>>> [0x7efebb3713a6]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.597604] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_lookup+0x8b)<br>
>>> [0x7efebb386beb]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.597632] D [logging.c:1954:_gf_msg_<wbr>internal]<br>
>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.<br>
>>> About to flush least recently used log message to disk<br>
>>> [2016-10-27 04:34:46.597415] D [MSGID: 0]<br>
>>> [io-threads.c:351:iot_<wbr>schedule] 0-testHeal4-io-threads: ENTRYLK<br>
>>> scheduled as normal fop<br>
>>> [2016-10-27 04:34:46.597632] D [MSGID: 0]<br>
>>> [io-threads.c:351:iot_<wbr>schedule] 0-testHeal4-io-threads: LOOKUP<br>
>>> scheduled as fast fop<br>
>>> [2016-10-27 04:34:46.597864] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_lookup_cbk+0x548)<br>
>>> [0x7efebb3864c8]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.598116] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_mknod+0x80)<br>
>>> [0x7efebb37b690]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.598134] D [MSGID: 0]<br>
>>> [server-resolve.c:330:resolve_<wbr>entry_simple] 0-testHeal4-server: inode<br>
>>> (pointer: 0x7efea1aebaa0 gfid:041e3b34-14c2-4bb1-82e2-<wbr>db352232c3cf<br>
>>> found for path ((null)) while type is RESOLVE_NOT<br>
>>> [2016-10-27 04:34:46.598147] D [MSGID: 115057]<br>
>>> [server-rpc-fops.c:563:server_<wbr>mknod_cbk] 0-testHeal4-server: 39: MKNOD<br>
>>> (null)<br>
>>> (be318638-e8a0-4c6d-977d-<wbr>7a937aa84806/76bc4b0f-bb18-<wbr>4736-8327-99098cd0d7ce.1)<br>
>>> ==> (File exists) [File exists]<br>
>>> [2016-10-27 04:34:46.598205] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_mknod_cbk+0x5ad)<br>
>>> [0x7efebb383c9d]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.598258] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_lookup+0x8b)<br>
>>> [0x7efebb386beb]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.598301] D [MSGID: 0]<br>
>>> [io-threads.c:351:iot_<wbr>schedule] 0-testHeal4-io-threads: LOOKUP<br>
>>> scheduled as fast fop<br>
>>> [2016-10-27 04:34:46.598580] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_lookup_cbk+0x548)<br>
>>> [0x7efebb3864c8]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.598599] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_entrylk+0x93)<br>
>>> [0x7efebb37d633]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.598619] D [MSGID: 0]<br>
>>> [io-threads.c:351:iot_<wbr>schedule] 0-testHeal4-io-threads: ENTRYLK<br>
>>> scheduled as normal fop<br>
>>> [2016-10-27 04:34:46.598754] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_entrylk_cbk+0xa6)<br>
>>> [0x7efebb3713a6]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.598921] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_lookup+0x8b)<br>
>>> [0x7efebb386beb]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.598938] W [MSGID: 115009]<br>
>>> [server-resolve.c:574:server_<wbr>resolve] 0-testHeal4-server: no<br>
>>> resolution type for (null) (LOOKUP)<br>
>>> [2016-10-27 04:34:46.598951] E [MSGID: 115050]<br>
>>> [server-rpc-fops.c:179:server_<wbr>lookup_cbk] 0-testHeal4-server: 42:<br>
>>> LOOKUP (null)<br>
>>> (00000000-0000-0000-0000-<wbr>000000000000/76bc4b0f-bb18-<wbr>4736-8327-99098cd0d7ce.1)<br>
>>> ==> (Invalid argument) [Invalid argument]<br>
>>> [2016-10-27 04:34:46.599007] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_lookup_cbk+0x548)<br>
>>> [0x7efebb3864c8]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.599059] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_lookup+0x8b)<br>
>>> [0x7efebb386beb]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.599081] D [MSGID: 0]<br>
>>> [io-threads.c:351:iot_<wbr>schedule] 0-testHeal4-io-threads: LOOKUP<br>
>>> scheduled as fast fop<br>
>>> [2016-10-27 04:34:46.599215] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_lookup+0x8b)<br>
>>> [0x7efebb386beb]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 3<br>
>>> [2016-10-27 04:34:46.599379] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_lookup_cbk+0x548)<br>
>>> [0x7efebb3864c8]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.599412] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_lookup+0x8b)<br>
>>> [0x7efebb386beb]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 3<br>
>>> [2016-10-27 04:34:46.599505] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_lookup_cbk+0x548)<br>
>>> [0x7efebb3864c8]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.599584] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_lookup_cbk+0x548)<br>
>>> [0x7efebb3864c8]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.599783] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_lookup+0x8b)<br>
>>> [0x7efebb386beb]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.599807] D [logging.c:1954:_gf_msg_<wbr>internal]<br>
>>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.<br>
>>> About to flush least recently used log message to disk<br>
>>> The message "D [MSGID: 0] [io-threads.c:351:iot_<wbr>schedule]<br>
>>> 0-testHeal4-io-threads: LOOKUP scheduled as fast fop" repeated 2 times<br>
>>> between [2016-10-27 04:34:46.599081] and [2016-10-27 04:34:46.599439]<br>
>>> [2016-10-27 04:34:46.599806] W [MSGID: 115009]<br>
>>> [server-resolve.c:574:server_<wbr>resolve] 0-testHeal4-server: no<br>
>>> resolution type for (null) (LOOKUP)<br>
>>> [2016-10-27 04:34:46.599833] E [MSGID: 115050]<br>
>>> [server-rpc-fops.c:179:server_<wbr>lookup_cbk] 0-testHeal4-server: 46:<br>
>>> LOOKUP (null)<br>
>>> (00000000-0000-0000-0000-<wbr>000000000000/76bc4b0f-bb18-<wbr>4736-8327-99098cd0d7ce.1)<br>
>>> ==> (Invalid argument) [Invalid argument]<br>
>>> [2016-10-27 04:34:46.599893] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_lookup_cbk+0x548)<br>
>>> [0x7efebb3864c8]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:46.600183] D [client_t.c:333:gf_client_ref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server3_3_lookup+0x8b)<br>
>>> [0x7efebb386beb]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>get_frame_from_request+0x257)<br>
>>> [0x7efebb36cfd7] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x68)<br>
>>> [0x7efecfadf608] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 2<br>
>>> [2016-10-27 04:34:46.600206] D [MSGID: 0]<br>
>>> [io-threads.c:351:iot_<wbr>schedule] 0-testHeal4-io-threads: LOOKUP<br>
>>> scheduled as fast fop<br>
>>> [2016-10-27 04:34:46.600336] D [client_t.c:417:gf_client_<wbr>unref]<br>
>>><br>
>>> (-->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_lookup_cbk+0x548)<br>
>>> [0x7efebb3864c8]<br>
>>><br>
>>> -->/usr/lib64/glusterfs/3.7.<wbr>16/xlator/protocol/server.so(<wbr>server_submit_reply+0x123)<br>
>>> [0x7efebb368f13] -->/lib64/libglusterfs.so.0(<wbr>gf_client_unref+0x77)<br>
>>> [0x7efecfadf787] ) 0-client_t:<br>
>>><br>
>>> fujitsu05.dctopenstack.org-<wbr>6064-2016/10/27-04:34:44:<wbr>217958-testHeal4-client-1-0-0:<br>
>>> ref-count 1<br>
>>> [2016-10-27 04:34:59.343124] D<br>
>>> [logging.c:1830:gf_log_flush_<wbr>timeout_cbk] 0-logging-infra: Log timer<br>
>>> timed out. About to flush outstanding messages if present<br>
>>> [2016-10-27 04:34:59.343202] D<br>
>>> [logging.c:1792:__gf_log_<wbr>inject_timer_event] 0-logging-infra: Starting<br>
>>> timer now. Timeout = 120, current buf size = 5<br>
>>><br>
>>><br>
>>> Thanks.<br>
>>><br>
>>> Regards,<br>
>>><br>
>>> Cwtan<br>
>>><br>
>>> On Wed, Oct 26, 2016 at 8:09 PM, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>><br>
>>> wrote:<br>
>>> > Do you also have the brick logs? Looks like the bricks are returning<br>
>>> > EINVAL<br>
>>> > on lookup<br>
>>> > which AFR is subsequently converting into an EIO. And sharding is<br>
>>> > merely<br>
>>> > delivering the same error code upwards.<br>
>>> ><br>
>>> > -Krutika<br>
>>> ><br>
>>> > On Wed, Oct 26, 2016 at 6:38 AM, qingwei wei <<a href="mailto:tchengwee@gmail.com">tchengwee@gmail.com</a>><br>
>>> > wrote:<br>
>>> >><br>
>>> >> Hi,<br>
>>> >><br>
>>> >> Pls see the client log below.<br>
>>> >><br>
>>> >> [2016-10-24 10:29:51.111603] I [fuse-bridge.c:5171:fuse_<wbr>graph_setup]<br>
>>> >> 0-fuse: switched to graph 0<br>
>>> >> [2016-10-24 10:29:51.111662] I [MSGID: 114035]<br>
>>> >> [client-handshake.c:193:<wbr>client_set_lk_version_cbk]<br>
>>> >> 0-testHeal-client-2: Server lk version = 1<br>
>>> >> [2016-10-24 10:29:51.112371] I [fuse-bridge.c:4083:fuse_init]<br>
>>> >> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22<br>
>>> >> kernel 7.22<br>
>>> >> [2016-10-24 10:29:51.113563] I [MSGID: 108031]<br>
>>> >> [afr-common.c:2071:afr_local_<wbr>discovery_cbk] 0-testHeal-replicate-0:<br>
>>> >> selecting local read_child testHeal-client-2<br>
>>> >> [2016-10-24 10:29:51.113604] I [MSGID: 108031]<br>
>>> >> [afr-common.c:2071:afr_local_<wbr>discovery_cbk] 0-testHeal-replicate-0:<br>
>>> >> selecting local read_child testHeal-client-0<br>
>>> >> [2016-10-24 10:29:51.113630] I [MSGID: 108031]<br>
>>> >> [afr-common.c:2071:afr_local_<wbr>discovery_cbk] 0-testHeal-replicate-0:<br>
>>> >> selecting local read_child testHeal-client-1<br>
>>> >> [2016-10-24 10:29:54.016802] W [MSGID: 108001]<br>
>>> >> [afr-transaction.c:789:afr_<wbr>handle_quorum] 0-testHeal-replicate-0:<br>
>>> >> /.shard/9061198a-eb7e-45a2-<wbr>93fb-eb396d1b2727.1: F<br>
>>> >> ailing MKNOD as quorum is not met<br>
>>> >> [2016-10-24 10:29:54.019330] W [MSGID: 114031]<br>
>>> >> [client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-0:<br>
>>> >> remote operation failed. Path: (null) (00000000-<br>
>>> >> 0000-0000-0000-000000000000) [Invalid argument]<br>
>>> >> [2016-10-24 10:29:54.019343] W [MSGID: 114031]<br>
>>> >> [client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-2:<br>
>>> >> remote operation failed. Path: (null) (00000000-<br>
>>> >> 0000-0000-0000-000000000000) [Invalid argument]<br>
>>> >> [2016-10-24 10:29:54.019373] W [MSGID: 114031]<br>
>>> >> [client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-1:<br>
>>> >> remote operation failed. Path: (null) (00000000-<br>
>>> >> 0000-0000-0000-000000000000) [Invalid argument]<br>
>>> >> [2016-10-24 10:29:54.019854] E [MSGID: 133010]<br>
>>> >> [shard.c:1582:shard_common_<wbr>lookup_shards_cbk] 0-testHeal-shard: Lookup<br>
>>> >> on shard 1 failed. Base file gfid = 9061198a<br>
>>> >> -eb7e-45a2-93fb-eb396d1b2727 [Input/output error]<br>
>>> >> [2016-10-24 10:29:54.020886] W [fuse-bridge.c:2227:fuse_<wbr>readv_cbk]<br>
>>> >> 0-glusterfs-fuse: 135: READ => -1<br>
>>> >> gfid=9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727 fd=0x7f70c80d12dc (<br>
>>> >> Input/output error)<br>
>>> >> [2016-10-24 10:29:54.118264] W [MSGID: 114031]<br>
>>> >> [client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-0:<br>
>>> >> remote operation failed. Path: (null) (00000000-<br>
>>> >> 0000-0000-0000-000000000000) [Invalid argument]<br>
>>> >> [2016-10-24 10:29:54.118308] W [MSGID: 114031]<br>
>>> >> [client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-2:<br>
>>> >> remote operation failed. Path: (null)<br>
>>> >> (00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
>>> >> [2016-10-24 10:29:54.118329] W [MSGID: 114031]<br>
>>> >> [client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-1:<br>
>>> >> remote operation failed. Path: (null)<br>
>>> >> (00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
>>> >> [2016-10-24 10:29:54.118751] E [MSGID: 133010]<br>
>>> >> [shard.c:1582:shard_common_<wbr>lookup_shards_cbk] 0-testHeal-shard: Lookup<br>
>>> >> on shard 1 failed. Base file gfid =<br>
>>> >> 9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727 [Input/output error]<br>
>>> >> [2016-10-24 10:29:54.118787] W [fuse-bridge.c:2227:fuse_<wbr>readv_cbk]<br>
>>> >> 0-glusterfs-fuse: 137: READ => -1<br>
>>> >> gfid=9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727 fd=0x7f70c80d12dc<br>
>>> >> (Input/output error)<br>
>>> >> [2016-10-24 10:29:54.119330] W [MSGID: 114031]<br>
>>> >> [client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-1:<br>
>>> >> remote operation failed. Path: (null)<br>
>>> >> (00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
>>> >> [2016-10-24 10:29:54.119338] W [MSGID: 114031]<br>
>>> >> [client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-0:<br>
>>> >> remote operation failed. Path: (null)<br>
>>> >> (00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
>>> >> [2016-10-24 10:29:54.119368] W [MSGID: 114031]<br>
>>> >> [client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-2:<br>
>>> >> remote operation failed. Path: (null)<br>
>>> >> (00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
>>> >> [2016-10-24 10:29:54.119674] E [MSGID: 133010]<br>
>>> >> [shard.c:1582:shard_common_<wbr>lookup_shards_cbk] 0-testHeal-shard: Lookup<br>
>>> >> on shard 1 failed. Base file gfid =<br>
>>> >> 9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727 [Input/output error]<br>
>>> >> [2016-10-24 10:29:54.119715] W [fuse-bridge.c:2227:fuse_<wbr>readv_cbk]<br>
>>> >> 0-glusterfs-fuse: 138: READ => -1<br>
>>> >> gfid=9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727 fd=0x7f70c80d12dc<br>
>>> >> (Input/output error)<br>
>>> >> [2016-10-24 10:36:13.140414] W [MSGID: 114031]<br>
>>> >> [client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-0:<br>
>>> >> remote operation failed. Path: (null)<br>
>>> >> (00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
>>> >> [2016-10-24 10:36:13.140451] W [MSGID: 114031]<br>
>>> >> [client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-2:<br>
>>> >> remote operation failed. Path: (null)<br>
>>> >> (00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
>>> >> [2016-10-24 10:36:13.140461] W [MSGID: 114031]<br>
>>> >> [client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-1:<br>
>>> >> remote operation failed. Path: (null)<br>
>>> >> (00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
>>> >> [2016-10-24 10:36:13.140956] E [MSGID: 133010]<br>
>>> >> [shard.c:1582:shard_common_<wbr>lookup_shards_cbk] 0-testHeal-shard: Lookup<br>
>>> >> on shard 1 failed. Base file gfid =<br>
>>> >> 9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727 [Input/output error]<br>
>>> >> [2016-10-24 10:36:13.140995] W [fuse-bridge.c:2227:fuse_<wbr>readv_cbk]<br>
>>> >> 0-glusterfs-fuse: 145: READ => -1<br>
>>> >> gfid=9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727 fd=0x7f70c80d12dc<br>
>>> >> (Input/output error)<br>
>>> >> [2016-10-25 03:22:01.220025] I [MSGID: 100011]<br>
>>> >> [glusterfsd.c:1323:<wbr>reincarnate] 0-glusterfsd: Fetching the volume file<br>
>>> >> from server...<br>
>>> >> [2016-10-25 03:22:01.220938] I<br>
>>> >> [glusterfsd-mgmt.c:1600:mgmt_<wbr>getspec_cbk] 0-glusterfs: No change in<br>
>>> >> volfile, continuing<br>
>>> >><br>
>>> >> I also attached the log in this email.<br>
>>> >><br>
>>> >> Thanks.<br>
>>> >><br>
>>> >> Cwtan<br>
>>> >><br>
>>> >><br>
>>> >> On Wed, Oct 26, 2016 at 12:30 AM, Krutika Dhananjay<br>
>>> >> <<a href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>><br>
>>> >> wrote:<br>
>>> >> > Tried it locally on my setup. Worked fine.<br>
>>> >> ><br>
>>> >> > Could you please attach the mount logs?<br>
>>> >> ><br>
>>> >> > -Krutika<br>
>>> >> ><br>
>>> >> > On Tue, Oct 25, 2016 at 6:55 PM, Pranith Kumar Karampuri<br>
>>> >> > <<a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>> wrote:<br>
>>> >> >><br>
>>> >> >> +Krutika<br>
>>> >> >><br>
>>> >> >> On Mon, Oct 24, 2016 at 4:10 PM, qingwei wei <<a href="mailto:tchengwee@gmail.com">tchengwee@gmail.com</a>><br>
>>> >> >> wrote:<br>
>>> >> >>><br>
>>> >> >>> Hi,<br>
>>> >> >>><br>
>>> >> >>> I am currently running a simple gluster setup using one server<br>
>>> >> >>> node<br>
>>> >> >>> with multiple disks. I realize that if i delete away all the<br>
>>> >> >>> .shard<br>
>>> >> >>> files in one replica in the backend, my application (dd) will<br>
>>> >> >>> report<br>
>>> >> >>> Input/Output error even though i have 3 replicas.<br>
>>> >> >>><br>
>>> >> >>> My gluster version is 3.7.16<br>
>>> >> >>><br>
>>> >> >>> gluster volume file<br>
>>> >> >>><br>
>>> >> >>> Volume Name: testHeal<br>
>>> >> >>> Type: Replicate<br>
>>> >> >>> Volume ID: 26d16d7f-bc4f-44a6-a18b-<wbr>eab780d80851<br>
>>> >> >>> Status: Started<br>
>>> >> >>> Number of Bricks: 1 x 3 = 3<br>
>>> >> >>> Transport-type: tcp<br>
>>> >> >>> Bricks:<br>
>>> >> >>> Brick1: 192.168.123.4:/mnt/sdb_mssd/<wbr>testHeal2<br>
>>> >> >>> Brick2: 192.168.123.4:/mnt/sde_mssd/<wbr>testHeal2<br>
>>> >> >>> Brick3: 192.168.123.4:/mnt/sdd_mssd/<wbr>testHeal2<br>
>>> >> >>> Options Reconfigured:<br>
>>> >> >>> cluster.self-heal-daemon: on<br>
>>> >> >>> features.shard-block-size: 16MB<br>
>>> >> >>> features.shard: on<br>
>>> >> >>> performance.readdir-ahead: on<br>
>>> >> >>><br>
>>> >> >>> dd error<br>
>>> >> >>><br>
>>> >> >>> [root@fujitsu05 .shard]# dd of=/home/test if=/mnt/fuseMount/ddTest<br>
>>> >> >>> bs=16M count=20 oflag=direct<br>
>>> >> >>> dd: error reading ‘/mnt/fuseMount/ddTest’: Input/output error<br>
>>> >> >>> 1+0 records in<br>
>>> >> >>> 1+0 records out<br>
>>> >> >>> 16777216 bytes (17 MB) copied, 0.111038 s, 151 MB/s<br>
>>> >> >>><br>
>>> >> >>> in the .shard folder where i deleted all the .shard file, i can<br>
>>> >> >>> see<br>
>>> >> >>> one .shard file is recreated<br>
>>> >> >>><br>
>>> >> >>> getfattr -d -e hex -m. 9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727.1<br>
>>> >> >>> # file: 9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727.1<br>
>>> >> >>> trusted.afr.testHeal-client-0=<wbr>0x000000010000000100000000<br>
>>> >> >>> trusted.afr.testHeal-client-2=<wbr>0x000000010000000100000000<br>
>>> >> >>> trusted.gfid=<wbr>0x41b653f7daa14627b1f91f9e8554<wbr>ddde<br>
>>> >> >>><br>
>>> >> >>> However, the gfid is not the same compare to the other replicas<br>
>>> >> >>><br>
>>> >> >>> getfattr -d -e hex -m. 9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727.1<br>
>>> >> >>> # file: 9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727.1<br>
>>> >> >>> trusted.afr.dirty=<wbr>0x000000000000000000000000<br>
>>> >> >>> trusted.afr.testHeal-client-1=<wbr>0x000000000000000000000000<br>
>>> >> >>> trusted.bit-rot.version=<wbr>0x0300000000000000580dde99000e<wbr>5e5d<br>
>>> >> >>> trusted.gfid=<wbr>0x9ee5c5eed7964a6cb9ac1a1419de<wbr>5a40<br>
>>> >> >>><br>
>>> >> >>> Is this consider a bug?<br>
>>> >> >>><br>
>>> >> >>> Regards,<br>
>>> >> >>><br>
>>> >> >>> Cwtan<br>
>>> >> >>> ______________________________<wbr>_________________<br>
>>> >> >>> Gluster-devel mailing list<br>
>>> >> >>> <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
>>> >> >>> <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br>
>>> >> >><br>
>>> >> >><br>
>>> >> >><br>
>>> >> >><br>
>>> >> >> --<br>
>>> >> >> Pranith<br>
>>> >> ><br>
>>> >> ><br>
>>> ><br>
>>> ><br>
>><br>
>><br>
><br>
</div></div></blockquote></div><br></div>