<div dir="ltr"><div>Do you also have the brick logs? Looks like the bricks are returning EINVAL on lookup<br>which AFR is subsequently converting into an EIO. And sharding is merely delivering the same error code upwards.<br><br></div>-Krutika<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 26, 2016 at 6:38 AM, qingwei wei <span dir="ltr"><<a href="mailto:tchengwee@gmail.com" target="_blank">tchengwee@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
Pls see the client log below.<br>
<br>
[2016-10-24 10:29:51.111603] I [fuse-bridge.c:5171:fuse_<wbr>graph_setup]<br>
0-fuse: switched to graph 0<br>
[2016-10-24 10:29:51.111662] I [MSGID: 114035]<br>
[client-handshake.c:193:<wbr>client_set_lk_version_cbk]<br>
0-testHeal-client-2: Server lk version = 1<br>
[2016-10-24 10:29:51.112371] I [fuse-bridge.c:4083:fuse_init]<br>
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.22<br>
kernel 7.22<br>
[2016-10-24 10:29:51.113563] I [MSGID: 108031]<br>
[afr-common.c:2071:afr_local_<wbr>discovery_cbk] 0-testHeal-replicate-0:<br>
selecting local read_child testHeal-client-2<br>
[2016-10-24 10:29:51.113604] I [MSGID: 108031]<br>
[afr-common.c:2071:afr_local_<wbr>discovery_cbk] 0-testHeal-replicate-0:<br>
selecting local read_child testHeal-client-0<br>
[2016-10-24 10:29:51.113630] I [MSGID: 108031]<br>
[afr-common.c:2071:afr_local_<wbr>discovery_cbk] 0-testHeal-replicate-0:<br>
selecting local read_child testHeal-client-1<br>
[2016-10-24 10:29:54.016802] W [MSGID: 108001]<br>
[afr-transaction.c:789:afr_<wbr>handle_quorum] 0-testHeal-replicate-0:<br>
/.shard/9061198a-eb7e-45a2-<wbr>93fb-eb396d1b2727.1: F<br>
ailing MKNOD as quorum is not met<br>
[2016-10-24 10:29:54.019330] W [MSGID: 114031]<br>
[client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-0:<br>
remote operation failed. Path: (null) (00000000-<br>
0000-0000-0000-000000000000) [Invalid argument]<br>
[2016-10-24 10:29:54.019343] W [MSGID: 114031]<br>
[client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-2:<br>
remote operation failed. Path: (null) (00000000-<br>
0000-0000-0000-000000000000) [Invalid argument]<br>
[2016-10-24 10:29:54.019373] W [MSGID: 114031]<br>
[client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-1:<br>
remote operation failed. Path: (null) (00000000-<br>
0000-0000-0000-000000000000) [Invalid argument]<br>
[2016-10-24 10:29:54.019854] E [MSGID: 133010]<br>
[shard.c:1582:shard_common_<wbr>lookup_shards_cbk] 0-testHeal-shard: Lookup<br>
on shard 1 failed. Base file gfid = 9061198a<br>
-eb7e-45a2-93fb-eb396d1b2727 [Input/output error]<br>
[2016-10-24 10:29:54.020886] W [fuse-bridge.c:2227:fuse_<wbr>readv_cbk]<br>
0-glusterfs-fuse: 135: READ => -1<br>
gfid=9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727 fd=0x7f70c80d12dc (<br>
Input/output error)<br>
[2016-10-24 10:29:54.118264] W [MSGID: 114031]<br>
[client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-0:<br>
remote operation failed. Path: (null) (00000000-<br>
0000-0000-0000-000000000000) [Invalid argument]<br>
[2016-10-24 10:29:54.118308] W [MSGID: 114031]<br>
[client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-2:<br>
remote operation failed. Path: (null)<br>
(00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
[2016-10-24 10:29:54.118329] W [MSGID: 114031]<br>
[client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-1:<br>
remote operation failed. Path: (null)<br>
(00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
[2016-10-24 10:29:54.118751] E [MSGID: 133010]<br>
[shard.c:1582:shard_common_<wbr>lookup_shards_cbk] 0-testHeal-shard: Lookup<br>
on shard 1 failed. Base file gfid =<br>
9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727 [Input/output error]<br>
[2016-10-24 10:29:54.118787] W [fuse-bridge.c:2227:fuse_<wbr>readv_cbk]<br>
0-glusterfs-fuse: 137: READ => -1<br>
gfid=9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727 fd=0x7f70c80d12dc<br>
(Input/output error)<br>
[2016-10-24 10:29:54.119330] W [MSGID: 114031]<br>
[client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-1:<br>
remote operation failed. Path: (null)<br>
(00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
[2016-10-24 10:29:54.119338] W [MSGID: 114031]<br>
[client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-0:<br>
remote operation failed. Path: (null)<br>
(00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
[2016-10-24 10:29:54.119368] W [MSGID: 114031]<br>
[client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-2:<br>
remote operation failed. Path: (null)<br>
(00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
[2016-10-24 10:29:54.119674] E [MSGID: 133010]<br>
[shard.c:1582:shard_common_<wbr>lookup_shards_cbk] 0-testHeal-shard: Lookup<br>
on shard 1 failed. Base file gfid =<br>
9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727 [Input/output error]<br>
[2016-10-24 10:29:54.119715] W [fuse-bridge.c:2227:fuse_<wbr>readv_cbk]<br>
0-glusterfs-fuse: 138: READ => -1<br>
gfid=9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727 fd=0x7f70c80d12dc<br>
(Input/output error)<br>
[2016-10-24 10:36:13.140414] W [MSGID: 114031]<br>
[client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-0:<br>
remote operation failed. Path: (null)<br>
(00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
[2016-10-24 10:36:13.140451] W [MSGID: 114031]<br>
[client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-2:<br>
remote operation failed. Path: (null)<br>
(00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
[2016-10-24 10:36:13.140461] W [MSGID: 114031]<br>
[client-rpc-fops.c:2981:<wbr>client3_3_lookup_cbk] 0-testHeal-client-1:<br>
remote operation failed. Path: (null)<br>
(00000000-0000-0000-0000-<wbr>000000000000) [Invalid argument]<br>
[2016-10-24 10:36:13.140956] E [MSGID: 133010]<br>
[shard.c:1582:shard_common_<wbr>lookup_shards_cbk] 0-testHeal-shard: Lookup<br>
on shard 1 failed. Base file gfid =<br>
9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727 [Input/output error]<br>
[2016-10-24 10:36:13.140995] W [fuse-bridge.c:2227:fuse_<wbr>readv_cbk]<br>
0-glusterfs-fuse: 145: READ => -1<br>
gfid=9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727 fd=0x7f70c80d12dc<br>
(Input/output error)<br>
[2016-10-25 03:22:01.220025] I [MSGID: 100011]<br>
[glusterfsd.c:1323:<wbr>reincarnate] 0-glusterfsd: Fetching the volume file<br>
from server...<br>
[2016-10-25 03:22:01.220938] I<br>
[glusterfsd-mgmt.c:1600:mgmt_<wbr>getspec_cbk] 0-glusterfs: No change in<br>
volfile, continuing<br>
<br>
I also attached the log in this email.<br>
<br>
Thanks.<br>
<br>
Cwtan<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
On Wed, Oct 26, 2016 at 12:30 AM, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>> wrote:<br>
> Tried it locally on my setup. Worked fine.<br>
><br>
> Could you please attach the mount logs?<br>
><br>
> -Krutika<br>
><br>
> On Tue, Oct 25, 2016 at 6:55 PM, Pranith Kumar Karampuri<br>
> <<a href="mailto:pkarampu@redhat.com">pkarampu@redhat.com</a>> wrote:<br>
>><br>
>> +Krutika<br>
>><br>
>> On Mon, Oct 24, 2016 at 4:10 PM, qingwei wei <<a href="mailto:tchengwee@gmail.com">tchengwee@gmail.com</a>> wrote:<br>
>>><br>
>>> Hi,<br>
>>><br>
>>> I am currently running a simple gluster setup using one server node<br>
>>> with multiple disks. I realize that if i delete away all the .shard<br>
>>> files in one replica in the backend, my application (dd) will report<br>
>>> Input/Output error even though i have 3 replicas.<br>
>>><br>
>>> My gluster version is 3.7.16<br>
>>><br>
>>> gluster volume file<br>
>>><br>
>>> Volume Name: testHeal<br>
>>> Type: Replicate<br>
>>> Volume ID: 26d16d7f-bc4f-44a6-a18b-<wbr>eab780d80851<br>
>>> Status: Started<br>
>>> Number of Bricks: 1 x 3 = 3<br>
>>> Transport-type: tcp<br>
>>> Bricks:<br>
>>> Brick1: 192.168.123.4:/mnt/sdb_mssd/<wbr>testHeal2<br>
>>> Brick2: 192.168.123.4:/mnt/sde_mssd/<wbr>testHeal2<br>
>>> Brick3: 192.168.123.4:/mnt/sdd_mssd/<wbr>testHeal2<br>
>>> Options Reconfigured:<br>
>>> cluster.self-heal-daemon: on<br>
>>> features.shard-block-size: 16MB<br>
>>> features.shard: on<br>
>>> performance.readdir-ahead: on<br>
>>><br>
>>> dd error<br>
>>><br>
>>> [root@fujitsu05 .shard]# dd of=/home/test if=/mnt/fuseMount/ddTest<br>
>>> bs=16M count=20 oflag=direct<br>
>>> dd: error reading ‘/mnt/fuseMount/ddTest’: Input/output error<br>
>>> 1+0 records in<br>
>>> 1+0 records out<br>
>>> 16777216 bytes (17 MB) copied, 0.111038 s, 151 MB/s<br>
>>><br>
>>> in the .shard folder where i deleted all the .shard file, i can see<br>
>>> one .shard file is recreated<br>
>>><br>
>>> getfattr -d -e hex -m. 9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727.1<br>
>>> # file: 9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727.1<br>
>>> trusted.afr.testHeal-client-0=<wbr>0x000000010000000100000000<br>
>>> trusted.afr.testHeal-client-2=<wbr>0x000000010000000100000000<br>
>>> trusted.gfid=<wbr>0x41b653f7daa14627b1f91f9e8554<wbr>ddde<br>
>>><br>
>>> However, the gfid is not the same compare to the other replicas<br>
>>><br>
>>> getfattr -d -e hex -m. 9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727.1<br>
>>> # file: 9061198a-eb7e-45a2-93fb-<wbr>eb396d1b2727.1<br>
>>> trusted.afr.dirty=<wbr>0x000000000000000000000000<br>
>>> trusted.afr.testHeal-client-1=<wbr>0x000000000000000000000000<br>
>>> trusted.bit-rot.version=<wbr>0x0300000000000000580dde99000e<wbr>5e5d<br>
>>> trusted.gfid=<wbr>0x9ee5c5eed7964a6cb9ac1a1419de<wbr>5a40<br>
>>><br>
>>> Is this consider a bug?<br>
>>><br>
>>> Regards,<br>
>>><br>
>>> Cwtan<br>
>>> ______________________________<wbr>_________________<br>
>>> Gluster-devel mailing list<br>
>>> <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
>>> <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br>
>><br>
>><br>
>><br>
>><br>
>> --<br>
>> Pranith<br>
><br>
><br>
</div></div></blockquote></div><br></div>