<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="MS Exchange Server version 14.02.5004.000">
<title></title>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"></span></p>
<table class="MsoNormalTable" border="1" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%;border-top:dotted windowtext 1.0pt;border-left:none;border-bottom:dotted windowtext 1.0pt;border-right:none;margin-left:-2.0pt;margin-right:-2.0pt;margin-bottom:16px;margin-top:1px">
<tbody>
<tr>
<td width="100%" colspan="2" valign="top" style="width:100.0%;border:none;padding:0in 5.4pt 0in 5.4pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Attachment (1):<o:p></o:p></span></p>
</td>
</tr>
<tr>
<td width="1%" valign="top" style="width:1.26%;border:none;padding:0in 5.4pt 0in 5.4pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">1<o:p></o:p></span></p>
</td>
<td width="98%" valign="top" style="width:98.74%;border:none;padding:0in 5.4pt 0in 5.4pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif""><a href="https://imap.commvault.com/webconsole/embedded.do?url=https://imap.commvault.com/webconsole/api/drive/publicshare/346714/file/1272e68278744f15bf1a54f2b31b559d/action/preview&downloadUrl=https://imap.commvault.com/webconsole/api/contentstore/publicshare/346714/file/1272e68278744f15bf1a54f2b31b559d/action/download">ecxattrs.txt</a>
<a href="https://imap.commvault.com/webconsole/api/contentstore/publicshare/346714/file/1272e68278744f15bf1a54f2b31b559d/action/download">
[Download]</a><span style="color:#1F497D"> </span>(5.92 KB)<o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
</div>
<!-- Converted from text/plain format -->
<p><font size="2">Xavi,<br>
Please find attached the extended attributes for a directory from all the bricks. Free space check failed for this with error number EIO.<br>
<br>
Thanks and Regards,<br>
Ram<br>
<br>
-----Original Message-----<br>
From: Xavier Hernandez [<a href="mailto:xhernandez@datalab.es">mailto:xhernandez@datalab.es</a>]<br>
Sent: Tuesday, January 10, 2017 6:45 AM<br>
To: Ankireddypalle Reddy; Gluster Devel (gluster-devel@gluster.org); gluster-users@gluster.org<br>
Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume<br>
<br>
Hi Ram,<br>
<br>
can you execute the following command on all bricks on a file that is giving EIO ?<br>
<br>
getfattr -m. -e hex -d <path to file in brick><br>
<br>
Xavi<br>
<br>
On 10/01/17 12:41, Ankireddypalle Reddy wrote:<br>
> Xavi,<br>
> We have been running 3.7.8 on these servers. We upgraded to 3.7.18 yesterday. We upgraded all the servers at a time. The volume was brought down during upgrade.<br>
><br>
> Thanks and Regards,<br>
> Ram<br>
><br>
> -----Original Message-----<br>
> From: Xavier Hernandez [<a href="mailto:xhernandez@datalab.es">mailto:xhernandez@datalab.es</a>]<br>
> Sent: Tuesday, January 10, 2017 6:35 AM<br>
> To: Ankireddypalle Reddy; Gluster Devel (gluster-devel@gluster.org);<br>
> gluster-users@gluster.org<br>
> Subject: Re: [Gluster-devel] Lot of EIO errors in disperse volume<br>
><br>
> Hi Ram,<br>
><br>
> how did you upgrade gluster ? from which version ?<br>
><br>
> Did you upgrade one server at a time and waited until self-heal finished before upgrading the next server ?<br>
><br>
> Xavi<br>
><br>
> On 10/01/17 11:39, Ankireddypalle Reddy wrote:<br>
>> Hi,<br>
>><br>
>> We upgraded to GlusterFS 3.7.18 yesterday. We see lot of<br>
>> failures in our applications. Most of the errors are EIO. The<br>
>> following log lines are commonly seen in the logs:<br>
>><br>
>><br>
>><br>
>> The message "W [MSGID: 122056] [ec-combine.c:873:ec_combine_check]<br>
>> 0-StoragePool-disperse-4: Mismatching xdata in answers of 'LOOKUP'"<br>
>> repeated 2 times between [2017-01-10 02:46:25.069809] and [2017-01-10<br>
>> 02:46:25.069835]<br>
>><br>
>> [2017-01-10 02:46:25.069852] W [MSGID: 122056]<br>
>> [ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-5:<br>
>> Mismatching xdata in answers of 'LOOKUP'<br>
>><br>
>> The message "W [MSGID: 122056] [ec-combine.c:873:ec_combine_check]<br>
>> 0-StoragePool-disperse-5: Mismatching xdata in answers of 'LOOKUP'"<br>
>> repeated 2 times between [2017-01-10 02:46:25.069852] and [2017-01-10<br>
>> 02:46:25.069873]<br>
>><br>
>> [2017-01-10 02:46:25.069910] W [MSGID: 122056]<br>
>> [ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-6:<br>
>> Mismatching xdata in answers of 'LOOKUP'<br>
>><br>
>> ...<br>
>><br>
>> [2017-01-10 02:46:26.520774] I [MSGID: 109036]<br>
>> [dht-common.c:9076:dht_log_new_layout_for_dir_selfheal]<br>
>> 0-StoragePool-dht: Setting layout of<br>
>> /Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854213/CHUNK_51334585 with<br>
>> [Subvol_name: StoragePool-disperse-0, Err: -1 , Start: 3221225466 ,<br>
>> Stop: 3758096376 , Hash: 1 ], [Subvol_name: StoragePool-disperse-1, Err:<br>
>> -1 , Start: 3758096377 , Stop: 4294967295 , Hash: 1 ], [Subvol_name:<br>
>> StoragePool-disperse-2, Err: -1 , Start: 0 , Stop: 536870910 , Hash:<br>
>> 1 ], [Subvol_name: StoragePool-disperse-3, Err: -1 , Start: 536870911<br>
>> ,<br>
>> Stop: 1073741821 , Hash: 1 ], [Subvol_name: StoragePool-disperse-4, Err:<br>
>> -1 , Start: 1073741822 , Stop: 1610612732 , Hash: 1 ], [Subvol_name:<br>
>> StoragePool-disperse-5, Err: -1 , Start: 1610612733 , Stop:<br>
>> 2147483643 ,<br>
>> Hash: 1 ], [Subvol_name: StoragePool-disperse-6, Err: -1 , Start:<br>
>> 2147483644 , Stop: 2684354554 , Hash: 1 ], [Subvol_name:<br>
>> StoragePool-disperse-7, Err: -1 , Start: 2684354555 , Stop:<br>
>> 3221225465 ,<br>
>> Hash: 1 ],<br>
>><br>
>> [2017-01-10 02:46:26.522841] N [MSGID: 122031]<br>
>> [ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-3:<br>
>> Mismatching dictionary in answers of 'GF_FOP_XATTROP'<br>
>><br>
>> The message "N [MSGID: 122031] [ec-generic.c:1130:ec_combine_xattrop]<br>
>> 0-StoragePool-disperse-3: Mismatching dictionary in answers of<br>
>> 'GF_FOP_XATTROP'" repeated 2 times between [2017-01-10<br>
>> 02:46:26.522841] and [2017-01-10 02:46:26.522894]<br>
>><br>
>> [2017-01-10 02:46:26.522898] W [MSGID: 122040]<br>
>> [ec-common.c:919:ec_prepare_update_cbk] 0-StoragePool-disperse-3:<br>
>> Failed to get size and version [Input/output error]<br>
>><br>
>> [2017-01-10 02:46:26.523115] N [MSGID: 122031]<br>
>> [ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-6:<br>
>> Mismatching dictionary in answers of 'GF_FOP_XATTROP'<br>
>><br>
>> The message "N [MSGID: 122031] [ec-generic.c:1130:ec_combine_xattrop]<br>
>> 0-StoragePool-disperse-6: Mismatching dictionary in answers of<br>
>> 'GF_FOP_XATTROP'" repeated 2 times between [2017-01-10<br>
>> 02:46:26.523115] and [2017-01-10 02:46:26.523143]<br>
>><br>
>> [2017-01-10 02:46:26.523147] W [MSGID: 122040]<br>
>> [ec-common.c:919:ec_prepare_update_cbk] 0-StoragePool-disperse-6:<br>
>> Failed to get size and version [Input/output error]<br>
>><br>
>> [2017-01-10 02:46:26.523302] N [MSGID: 122031]<br>
>> [ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-2:<br>
>> Mismatching dictionary in answers of 'GF_FOP_XATTROP'<br>
>><br>
>> The message "N [MSGID: 122031] [ec-generic.c:1130:ec_combine_xattrop]<br>
>> 0-StoragePool-disperse-2: Mismatching dictionary in answers of<br>
>> 'GF_FOP_XATTROP'" repeated 2 times between [2017-01-10<br>
>> 02:46:26.523302] and [2017-01-10 02:46:26.523324]<br>
>><br>
>> [2017-01-10 02:46:26.523328] W [MSGID: 122040]<br>
>> [ec-common.c:919:ec_prepare_update_cbk] 0-StoragePool-disperse-2:<br>
>> Failed to get size and version [Input/output error]<br>
>><br>
>><br>
>><br>
>> [root@glusterfs3 Log_Files]# gluster --version<br>
>><br>
>> glusterfs 3.7.18 built on Dec 8 2016 06:34:26<br>
>><br>
>><br>
>><br>
>> [root@glusterfs3 Log_Files]# gluster volume info<br>
>><br>
>><br>
>><br>
>> Volume Name: StoragePool<br>
>><br>
>> Type: Distributed-Disperse<br>
>><br>
>> Volume ID: 149e976f-4e21-451c-bf0f-f5691208531f<br>
>><br>
>> Status: Started<br>
>><br>
>> Number of Bricks: 8 x (2 + 1) = 24<br>
>><br>
>> Transport-type: tcp<br>
>><br>
>> Bricks:<br>
>><br>
>> Brick1: glusterfs1sds:/ws/disk1/ws_brick<br>
>><br>
>> Brick2: glusterfs2sds:/ws/disk1/ws_brick<br>
>><br>
>> Brick3: glusterfs3sds:/ws/disk1/ws_brick<br>
>><br>
>> Brick4: glusterfs1sds:/ws/disk2/ws_brick<br>
>><br>
>> Brick5: glusterfs2sds:/ws/disk2/ws_brick<br>
>><br>
>> Brick6: glusterfs3sds:/ws/disk2/ws_brick<br>
>><br>
>> Brick7: glusterfs1sds:/ws/disk3/ws_brick<br>
>><br>
>> Brick8: glusterfs2sds:/ws/disk3/ws_brick<br>
>><br>
>> Brick9: glusterfs3sds:/ws/disk3/ws_brick<br>
>><br>
>> Brick10: glusterfs1sds:/ws/disk4/ws_brick<br>
>><br>
>> Brick11: glusterfs2sds:/ws/disk4/ws_brick<br>
>><br>
>> Brick12: glusterfs3sds:/ws/disk4/ws_brick<br>
>><br>
>> Brick13: glusterfs1sds:/ws/disk5/ws_brick<br>
>><br>
>> Brick14: glusterfs2sds:/ws/disk5/ws_brick<br>
>><br>
>> Brick15: glusterfs3sds:/ws/disk5/ws_brick<br>
>><br>
>> Brick16: glusterfs1sds:/ws/disk6/ws_brick<br>
>><br>
>> Brick17: glusterfs2sds:/ws/disk6/ws_brick<br>
>><br>
>> Brick18: glusterfs3sds:/ws/disk6/ws_brick<br>
>><br>
>> Brick19: glusterfs1sds:/ws/disk7/ws_brick<br>
>><br>
>> Brick20: glusterfs2sds:/ws/disk7/ws_brick<br>
>><br>
>> Brick21: glusterfs3sds:/ws/disk7/ws_brick<br>
>><br>
>> Brick22: glusterfs1sds:/ws/disk8/ws_brick<br>
>><br>
>> Brick23: glusterfs2sds:/ws/disk8/ws_brick<br>
>><br>
>> Brick24: glusterfs3sds:/ws/disk8/ws_brick<br>
>><br>
>> Options Reconfigured:<br>
>><br>
>> performance.readdir-ahead: on<br>
>><br>
>> diagnostics.client-log-level: INFO<br>
>><br>
>><br>
>><br>
>> Thanks and Regards,<br>
>><br>
>> Ram<br>
>><br>
>> ***************************Legal<br>
>> Disclaimer***************************<br>
>> "This communication may contain confidential and privileged material<br>
>> for the sole use of the intended recipient. Any unauthorized review,<br>
>> use or distribution by others is strictly prohibited. If you have<br>
>> received the message by mistake, please advise the sender by reply<br>
>> email and delete the message. Thank you."<br>
>> *********************************************************************<br>
>> *<br>
>><br>
>><br>
>> _______________________________________________<br>
>> Gluster-devel mailing list<br>
>> Gluster-devel@gluster.org<br>
>> <a href="http://www.gluster.org/mailman/listinfo/gluster-devel">http://www.gluster.org/mailman/listinfo/gluster-devel</a><br>
>><br>
><br>
> ***************************Legal Disclaimer***************************<br>
> "This communication may contain confidential and privileged material<br>
> for the sole use of the intended recipient. Any unauthorized review,<br>
> use or distribution by others is strictly prohibited. If you have<br>
> received the message by mistake, please advise the sender by reply email and delete the message. Thank you."<br>
> **********************************************************************<br>
><br>
<br>
</font></p>
<div>***************************Legal Disclaimer***************************<div>"This communication may contain confidential and privileged material for the<div>sole use of the intended recipient. Any unauthorized review, use or distribution<div>by others is strictly prohibited. If you have received the message by mistake,<div>please advise the sender by reply email and delete the message. Thank you."<div>**********************************************************************</body>
</html>