<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"> We upgraded to GlusterFS 3.7.18 yesterday. We see lot of failures in our applications. Most of the errors are EIO. The following log lines are commonly seen in the logs:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The message "W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-4: Mismatching xdata in answers of 'LOOKUP'" repeated 2 times between [2017-01-10 02:46:25.069809] and [2017-01-10 02:46:25.069835]<o:p></o:p></p>
<p class="MsoNormal">[2017-01-10 02:46:25.069852] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-5: Mismatching xdata in answers of 'LOOKUP'<o:p></o:p></p>
<p class="MsoNormal">The message "W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-5: Mismatching xdata in answers of 'LOOKUP'" repeated 2 times between [2017-01-10 02:46:25.069852] and [2017-01-10 02:46:25.069873]<o:p></o:p></p>
<p class="MsoNormal">[2017-01-10 02:46:25.069910] W [MSGID: 122056] [ec-combine.c:873:ec_combine_check] 0-StoragePool-disperse-6: Mismatching xdata in answers of 'LOOKUP'<o:p></o:p></p>
<p class="MsoNormal">…<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">[2017-01-10 02:46:26.520774] I [MSGID: 109036] [dht-common.c:9076:dht_log_new_layout_for_dir_selfheal] 0-StoragePool-dht: Setting layout
</span><span style="color:red">of /Folder_07.11.2016_23.02/CV_MAGNETIC/V_8854213/CHUNK_51334585
</span><span style="color:#1F497D">with [Subvol_name: StoragePool-disperse-0, Err: -1 , Start: 3221225466 , Stop: 3758096376 , Hash: 1 ], [Subvol_name: StoragePool-disperse-1, Err: -1 , Start: 3758096377 , Stop: 4294967295 , Hash: 1 ], [Subvol_name: StoragePool-disperse-2,
Err: -1 , Start: 0 , Stop: 536870910 , Hash: 1 ], [Subvol_name: StoragePool-disperse-3, Err: -1 , Start: 536870911 , Stop: 1073741821 , Hash: 1 ], [Subvol_name: StoragePool-disperse-4, Err: -1 , Start: 1073741822 , Stop: 1610612732 , Hash: 1 ], [Subvol_name:
StoragePool-disperse-5, Err: -1 , Start: 1610612733 , Stop: 2147483643 , Hash: 1 ], [Subvol_name: StoragePool-disperse-6, Err: -1 , Start: 2147483644 , Stop: 2684354554 , Hash: 1 ], [Subvol_name: StoragePool-disperse-7, Err: -1 , Start: 2684354555 , Stop:
3221225465 , Hash: 1 ],<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2017-01-10 02:46:26.522841] N [MSGID: 122031] [ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-3: Mismatching dictionary in answers of 'GF_FOP_XATTROP'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">The message "N [MSGID: 122031] [ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-3: Mismatching dictionary in answers of 'GF_FOP_XATTROP'" repeated 2 times between [2017-01-10 02:46:26.522841] and
[2017-01-10 02:46:26.522894]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2017-01-10 02:46:26.522898] W [MSGID: 122040] [ec-common.c:919:ec_prepare_update_cbk] 0-StoragePool-disperse-3: Failed to get size and version [</span><span style="color:red">Input/output error</span><span style="color:#1F497D">]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2017-01-10 02:46:26.523115] N [MSGID: 122031] [ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-6: Mismatching dictionary in answers of 'GF_FOP_XATTROP'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">The message "N [MSGID: 122031] [ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-6: Mismatching dictionary in answers of 'GF_FOP_XATTROP'" repeated 2 times between [2017-01-10 02:46:26.523115] and
[2017-01-10 02:46:26.523143]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2017-01-10 02:46:26.523147] W [MSGID: 122040] [ec-common.c:919:ec_prepare_update_cbk] 0-StoragePool-disperse-6: Failed to get size and version [</span><span style="color:red">Input/output error</span><span style="color:#1F497D">]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2017-01-10 02:46:26.523302] N [MSGID: 122031] [ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-2: Mismatching dictionary in answers of 'GF_FOP_XATTROP'<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">The message "N [MSGID: 122031] [ec-generic.c:1130:ec_combine_xattrop] 0-StoragePool-disperse-2: Mismatching dictionary in answers of 'GF_FOP_XATTROP'" repeated 2 times between [2017-01-10 02:46:26.523302] and
[2017-01-10 02:46:26.523324]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">[2017-01-10 02:46:26.523328] W [MSGID: 122040] [ec-common.c:919:ec_prepare_update_cbk] 0-StoragePool-disperse-2: Failed to get size and version [</span><span style="color:red">Input/output error</span><span style="color:#1F497D">]<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@glusterfs3 Log_Files]# gluster --version<o:p></o:p></p>
<p class="MsoNormal">glusterfs <span style="color:red">3.7.18 </span>built on Dec 8 2016 06:34:26<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@glusterfs3 Log_Files]# gluster volume info<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Volume Name: StoragePool<o:p></o:p></p>
<p class="MsoNormal">Type: <span style="color:red">Distributed-Disperse</span><o:p></o:p></p>
<p class="MsoNormal">Volume ID: 149e976f-4e21-451c-bf0f-f5691208531f<o:p></o:p></p>
<p class="MsoNormal">Status: Started<o:p></o:p></p>
<p class="MsoNormal">Number of Bricks: 8 x (2 + 1) = 24<o:p></o:p></p>
<p class="MsoNormal">Transport-type: tcp<o:p></o:p></p>
<p class="MsoNormal">Bricks:<o:p></o:p></p>
<p class="MsoNormal">Brick1: glusterfs1sds:/ws/disk1/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick2: glusterfs2sds:/ws/disk1/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick3: glusterfs3sds:/ws/disk1/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick4: glusterfs1sds:/ws/disk2/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick5: glusterfs2sds:/ws/disk2/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick6: glusterfs3sds:/ws/disk2/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick7: glusterfs1sds:/ws/disk3/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick8: glusterfs2sds:/ws/disk3/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick9: glusterfs3sds:/ws/disk3/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick10: glusterfs1sds:/ws/disk4/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick11: glusterfs2sds:/ws/disk4/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick12: glusterfs3sds:/ws/disk4/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick13: glusterfs1sds:/ws/disk5/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick14: glusterfs2sds:/ws/disk5/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick15: glusterfs3sds:/ws/disk5/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick16: glusterfs1sds:/ws/disk6/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick17: glusterfs2sds:/ws/disk6/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick18: glusterfs3sds:/ws/disk6/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick19: glusterfs1sds:/ws/disk7/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick20: glusterfs2sds:/ws/disk7/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick21: glusterfs3sds:/ws/disk7/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick22: glusterfs1sds:/ws/disk8/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick23: glusterfs2sds:/ws/disk8/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Brick24: glusterfs3sds:/ws/disk8/ws_brick<o:p></o:p></p>
<p class="MsoNormal">Options Reconfigured:<o:p></o:p></p>
<p class="MsoNormal">performance.readdir-ahead: on<o:p></o:p></p>
<p class="MsoNormal">diagnostics.client-log-level: INFO<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks and Regards,<o:p></o:p></p>
<p class="MsoNormal">Ram<o:p></o:p></p>
</div>
<div>***************************Legal Disclaimer***************************<div>"This communication may contain confidential and privileged material for the<div>sole use of the intended recipient. Any unauthorized review, use or distribution<div>by others is strictly prohibited. If you have received the message by mistake,<div>please advise the sender by reply email and delete the message. Thank you."<div>**********************************************************************</body>
</html>