<div dir="ltr"><div>Yes, same message on gluster03's brick log:</div><div>[2016-06-16 10:07:55.619621] E [MSGID: 115059] [server-rpc-fops.c:811:server_getxattr_cbk] 0-storage-server: 23783173: GETXATTR /data/climate/ANUSPLIN/ANUSPLIN300/monthly/pcp_grids/1918/pcp300_08.asc (0e98a94b-7b86-4a72-88a9-a99a787e059d) ((null)) ==> (Numerical result out of range) [Numerical result out of range]</div><div><br></div><div>nothing indicated in etc-glusterfs-glusterd.vol.log</div><div><br></div><div>Also it seems hard to believe TSM would be sending back a larger value than was sent to it from the initial backup done on gluster storage. ie: File on gluster fuse mount -> TSM backup ... TSM restore -> File restore to gluster fuse mount.<br></div><div><br></div><div>I can't actually get the xattrs from the file, because the file doesn't exist after TSM errors out. My guess is that TSM restores the file, then tries to verify the xattrs and on failure removes the file. BUT I suppose if there was some corruption on TSM side, it might be trying to send garbage too large to store in the xattr (if I'm understanding the issue).</div><div><br></div><div>If I restore the single file above after the failure I don't get any errors, which is why I started to suspect gluster as the culprit.<br></div><div><br></div><div>I'm capturing the strace output now, hopefully something useful is shown.</div><div><br></div><div>Thanks</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jun 16, 2016 at 7:31 PM, Vijay Bellur <span dir="ltr"><<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Thu, Jun 16, 2016 at 3:05 PM, Steve Dainard <<a href="mailto:sdainard@spd1.com">sdainard@spd1.com</a>> wrote:<br>
> I'm restoring some data to gluster from TSM backups and the client errors<br>
> out trying to retrieve xattrs at some point during the restore, killing<br>
> progress:<br>
> ...<br>
> Restoring 8,118,878<br>
> /storage/data/climate/ANUSPLIN/ANUSPLIN300/monthly/pcp_grids/1918/pcp300_04.asc<br>
> [Done]<br>
> ANS1587W Unable to read extended attributes for object<br>
> /storage/data/climate/ANUSPLIN/ANUSPLIN300/monthly/pcp_grids/1918/pcp300_08.asc<br>
> due to errno: 34, reason: Numerical result out of range<br>
> ** Unsuccessful **<br>
> ...<br>
><br>
> In the gluster fuse logs for the volume I see this:<br>
> [2016-06-16 10:07:55.622020] W [MSGID: 114031]<br>
> [client-rpc-fops.c:1161:client3_3_getxattr_cbk] 0-storage-client-2: remote<br>
> operation failed. Path:<br>
> /data/climate/ANUSPLIN/ANUSPLIN300/monthly/pcp_grids/1918/pcp300_08.asc<br>
> (0e98a94b-7b86-4a72-88a9-a99a787e059d). Key: (null) [Numerical result out of<br>
> range]<br>
> [2016-06-16 10:07:55.622110] W [fuse-bridge.c:3353:fuse_xattr_cbk]<br>
> 0-glusterfs-fuse: 76197165: GETXATTR((null))<br>
> /data/climate/ANUSPLIN/ANUSPLIN300/monthly/pcp_grids/1918/pcp300_08.asc =><br>
> -1 (Numerical result out of range)<br>
><br>
> I'm trying to understand if gluster is bubbling up errors to the TSM client<br>
> (gluster fault), or reporting errors the TSM client is generating (TSM<br>
> fault).<br>
><br>
<br>
</div></div>Do you happen to see the same error reported by posix translator(s) in<br>
any of the brick(s)? Doing that might help in figuring out where the<br>
problem could be stemming from.<br>
<br>
As per man (2) getxattr, ERANGE is seen when the size of the value<br>
buffer is too small to hold the result. Would it be possible to strace<br>
the TSM client and see the size of the value buffer being passed?<br>
Also, doing an extended attribute dump of the file on the brick<br>
directory (either through attr or getfattr) can help in determining<br>
the size necessary to hold all attributes.<br>
<br>
HTH,<br>
Vijay<br>
</blockquote></div><br></div>