<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<br>
<div class="moz-cite-prefix">On 03/31/2015 09:05 AM, Félix de
Lelelis wrote:<br>
</div>
<blockquote
cite="mid:CAL4JRL=n8LKjN4Q30PtAkGw7-iAQPPcCk0Ck0PBembkbGPZLjA@mail.gmail.com"
type="cite">
<div dir="ltr">Hi,
<div><br>
</div>
<div>I had a problem with a filesystem xfs on gluster. The
filesystem metadata was filled:</div>
<div><br>
</div>
<div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: space
map metadata: unable to allocate new metadata block</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
252:2: metadata operation 'dm_thin_insert_block' failed:
error = -28</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
252:2: aborting current metadata transaction</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
252:2: switching pool to read-only mode</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): metadata
I/O error: block 0x701830 ("xfs_buf_iodone_callbacks") error
5 numblks 8</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access
beyond end of device</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0,
want=562056, limit=24576</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
process_bio_read_only: dm_thin_find_block() failed: error =
-5</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access
beyond end of device</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0,
want=562056, limit=24576</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
process_bio_read_only: dm_thin_find_block() failed: error =
-5</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): metadata
I/O error: block 0x68047c ("xlog_iodone") error 5 numblks 64</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4):
xfs_do_force_shutdown(0x2) called from line 1170 of file
fs/xfs/xfs_log.c. Return address = 0xffffffffa012a4c1</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): Log I/O
Error Detected. Shutting down filesystem</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): Please
umount the filesystem and rectify the problem(s)</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access
beyond end of device</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0,
want=562056, limit=24576</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
process_bio_read_only: dm_thin_find_block() failed: error =
-5</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): metadata
I/O error: block 0x6804bc ("xlog_iodone") error 5 numblks 64</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4):
xfs_do_force_shutdown(0x2) called from line 1170 of file
fs/xfs/xfs_log.c. Return address = 0xffffffffa012a4c1</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access
beyond end of device</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4):
xfs_log_force: error 5 returned.</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0,
want=562056, limit=24576</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
process_bio_read_only: dm_thin_find_block() failed: error =
-5</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access
beyond end of device</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0,
want=562056, limit=24576</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
process_bio_read_only: dm_thin_find_block() failed: error =
-5</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: attempt to access
beyond end of device</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: dm-0: rw=0,
want=562056, limit=24576</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: device-mapper: thin:
process_bio_read_only: dm_thin_find_block() failed: error =
-5</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4): metadata
I/O error: block 0x6804fc ("xlog_iodone") error 5 numblks 64</div>
<div>Mar 27 13:58:18 srv-vln-des2 kernel: XFS (dm-4):
xfs_do_force_shutdown(0x2) called from line 1170 of file
fs/xfs/xfs_log.c. Return address = 0xffffffffa012a4c</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>After that, gluster was shutdown and with it the 2 server
are shtudown too. The lvm partition was missing and so far I
haven't been able restore the file system. All data is
missing??</div>
<div>I don't understand the situation and I don't know if it's
due a xfs filesystem or glusterfs fail. Someone it has been
this situation?</div>
<div><br>
</div>
<div>Thanks.</div>
<div><br>
</div>
<br>
</div>
</blockquote>
Hi,<br>
<br>
Do you run thin provisioned on LVM? <br>
<br>
There are some discussions about this:<br>
<br>
<blockquote>
<blockquote type="cite">
<pre><a class="moz-txt-link-freetext" href="https://www.redhat.com/archives/linux-lvm/2014-December/msg00015.html">https://www.redhat.com/archives/linux-lvm/2014-December/msg00015.html</a></pre>
</blockquote>
<br>
<blockquote type="cite">
<pre>You definitely ran out of metadata space. Which version of the kernel
and lvm2 userspace are you using?
See the "Metadata space exhaustion" section of the lvmthin manpage in a
recent lvm2 release to guide you on how to recover.
Also, once you've gotten past ths you really should configure lvm2 to
autoextend the thin-pool (both data and metadata) as needed in response
to low watermark, etc. See "Automatically extend thin pool LV" in
lvmthin manpage.
</pre>
</blockquote>
Maybe related to:<br>
<blockquote type="cite"><a class="moz-txt-link-freetext" href="https://bugzilla.redhat.com/show_bug.cgi?id=1097948">https://bugzilla.redhat.com/show_bug.cgi?id=1097948</a></blockquote>
<br>
</blockquote>
Hope this helps.<br>
<br>
<br>
<BR />
<BR />
<b style="color:#604c78"></b><br><span style="color:#604c78;"><font color="000000"><span style="mso-fareast-language:en-gb;" lang="NL">Met vriendelijke groet, With kind regards,<br><br></span>Jorick Astrego</font></span><b style="color:#604c78"><br><br>Netbulae Virtualization Experts </b><br><hr style="border:none;border-top:1px solid #ccc;"><table style="width: 522px"><tbody><tr><td style="width: 130px;font-size: 10px">Tel: 053 20 30 270</td> <td style="width: 130px;font-size: 10px">info@netbulae.eu</td> <td style="width: 130px;font-size: 10px">Staalsteden 4-3A</td> <td style="width: 130px;font-size: 10px">KvK 08198180</td></tr><tr> <td style="width: 130px;font-size: 10px">Fax: 053 20 30 271</td> <td style="width: 130px;font-size: 10px">www.netbulae.eu</td> <td style="width: 130px;font-size: 10px">7547 TA Enschede</td> <td style="width: 130px;font-size: 10px">BTW NL821234584B01</td></tr></tbody></table><br><hr style="border:none;border-top:1px solid #ccc;"><BR />
</body>
</html>