<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hi,</div><div class=""><br class=""></div><div class="">I always though that hardware RAID is a requirement for SDS as it hides all dirty work with raw disks from software which just cannot deal with all kinds of hardware faults. If disk starts to experiencing long delays, then after about 7 seconds RAID controller marks this disk as failed (this is what TLER/ERC for). If your RAID card behaves differently you can try to decrease OS timeouts (disk driver will send i/o error to filesystem then). But in this case complete brick will go offline and you will definitely need replicated setup.</div><br class=""><div><blockquote type="cite" class=""><div class="">24 нояб. 2016 г., в 9:12, Christian Rice <<a href="mailto:crice@pandora.com" class="">crice@pandora.com</a>> написал(а):</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; font-family: Helvetica; font-size: 14.25px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);"><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri;" class=""><span style="font-size: 11pt;" class="">This is a long-standing problem for me, and I’m wondering how to insulate myself from it…pardon the long-windedness in advance.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri;" class=""><span style="font-size: 11pt;" class=""> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri;" class=""><span style="font-size: 11pt;" class="">I use gluster internationally as regional repositories of files, and it’s pretty constantly being rsync’d to (ie, written to solely by rsync, optimized with –inplace or similar).<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri;" class=""><span style="font-size: 11pt;" class=""> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri;" class=""><span style="font-size: 11pt;" class="">These regional repositories are also being read from, each to the tune of 10-50MB/s. Each gluster pool is anywhere between 4 to 16 servers, each with one brick of RAID6, all pools in a distributed-only config. I’m not currently using distributed-replicated, but even that configuration is not immune to my problem.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri;" class=""><span style="font-size: 11pt;" class=""> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri;" class=""><span style="font-size: 11pt;" class="">So, here’s the problem:<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri;" class=""><span style="font-size: 11pt;" class=""> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri;" class=""><span style="font-size: 11pt;" class="">If one disk on one gluster brick experiences timeouts, all the gluster clients block. This is likely because the rate at which the disks are being exercised by rsyncs (writes and stats) plus reads (client file access) causes an overwhelming backlog of gluster ops, something perhaps is bottlenecked and locking up, but in general it’s fairly useless to me. Running a ‘df’ hangs completely.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri;" class=""><span style="font-size: 11pt;" class=""> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri;" class=""><span style="font-size: 11pt;" class="">This has been an issue for me for years. My usual procedure is to manually fail the disk that’s experiencing timeouts, if it hasn’t been ejected already by the raid controller, and remove the load from the gluster file system—it only takes a fraction of a minute before the gluster volume recovers and I can add the load back. Rebuilding parity to the brick’s raid is not the problem—it’s the moments before the disk ultimately fails that causes the backlog of requests that really causes problems.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri;" class=""><span style="font-size: 11pt;" class=""> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri;" class=""><span style="font-size: 11pt;" class="">I’m looking for advice as to how to insulate myself from this problem better. My RAID cards don’t support modifying disk timeouts to be incredibly short. I can see disk timeout messages from the raid card, and write an omprog function to fail the disk, but that’s kinda brutal. Maybe I could get a different raid card that supports shorter timeouts or fast disk failures, but if anyone has experience with, say md raid1 not having this problem, or something similar, it might be worth the expense to go that route.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri;" class=""><span style="font-size: 11pt;" class=""> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri;" class=""><span style="font-size: 11pt;" class="">If my memory is correct, gluster still has this problem with a distributed-replicated configuration, because writes need to succeed on both leafs before an operation is considered complete, so a timeout on one node is still detrimental.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri;" class=""><span style="font-size: 11pt;" class=""> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: Calibri;" class=""><span style="font-size: 11pt;" class="">Insight, experience designing around this, tunables I haven’t considered—I’ll take anything. I really like gluster, I’ll keep using it, but this is its Achille’s heel for me. Is there a magic bullet? Or do I just need to fail faster?<o:p class=""></o:p></span></div></div><span style="font-family: Helvetica; font-size: 14.25px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 14.25px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class=""><span style="font-family: Helvetica; font-size: 14.25px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">Gluster-users mailing list</span><br style="font-family: Helvetica; font-size: 14.25px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class=""><a href="mailto:Gluster-users@gluster.org" style="color: rgb(149, 79, 114); text-decoration: underline; font-family: Helvetica; font-size: 14.25px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">Gluster-users@gluster.org</a><br style="font-family: Helvetica; font-size: 14.25px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class=""><a href="http://www.gluster.org/mailman/listinfo/gluster-users" style="color: rgb(149, 79, 114); text-decoration: underline; font-family: Helvetica; font-size: 14.25px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">http://www.gluster.org/mailman/listinfo/gluster-users</a></div></blockquote></div><br class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">--</div><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class="">Dmitry Glushenok</div><div class="">Jet Infosystems</div></div></div></div></div></div></div></div></div></div></div></div>
</div>
<br class=""></body></html>