<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:Calibri;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">This is a long-standing problem for me, and I’m wondering how to insulate myself from it…pardon the long-windedness in advance.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I use gluster internationally as regional repositories of files, and it’s pretty constantly being rsync’d to (ie, written to solely by rsync, optimized with –inplace or similar).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">These regional repositories are also being read from, each to the tune of 10-50MB/s. Each gluster pool is anywhere between 4 to 16 servers, each with one brick of RAID6, all pools in a distributed-only config.
I’m not currently using distributed-replicated, but even that configuration is not immune to my problem.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">So, here’s the problem:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">If one disk on one gluster brick experiences timeouts, all the gluster clients block. This is likely because the rate at which the disks are being exercised by rsyncs (writes and stats) plus reads (client
file access) causes an overwhelming backlog of gluster ops, something perhaps is bottlenecked and locking up, but in general it’s fairly useless to me. Running a ‘df’ hangs completely.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This has been an issue for me for years. My usual procedure is to manually fail the disk that’s experiencing timeouts, if it hasn’t been ejected already by the raid controller, and remove the load from the
gluster file system—it only takes a fraction of a minute before the gluster volume recovers and I can add the load back. Rebuilding parity to the brick’s raid is not the problem—it’s the moments before the disk ultimately fails that causes the backlog of
requests that really causes problems.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I’m looking for advice as to how to insulate myself from this problem better. My RAID cards don’t support modifying disk timeouts to be incredibly short. I can see disk timeout messages from the raid card,
and write an omprog function to fail the disk, but that’s kinda brutal. Maybe I could get a different raid card that supports shorter timeouts or fast disk failures, but if anyone has experience with, say md raid1 not having this problem, or something similar,
it might be worth the expense to go that route.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">If my memory is correct, gluster still has this problem with a distributed-replicated configuration, because writes need to succeed on both leafs before an operation is considered complete, so a timeout on
one node is still detrimental.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Insight, experience designing around this, tunables I haven’t considered—I’ll take anything. I really like gluster, I’ll keep using it, but this is its Achille’s heel for me. Is there a magic bullet? Or
do I just need to fail faster?<o:p></o:p></span></p>
</div>
</body>
</html>