<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<big><tt>Hi,<br>
<br>
</tt><tt></tt><tt>W</tt><tt>e are planning to introduce a
throttling xlator on the server (brick)</tt><tt><br>
</tt><tt>process to regulate FOPS. The main motivation is to solve
complaints about</tt><tt><br>
</tt><tt>AFR selfheal taking too much of CPU resources. (due to
too many fops for entry </tt><tt><br>
</tt><tt>self-heal, rchecksums for data self-heal etc.)</tt><tt><br>
</tt><tt><br>
</tt><tt>The throttling is achieved using the Token Bucket Filter
algorithm (TBF). TBF</tt><tt><br>
</tt><tt>is already used by bitrot's bitd signer (which is a
client process) in </tt><tt><br>
</tt><tt>gluster to regulate the CPU intensive check-sum
calculation. By putting the </tt><tt><br>
</tt><tt>logic on the brick side, multiple clients- selfheal,
bitrot, rebalance or </tt><tt><br>
</tt><tt>even the mounts themselves can avail the benefits of
throttling.</tt><tt><br>
</tt><tt><br>
</tt><tt>The TBF algorithm in a nutshell is as follows: There is a
bucket which is filled</tt><tt><br>
</tt><tt>at a steady (configurable) rate with tokens. Each FOP
will need a fixed amount</tt><tt><br>
</tt><tt>of tokens to be processed. If the bucket has that many
tokens, the FOP is</tt><tt><br>
</tt><tt>allowed and that many tokens are removed from the bucket.
If not, the FOP is</tt><tt><br>
</tt><tt>queued until the bucket is filled.</tt><tt><br>
</tt><tt><br>
</tt><tt>The xlator will need to reside above io-threads and can
have different buckets,</tt><tt><br>
</tt><tt>one per client. There has to be a communication mechanism
between the client and</tt><tt><br>
</tt><tt>the brick (IPC?) to tell what FOPS need to be regulated
from it, and the no. of</tt><tt><br>
</tt><tt>tokens needed etc. These need to be re configurable via
appropriate mechanisms.</tt><tt><br>
</tt><tt>Each bucket will have a token filler thread which will
fill the tokens in it. </tt><tt><br>
</tt><tt>The main thread will enqueue heals in a list in the
bucket if there aren't </tt><tt><br>
</tt><tt>enough tokens. Once the token filler detects some FOPS
can be serviced, it will </tt><tt><br>
</tt><tt>send a cond-broadcast to a dequeue thread which will
process (stack wind) all </tt><tt><br>
</tt><tt>the FOPS that have the required no. of tokens from all
buckets.<br>
<br>
This is just a high level abstraction: requesting feedback on
any aspect of <br>
this feature. what kind of mechanism is best between the
client/bricks for <br>
tuning various parameters? What other requirements do you
foresee?<br>
<br>
Thanks,<br>
Ravi<br>
</tt></big>
</body>
</html>