<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<div class="moz-cite-prefix">On 01/23/2016 02:17 PM, Steve Spence
wrote:<br>
</div>
<blockquote
cite="mid:CAJqrPrqx4TEg=62RdKzCzWUT_wxhaKDQwvOQt_Y++m8yn561Kw@mail.gmail.com"
type="cite">
<div dir="ltr">We've a simple two-server one volume arrangement,
replicating ~340k files (15GB) between our web servers.
<div><br>
</div>
<div>The servers are in AWS, sat in different availability
zones. One of the operations for this weekend is to add
another pair of machines, one in each AZ.</div>
<div><br>
</div>
<div>I've deployed the same OS image of the gluster server (3.6)
and was under the impression I could add a brick to the
existing replica simply by issuing the below:</div>
<div><br>
</div>
<div>gluster volume add-brick volume1 replica 3
pd-wfe3:/gluster-store<br>
</div>
<div><br>
</div>
<div>And then presumably would add the fourth server by
repeating the above with "replica 4" and the fourth server
name.</div>
<div><br>
</div>
<div>The operation appeared to succeed, the brick appears
alongside the others:</div>
<div><br>
</div>
<div>
<div>Status: Started</div>
<div>Number of Bricks: 1 x 3 = 3</div>
<div>Transport-type: tcp</div>
<div>Bricks:</div>
<div>Brick1: pd-wfe1:/gluster-store</div>
<div>Brick2: pd-wfe2:/gluster-store</div>
<div>Brick3: pd-wfe3:/gluster-store</div>
</div>
<div><br>
</div>
<div>but almost immediately pd-wfe1 crept up to 100% CPU with
the gluster processes, and nginx began timing out serving
content from the volume.</div>
</div>
</blockquote>
<br>
Could you disable client-side healing?<br>
"gluster volume set <volname> cluster.entry-self-heal off<br>
"gluster volume set <volname> cluster.data-self-heal off<br>
"gluster volume set <volname> cluster.metadata-self-heal off<br>
<br>
We are in the process of making this experience smooth for 3.8. by
introducing throttling of self-heal traffic, automatic healing.<br>
+Anuradha,<br>
Could you give him the steps he need to perform after doing
add-brick until the patch you sent is merged?<br>
<br>
Pranith<br>
<blockquote
cite="mid:CAJqrPrqx4TEg=62RdKzCzWUT_wxhaKDQwvOQt_Y++m8yn561Kw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>The glusterfs-glusterd-vol log is filled with this error at
pd-wfe1:</div>
<div><br>
</div>
<div>[2016-01-23 08:43:28.459215] W [socket.c:620:__socket_rwv]
0-management: readv on
/var/run/c8bc2f99e7584cb9cf077c4f98d1db2e.socket failed
(Invalid argument)</div>
<div><br>
</div>
<div>while I see this error for the log named by the mount
point:</div>
<div><br>
</div>
<div>[2016-01-23 08:43:28.986379] W
[client-rpc-fops.c:306:client3_3_mkdir_cbk]
2-volume1-client-2: remote operation failed: Permission
denied. Path: (null)<br>
</div>
<div><br>
</div>
<div>Does anyone have any suggestions how to proceed? I would
appreciate any input on this one.</div>
<div><br>
</div>
<div>Steve</div>
<div><br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>