<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
hey,<br>
Which process is consuming so much cpu? I went through the
logs you gave me. I see that the following files are in gfid
mismatch state:<br>
<br>
<066e4525-8f8b-43aa-b7a1-86bbcecc68b9/safebrowsing-backup>,<br>
<1d48754b-b38c-403d-94e2-0f5c41d5f885/recovery.bak>,<br>
<ddc92637-303a-4059-9c56-ab23b1bb6ae9/patch0008.cnvrg>,<br>
<br>
Could you give me the output of "ls
<brick-path>/indices/xattrop | wc -l" output on all the bricks
which are acting this way? This will tell us the number of pending
self-heals on the system.<br>
<br>
Pranith<br>
<br>
<div class="moz-cite-prefix">On 01/20/2016 09:26 PM, David Robinson
wrote:<br>
</div>
<blockquote
cite="mid:emfb9a6149-226c-41af-901c-6c3df3d939ec@dfrobins-vaio"
type="cite">
<style id="eMClientCss">blockquote.cite { margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px; border-left: 1px solid #cccccc }
blockquote.cite2 {margin-left: 5px; margin-right: 0px; padding-left: 10px; padding-right:0px; border-left: 1px solid #cccccc; margin-top: 3px; padding-top: 0px; }
.plain pre, .plain tt { font-family: monospace; font-size: 100%; font-weight: normal; font-style: normal;}
a img { border: 0px; }body {font-family: Times New Roman;font-size: 12pt;}
.plain pre, .plain tt {font-family: Times New Roman;font-size: 12pt;}
</style>
<style>#xf4f1402e9c124036a307874ac26fcbf8 BLOCKQUOTE.cite
{PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px}
#xf4f1402e9c124036a307874ac26fcbf8 BLOCKQUOTE.cite2
{MARGIN-TOP: 3px; PADDING-TOP: 0px; PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px}
#xf4f1402e9c124036a307874ac26fcbf8 .plain PRE, #xf4f1402e9c124036a307874ac26fcbf8 .plain TT
{FONT-SIZE: 100%; FONT-FAMILY: monospace; FONT-WEIGHT: normal; FONT-STYLE: normal}
#xf4f1402e9c124036a307874ac26fcbf8 A IMG
{BORDER-TOP: 0px; BORDER-RIGHT: 0px; BORDER-BOTTOM: 0px; BORDER-LEFT: 0px}
#xf4f1402e9c124036a307874ac26fcbf8 .plain PRE, #xf4f1402e9c124036a307874ac26fcbf8 .plain TT, #xf4f1402e9c124036a307874ac26fcbf8
{FONT-SIZE: 12pt; FONT-FAMILY: Times New Roman}
#xf4f1402e9c124036a307874ac26fcbf8 #xb0ba93f1518347a49cb4c8e25d6d3e59 BLOCKQUOTE.cite
{PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px}
#xf4f1402e9c124036a307874ac26fcbf8 #xb0ba93f1518347a49cb4c8e25d6d3e59 BLOCKQUOTE.cite2
{MARGIN-TOP: 3px; PADDING-TOP: 0px; PADDING-LEFT: 10px; MARGIN-LEFT: 5px; BORDER-LEFT: #cccccc 1px solid; PADDING-RIGHT: 0px; MARGIN-RIGHT: 0px}
#xf4f1402e9c124036a307874ac26fcbf8 #xb0ba93f1518347a49cb4c8e25d6d3e59 .plain PRE, #xf4f1402e9c124036a307874ac26fcbf8 #xb0ba93f1518347a49cb4c8e25d6d3e59 .plain TT
{FONT-SIZE: 100%; FONT-FAMILY: monospace; FONT-WEIGHT: normal; FONT-STYLE: normal}
#xf4f1402e9c124036a307874ac26fcbf8 #xb0ba93f1518347a49cb4c8e25d6d3e59 A IMG
{BORDER-TOP: 0px; BORDER-RIGHT: 0px; BORDER-BOTTOM: 0px; BORDER-LEFT: 0px}
#xf4f1402e9c124036a307874ac26fcbf8 #xb0ba93f1518347a49cb4c8e25d6d3e59, #xf4f1402e9c124036a307874ac26fcbf8 #xb0ba93f1518347a49cb4c8e25d6d3e59 .plain PRE, #xf4f1402e9c124036a307874ac26fcbf8 #xb0ba93f1518347a49cb4c8e25d6d3e59 .plain TT
{FONT-SIZE: 12pt; FONT-FAMILY: Times New Roman}
#xf4f1402e9c124036a307874ac26fcbf8 #xb0ba93f1518347a49cb4c8e25d6d3e59 DIV.MsoNormal, #xf4f1402e9c124036a307874ac26fcbf8 #xb0ba93f1518347a49cb4c8e25d6d3e59 LI.MsoNormal
{FONT-SIZE: 11pt; FONT-FAMILY: "Calibri","sans-serif"; MARGIN: 0in 0in 0pt}
#xf4f1402e9c124036a307874ac26fcbf8 #xb0ba93f1518347a49cb4c8e25d6d3e59 SPAN.MsoHyperlink
{TEXT-DECORATION: underline; COLOR: blue; mso-style-priority: 99}
#xf4f1402e9c124036a307874ac26fcbf8 #xb0ba93f1518347a49cb4c8e25d6d3e59 SPAN.MsoHyperlinkFollowed
{TEXT-DECORATION: underline; COLOR: purple; mso-style-priority: 99}
</style>
<div>resending with parsed logs... </div>
<div> </div>
<div id="xf4f1402e9c124036a307874ac26fcbf8" class="class">
<blockquote class="cite2"
cite="em5ee26b0e-002a-4230-bdec-3020b98cff3c@dfrobins-vaio"
type="cite">
<div> </div>
<div> </div>
<div id="xb0ba93f1518347a49cb4c8e25d6d3e59">
<blockquote class="cite2"
cite="eme3b2cb80-8be2-4fa5-9d08-4710955e237c@dfrobins-vaio"
type="cite">
<div>I am having issues with 3.6.6 where the load will
spike up to 800% for one of the glusterfsd processes and
the users can no longer access the system. If I reboot
the node, the heal will finish normally after a few
minutes and the system will be responsive, but a few
hours later the issue will start again. It look like it
is hanging in a heal and spinning up the load on one of
the bricks. The heal gets stuck and says it is crawling
and never returns. After a few minutes of the heal
saying it is crawling, the load spikes up and the mounts
become unresponsive.</div>
<div> </div>
<div>Any suggestions on how to fix this? It has us
stopped cold as the user can no longer access the
systems when the load spikes... Logs attached.</div>
<div> </div>
<div>System setup info is: </div>
<div> </div>
<div>[root@gfs01a ~]# gluster volume info homegfs<br>
<br>
Volume Name: homegfs<br>
Type: Distributed-Replicate<br>
Volume ID: 1e32672a-f1b7-4b58-ba94-58c085e59071<br>
Status: Started<br>
Number of Bricks: 4 x 2 = 8<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: gfsib01a.corvidtec.com:/data/brick01a/homegfs<br>
Brick2: gfsib01b.corvidtec.com:/data/brick01b/homegfs<br>
Brick3: gfsib01a.corvidtec.com:/data/brick02a/homegfs<br>
Brick4: gfsib01b.corvidtec.com:/data/brick02b/homegfs<br>
Brick5: gfsib02a.corvidtec.com:/data/brick01a/homegfs<br>
Brick6: gfsib02b.corvidtec.com:/data/brick01b/homegfs<br>
Brick7: gfsib02a.corvidtec.com:/data/brick02a/homegfs<br>
Brick8: gfsib02b.corvidtec.com:/data/brick02b/homegfs<br>
Options Reconfigured:<br>
performance.io-thread-count: 32<br>
performance.cache-size: 128MB<br>
performance.write-behind-window-size: 128MB<br>
server.allow-insecure: on<br>
network.ping-timeout: 42<br>
storage.owner-gid: 100<br>
geo-replication.indexing: off<br>
geo-replication.ignore-pid-check: on<br>
changelog.changelog: off<br>
changelog.fsync-interval: 3<br>
changelog.rollover-time: 15<br>
server.manage-gids: on<br>
diagnostics.client-log-level: WARNING</div>
<div> </div>
<div>[root@gfs01a ~]# rpm -qa | grep gluster<br>
gluster-nagios-common-0.1.1-0.el6.noarch<br>
glusterfs-fuse-3.6.6-1.el6.x86_64<br>
glusterfs-debuginfo-3.6.6-1.el6.x86_64<br>
glusterfs-libs-3.6.6-1.el6.x86_64<br>
glusterfs-geo-replication-3.6.6-1.el6.x86_64<br>
glusterfs-api-3.6.6-1.el6.x86_64<br>
glusterfs-devel-3.6.6-1.el6.x86_64<br>
glusterfs-api-devel-3.6.6-1.el6.x86_64<br>
glusterfs-3.6.6-1.el6.x86_64<br>
glusterfs-cli-3.6.6-1.el6.x86_64<br>
glusterfs-rdma-3.6.6-1.el6.x86_64<br>
samba-vfs-glusterfs-4.1.11-2.el6.x86_64<br>
glusterfs-server-3.6.6-1.el6.x86_64<br>
glusterfs-extra-xlators-3.6.6-1.el6.x86_64<br>
</div>
<div> </div>
<div id="signature_old">
<div style="FONT-SIZE: 12pt; FONT-FAMILY: Times New
Roman"><span id="xc493ce70c1c64653841aa9f5ac6ad16e"><span
id="x834d59d53b4a45d8a1f25809459ef478">
<div class="WordSection1"> </div>
</span></span></div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-devel">http://www.gluster.org/mailman/listinfo/gluster-devel</a></pre>
</blockquote>
<br>
</body>
</html>