<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Im not sure what you mean by Mode 1000 are you referring to the .gluster directory or the Brick files them selves?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Joe Julian [mailto:joe@julianfamily.org]
<br>
<b>Sent:</b> Sunday, February 15, 2015 10:54 AM<br>
<b>To:</b> Thomas Holkenbrink; 'gluster-users@gluster.org'<br>
<b>Subject:</b> Re: [Gluster-users] Gluster replicate-brick issues (Distrubuted-Replica)<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">Of those missing files, are they maybe dht link files? Mode 1000, size 0.<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">On February 14, 2015 12:58:12 AM PST, Thomas Holkenbrink <<a href="mailto:thomas.holkenbrink@fibercloud.com">thomas.holkenbrink@fibercloud.com</a>> wrote:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">We have tried to migrate a Brick from one server to another using the following commands. But the data is NOT being replicated… and the BRICK is not showing up anymore.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">Gluster still appears to be working but the Bricks are not balanced and I need to add the other Brick for Server3 that I don’t want to do until after Server1:Brick2 gets replicated.<o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">This is the command to create the Original Volume:<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server1 ~]#
</span><span style="font-size:8.0pt;font-family:"Courier New";color:#002060">gluster volume create Storage1 replica 2 transport tcp Server1:/exp/br01/brick1 Server2:/exp/br01/brick1 Server1:/exp/br02/brick2 Server2:/exp/br02/brick2<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;color:#002060"> <o:p></o:p></span></p>
<p><span style="font-size:8.0pt;color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">This is the Current configuration BEFORE the migration.. Server3 has been Peer Probed successfully but that has been it<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New";color:black">[root@Server1 ~]# gluster --version<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New";color:black">glusterfs 3.6.2 built on Jan 22 2015 12:58:11</span><span style="color:black"><o:p></o:p></span></p>
<p><span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# gluster volume status<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Status of volume: Storage1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Gluster process Port Online Pid<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server1:/exp/br01/brick1 49152 Y 2167<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server2:/exp/br01/brick1 49152 Y 2192<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:8.0pt;font-family:"Courier New";color:red">Brick Server1:/exp/br02/brick2 49153 Y 2172 <--- this is the one that goes missing<o:p></o:p></span></b></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server2:/exp/br02/brick2 49153 Y 2193<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on localhost 2049 Y 2181<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on localhost N/A Y 2186<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on Server2 2049 Y 2205<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on Server2 N/A Y 2210<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on Server3 2049 Y 6015<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on Server3 N/A Y 6016<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Task Status of Volume Storage1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">There are no active volume tasks<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server1 ~]# gluster volume info<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Volume Name: Storage1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Type: Distributed-Replicate<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Volume ID: 9616ce42-48bd-4fe3-883f-decd6c4fcd00<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Status: Started<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of Bricks: 2 x 2 = 4<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Transport-type: tcp<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Bricks:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick1: Server1:/exp/br01/brick1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick2: Server2:/exp/br01/brick1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick3: Server1:/exp/br02/brick2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick4: Server2:/exp/br02/brick2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Options Reconfigured:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">diagnostics.brick-log-level: WARNING<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">diagnostics.client-log-level: WARNING<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">cluster.entry-self-heal: off<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">cluster.data-self-heal: off<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">cluster.metadata-self-heal: off<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.cache-size: 1024MB<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.cache-max-file-size: 2MB<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.cache-refresh-timeout: 1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.stat-prefetch: off<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.read-ahead: on<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.quick-read: off<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.write-behind-window-size: 4MB<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.flush-behind: on<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.write-behind: on<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.io-thread-count: 32<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.io-cache: on<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">network.ping-timeout: 2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">nfs.addr-namelookup: off<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.strict-write-ordering: on<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server1 ~]#<o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">So we start the Migration of the Brick to the new server using the replace Brick command<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# volname=Storage1<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# from=Server1:/exp/br02/brick2<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# to=Server3:/exp/br02/brick2<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# gluster volume replace-brick $volname $from $to start<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">All replace-brick commands except commit force are deprecated. Do you want to continue? (y/n) y<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">volume replace-brick: success: replace-brick started successfully<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">ID: 0062d555-e7eb-4ebe-a264-7e0baf6e7546<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# gluster volume replace-brick $volname $from $to status<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">All replace-brick commands except commit force are deprecated. Do you want to continue? (y/n) y<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">volume replace-brick: success: Number of files migrated = 281 Migration complete<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">At this point everything seems to be in order with no outstanding issues.<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# gluster volume status<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Status of volume: Storage1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Gluster process Port Online Pid<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server1:/exp/br01/brick1 49152 Y 2167<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server2:/exp/br01/brick1 49152 Y 2192<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server1:/exp/br02/brick2 49153 Y 27557<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server2:/exp/br02/brick2 49153 Y 2193<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on localhost 2049 Y 27562<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on localhost N/A Y 2186<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on Server2 2049 Y 2205<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on Server2 N/A Y 2210<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on Server3 2049 Y 6015<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on Server3 N/A Y 6016<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Task Status of Volume Storage1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Task : Replace brick<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">ID : 0062d555-e7eb-4ebe-a264-7e0baf6e7546<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Source Brick : Server1:/exp/br02/brick2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Destination Brick : Server3:/exp/br02/brick2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Status : completed<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">The volume reports that the replace Brick command completed.. so the next step is to commit the change<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# gluster volume replace-brick $volname $from $to commit<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">All replace-brick commands except commit force are deprecated. Do you want to continue? (y/n) y<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">volume replace-brick: success: replace-brick commit successful<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">At this point when I take a look at the status I see that the OLD brick is now missing (Server1:/exp/br02/brick2) AND I don’t see the new Brick… WTF… panic!<o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# gluster volume status<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Status of volume: Storage1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Gluster process Port Online Pid<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server1:/exp/br01/brick1 49152 Y 2167<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server2:/exp/br01/brick1 49152 Y 2192<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server2:/exp/br02/brick2 49153 Y 2193<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on localhost 2049 Y 28906<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on localhost N/A Y 28911<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on Server2 2049 Y 2205<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on Server2 N/A Y 2210<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on Server3 2049 Y 6015<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on Server3 N/A Y 6016<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Task Status of Volume Storage1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">There are no active volume tasks<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">After the commit on Server1 it does not have the Tasks listed anymore… yet server2 and server3 see this<o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server2 ~]# gluster volume status<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Status of volume: Storage1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Gluster process Port Online Pid<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server1:/exp/br01/brick1 49152 Y 2167<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server2:/exp/br01/brick1 49152 Y 2192<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server2:/exp/br02/brick2 49153 Y 2193<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">NFS Server on localhost 2049 Y 2205<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Self-heal Daemon on localhost N/A Y 2210<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">NFS Server on 10.45.16.17 2049 Y 28906<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Self-heal Daemon on 10.45.16.17 N/A Y 28911<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">NFS Server on server3 2049 Y 6015<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Self-heal Daemon on server3 N/A Y 6016<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Task Status of Volume Storage1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Task : Replace brick<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">ID : 0062d555-e7eb-4ebe-a264-7e0baf6e7546<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Source Brick : Server1:/exp/br02/brick2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Destination Brick : server3:/exp/br02/brick2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Status : completed<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">If I navigate the brick on Server3 the brick is NOT empty.. but missing A LOT! It’s like the replace brick stopped… and never restarted again.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">The replace brick reported back </span>
“Number of files migrated = 281 Migration complete” <span style="color:#002060">
but when I look on Server3 Brick I get:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black"> [root@Server3 brick2]# find . -type f -print | wc -l<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">16<o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">I’m missing 265 files.. (they still exist on the OLD brick.. but how can I move it?)<o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">If I try to add the old brick back with another brick on the new server as such<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server1 ~]# gluster volume add-brick Storage1 Server1:/exp/br02/brick2 Server3:/exp/br01/brick1</span><span style="font-size:8.0pt;font-family:"Courier New";color:#002060"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">volume add-brick: failed: /exp/br02/brick2 is already part of a volume<o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">Im fearfull of running:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server1 ~]# setfattr -n trusted.glusterfs.volume-id -v 0x$(grep volume-id /var/lib/glusterd/vols/$volname/info | cut -d= -f2 | sed 's/-//g') /exp/br02/brick2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">Although it should allow me to add the brick<o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">Gluster Heal info returns:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server2 ~]# gluster volume heal Storage1 info<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server1:/exp/br01/brick1/<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of entries: 0<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server2:/exp/br01/brick1/<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of entries: 0<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server1:/exp/br02/brick2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Status: Transport endpoint is not connected<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server2:/exp/br02/brick2/<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of entries: 0<o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">I have restarted glusterd numerous times.<o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">at this time I’m not sure where to go from here… I know that the Server1:/exp/br02/brick2 still has all the data.. and Server3:/exp/br01/brick1 is not complete<o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">How do I actually get the brick to replicate?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">How can I add Server1:/exp/br02/brick2 back into the trusted pool if I can’t replicate it, or re-add it?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">How can I fix this to get it back into a replicated state between the three servers?<o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">Thomas<o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">----DATA----<o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">Gluster volume info at this point<o:p></o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server1 ~]# gluster volume info<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Volume Name: Storage1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Type: Distributed-Replicate<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Volume ID: 9616ce42-48bd-4fe3-883f-decd6c4fcd00<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Status: Started<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of Bricks: 2 x 2 = 4<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Transport-type: tcp<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Bricks:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick1: Server1:/exp/br01/brick1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick2: Server2:/exp/br01/brick1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick3: server3:/exp/br02/brick2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick4: Server2:/exp/br02/brick2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Options Reconfigured:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">diagnostics.brick-log-level: WARNING<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">diagnostics.client-log-level: WARNING<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">cluster.entry-self-heal: off<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">cluster.data-self-heal: off<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">cluster.metadata-self-heal: off<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.cache-size: 1024MB<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.cache-max-file-size: 2MB<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.cache-refresh-timeout: 1<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.stat-prefetch: off<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.read-ahead: on<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.quick-read: off<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.write-behind-window-size: 4MB<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.flush-behind: on<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.write-behind: on<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.io-thread-count: 32<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.io-cache: on<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">network.ping-timeout: 2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">nfs.addr-namelookup: off<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.strict-write-ordering: on<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server1 ~]#<o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@server3 brick2]# gluster volume heal Storage1 info<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server1:/exp/br01/brick1/<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of entries: 0<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server2:/exp/br01/brick1/<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of entries: 0<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server3:/exp/br02/brick2/<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of entries: 0<o:p></o:p></span></p>
<p><span style="font-size:8.0pt;font-family:"Courier New";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server2:/exp/br02/</span><span style="font-size:8.0pt;font-family:"Courier New";color:#002060">brick2/<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of entries: 0<o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#002060">Gluster LOG ( there are a few errors but I’m not sure how to decipher them)<o:p></o:p></span></p>
<p><span style="color:#002060"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:29:19.862809] I [MSGID: 106005] [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick Server1:/exp/br02/brick2 has disconnected from
glusterd.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:29:19.862836] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/7565ec897c6454bd3e2f4800250a7221.socket failed (Invalid argument)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:29:19.862853] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: nfs has disconnected from glusterd.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:29:19.953762] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /exp/br02/brick2 on port 49153<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:31:12.977450] I [glusterd-replace-brick.c:99:__glusterd_handle_replace_brick] 0-management: Received replace brick req<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:31:12.977495] I [glusterd-replace-brick.c:154:__glusterd_handle_replace_brick] 0-management: Received replace brick status request<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:31:13.048852] I [glusterd-replace-brick.c:1412:rb_update_srcbrick_port] 0-: adding src-brick port no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:31:19.588380] I [glusterd-replace-brick.c:99:__glusterd_handle_replace_brick] 0-management: Received replace brick req<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:31:19.588422] I [glusterd-replace-brick.c:154:__glusterd_handle_replace_brick] 0-management: Received replace brick status request<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:31:19.661101] I [glusterd-replace-brick.c:1412:rb_update_srcbrick_port] 0-: adding src-brick port no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:31:45.115355] W [glusterd-op-sm.c:4021:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:31:45.118597] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Storage1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:10.956357] I [glusterd-replace-brick.c:99:__glusterd_handle_replace_brick] 0-management: Received replace brick req<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:10.956385] I [glusterd-replace-brick.c:154:__glusterd_handle_replace_brick] 0-management: Received replace brick commit request<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:11.028472] I [glusterd-replace-brick.c:1412:rb_update_srcbrick_port] 0-: adding src-brick port no<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:12.122552] I [glusterd-utils.c:6276:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:12.131836] I [glusterd-utils.c:6281:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:12.141107] I [glusterd-utils.c:6286:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:12.150375] I [glusterd-utils.c:6291:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:12.159630] I [glusterd-utils.c:6296:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:12.168889] I [glusterd-utils.c:6301:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.254689] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.254799] W [socket.c:2992:socket_connect] 0-management: Ignore failed connection attempt on , (No such file or directory)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.257790] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.257908] W [socket.c:2992:socket_connect] 0-management: Ignore failed connection attempt on , (No such file or directory)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.258031] W [socket.c:611:__socket_rwv] 0-socket.management: writev on 127.0.0.1:1019 failed (Broken pipe)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.258111] W [socket.c:611:__socket_rwv] 0-socket.management: writev on 127.0.0.1:1021 failed (Broken pipe)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.258130] W [socket.c:611:__socket_rwv] 0-socket.management: writev on 10.45.16.17:1018 failed (Broken pipe)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.711948] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=588 max=0 total=0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.711967] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=124 max=0 total=0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.712008] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=588 max=0 total=0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.712021] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=124 max=0 total=0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.731311] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=588 max=0 total=0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.731326] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=124 max=0 total=0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.731356] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /exp/br02/brick2 on port 49153<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.823129] I [socket.c:2344:socket_event_handler] 0-transport: disconnecting now<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.840668] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/7565ec897c6454bd3e2f4800250a7221.socket failed (Invalid argument)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.840693] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: nfs has disconnected from glusterd.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.840712] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/ac4c043d3c6a2e5159c86e8c75c51829.socket failed (Invalid argument)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.840728] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: glustershd has disconnected from glusterd.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:14.729667] E [glusterd-rpc-ops.c:1169:__glusterd_commit_op_cbk] 0-management: Received commit RJT from uuid: 294aa603-ec24-44b9-864b-0fe743faa8d9<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:14.743623] E [glusterd-rpc-ops.c:1169:__glusterd_commit_op_cbk] 0-management: Received commit RJT from uuid: 92aabaf4-4b6c-48da-82b6-c465aff2ec6d<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:18.762975] W [glusterd-op-sm.c:4021:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:18.764552] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Storage1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:18.769051] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not
match. Not aggregating tasks status.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:18.769070] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from node/brick<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:18.771095] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not
match. Not aggregating tasks status.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:18.771108] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from node/brick<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:48.570796] W [glusterd-op-sm.c:4021:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:48.572352] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Storage1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:48.576899] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not
match. Not aggregating tasks status.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:48.576918] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from node/brick<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:48.578982] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not
match. Not aggregating tasks status.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:48.579001] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from node/brick<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:36:57.840738] W [glusterd-op-sm.c:4021:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:36:57.842370] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Storage1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:36:57.846919] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not
match. Not aggregating tasks status.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:36:57.846941] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from node/brick<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:36:57.849026] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not
match. Not aggregating tasks status.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:36:57.849046] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from node/brick<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:37:20.208081] W [glusterd-op-sm.c:4021:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:37:20.211279] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Storage1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:37:20.215792] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not
match. Not aggregating tasks status.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:37:20.215809] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from node/brick<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:37:20.216295] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not
match. Not aggregating tasks status.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:37:20.216308] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from node/brick<o:p></o:p></span></p>
<pre style="text-align:center"><hr size="2" width="100%" align="center"></pre>
<pre><br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br><a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></pre>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><br>
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my brevity.<o:p></o:p></span></p>
</div>
</body>
</html>