<html v="urn:schemas-microsoft-com:vml" o="urn:schemas-microsoft-com:office:office" w="urn:schemas-microsoft-com:office:word" m="http://schemas.microsoft.com/office/2004/12/omml"><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii" /><meta name="Generator" content="Microsoft Word 14 (filtered medium)" /><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang="EN-US" link="blue" vlink="purple">Of those missing files, are they maybe dht link files? Mode 1000, size 0.<br><br><div class="gmail_quote">On February 14, 2015 12:58:12 AM PST, Thomas Holkenbrink <thomas.holkenbrink@fibercloud.com> wrote:<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#002060">We have tried to migrate a Brick from one server to another using the following commands. But the data is NOT being replicated… and the BRICK is not showing up anymore.<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060">Gluster still appears to be working but the Bricks are not balanced and I need to add the other Brick for Server3 that I don’t want to do until after Server1:Brick2 gets replicated.<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">This is the command to create the Original Volume:<p></p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server1 ~]#
</span><span style="font-size:8.0pt;font-family:"Courier New";color:#002060">gluster volume create Storage1 replica 2 transport tcp Server1:/exp/br01/brick1 Server2:/exp/br01/brick1 Server1:/exp/br02/brick2 Server2:/exp/br02/brick2<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">This is the Current configuration BEFORE the migration.. Server3 has been Peer Probed successfully but that has been it<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New";color:black">[root@Server1 ~]# gluster --version<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New";color:black">glusterfs 3.6.2 built on Jan 22 2015 12:58:11</span><span style="color:black"><p></p></span></p>
<p class="MsoNormal"><span style="color:black"><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# gluster volume status<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Status of volume: Storage1<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Gluster process Port Online Pid<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">------------------------------------------------------------------------------<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server1:/exp/br01/brick1 49152 Y 2167<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server2:/exp/br01/brick1 49152 Y 2192<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:8.0pt;font-family:"Courier New";color:red">Brick Server1:/exp/br02/brick2 49153 Y 2172 <--- this is the one that goes missing<p></p></span></b></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server2:/exp/br02/brick2 49153 Y 2193<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on localhost 2049 Y 2181<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on localhost N/A Y 2186<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on Server2 2049 Y 2205<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on Server2 N/A Y 2210<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on Server3 2049 Y 6015<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on Server3 N/A Y 6016<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Task Status of Volume Storage1<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">------------------------------------------------------------------------------<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">There are no active volume tasks<p></p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server1 ~]# gluster volume info<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black"><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Volume Name: Storage1<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Type: Distributed-Replicate<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Volume ID: 9616ce42-48bd-4fe3-883f-decd6c4fcd00<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Status: Started<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of Bricks: 2 x 2 = 4<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Transport-type: tcp<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Bricks:<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick1: Server1:/exp/br01/brick1<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick2: Server2:/exp/br01/brick1<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick3: Server1:/exp/br02/brick2<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick4: Server2:/exp/br02/brick2<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Options Reconfigured:<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">diagnostics.brick-log-level: WARNING<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">diagnostics.client-log-level: WARNING<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">cluster.entry-self-heal: off<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">cluster.data-self-heal: off<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">cluster.metadata-self-heal: off<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.cache-size: 1024MB<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.cache-max-file-size: 2MB<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.cache-refresh-timeout: 1<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.stat-prefetch: off<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.read-ahead: on<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.quick-read: off<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.write-behind-window-size: 4MB<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.flush-behind: on<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.write-behind: on<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.io-thread-count: 32<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.io-cache: on<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">network.ping-timeout: 2<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">nfs.addr-namelookup: off<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.strict-write-ordering: on<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server1 ~]#<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">So we start the Migration of the Brick to the new server using the replace Brick command<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# volname=Storage1<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# from=Server1:/exp/br02/brick2<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# to=Server3:/exp/br02/brick2<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# gluster volume replace-brick $volname $from $to start<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">All replace-brick commands except commit force are deprecated. Do you want to continue? (y/n) y<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">volume replace-brick: success: replace-brick started successfully<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">ID: 0062d555-e7eb-4ebe-a264-7e0baf6e7546<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# gluster volume replace-brick $volname $from $to status<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">All replace-brick commands except commit force are deprecated. Do you want to continue? (y/n) y<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">volume replace-brick: success: Number of files migrated = 281 Migration complete<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">At this point everything seems to be in order with no outstanding issues.<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# gluster volume status<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Status of volume: Storage1<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Gluster process Port Online Pid<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">------------------------------------------------------------------------------<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server1:/exp/br01/brick1 49152 Y 2167<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server2:/exp/br01/brick1 49152 Y 2192<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server1:/exp/br02/brick2 49153 Y 27557<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server2:/exp/br02/brick2 49153 Y 2193<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on localhost 2049 Y 27562<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on localhost N/A Y 2186<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on Server2 2049 Y 2205<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on Server2 N/A Y 2210<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on Server3 2049 Y 6015<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on Server3 N/A Y 6016<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Task Status of Volume Storage1<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">------------------------------------------------------------------------------<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Task : Replace brick<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">ID : 0062d555-e7eb-4ebe-a264-7e0baf6e7546<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Source Brick : Server1:/exp/br02/brick2<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Destination Brick : Server3:/exp/br02/brick2<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Status : completed<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">The volume reports that the replace Brick command completed.. so the next step is to commit the change<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# gluster volume replace-brick $volname $from $to commit<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">All replace-brick commands except commit force are deprecated. Do you want to continue? (y/n) y<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">volume replace-brick: success: replace-brick commit successful<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">At this point when I take a look at the status I see that the OLD brick is now missing (Server1:/exp/br02/brick2) AND I don’t see the new Brick… WTF… panic!<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">[root@Server1 ~]# gluster volume status<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Status of volume: Storage1<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Gluster process Port Online Pid<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">------------------------------------------------------------------------------<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server1:/exp/br01/brick1 49152 Y 2167<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server2:/exp/br01/brick1 49152 Y 2192<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Brick Server2:/exp/br02/brick2 49153 Y 2193<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on localhost 2049 Y 28906<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on localhost N/A Y 28911<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on Server2 2049 Y 2205<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on Server2 N/A Y 2210<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">NFS Server on Server3 2049 Y 6015<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Self-heal Daemon on Server3 N/A Y 6016<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">Task Status of Volume Storage1<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">------------------------------------------------------------------------------<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New"">There are no active volume tasks<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">After the commit on Server1 it does not have the Tasks listed anymore… yet server2 and server3 see this<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server2 ~]# gluster volume status<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Status of volume: Storage1<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Gluster process Port Online Pid<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">------------------------------------------------------------------------------<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server1:/exp/br01/brick1 49152 Y 2167<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server2:/exp/br01/brick1 49152 Y 2192<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server2:/exp/br02/brick2 49153 Y 2193<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">NFS Server on localhost 2049 Y 2205<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Self-heal Daemon on localhost N/A Y 2210<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">NFS Server on 10.45.16.17 2049 Y 28906<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Self-heal Daemon on 10.45.16.17 N/A Y 28911<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">NFS Server on server3 2049 Y 6015<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Self-heal Daemon on server3 N/A Y 6016<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black"><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Task Status of Volume Storage1<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">------------------------------------------------------------------------------<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Task : Replace brick<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">ID : 0062d555-e7eb-4ebe-a264-7e0baf6e7546<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Source Brick : Server1:/exp/br02/brick2<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Destination Brick : server3:/exp/br02/brick2<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Status : completed<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New""><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">If I navigate the brick on Server3 the brick is NOT empty.. but missing A LOT! It’s like the replace brick stopped… and never restarted again.<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060">The replace brick reported back </span>
“Number of files migrated = 281 Migration complete” <span style="color:#002060">
but when I look on Server3 Brick I get:<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black"> [root@Server3 brick2]# find . -type f -print | wc -l<p></p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">16<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">I’m missing 265 files.. (they still exist on the OLD brick.. but how can I move it?)<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">If I try to add the old brick back with another brick on the new server as such<p></p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server1 ~]# gluster volume add-brick Storage1 Server1:/exp/br02/brick2 Server3:/exp/br01/brick1</span><span style="font-size:8.0pt;font-family:"Courier New";color:#002060"><p></p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">volume add-brick: failed: /exp/br02/brick2 is already part of a volume<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">Im fearfull of running:<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server1 ~]# setfattr -n trusted.glusterfs.volume-id -v 0x$(grep volume-id /var/lib/glusterd/vols/$volname/info | cut -d= -f2 | sed 's/-//g')
</span><span style="font-size:8.0pt;font-family:"Courier New";color:black">/exp/br02/brick2</span><span style="font-size:8.0pt;font-family:"Courier New";color:black"><p></p></span></p>
<p class="MsoNormal"><span style="color:#002060">Although it should allow me to add the brick<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">Gluster Heal info returns:<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server2 ~]# gluster volume heal Storage1 info<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server1:/exp/br01/brick1/<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of entries: 0<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black"><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server2:/exp/br01/brick1/<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of entries: 0<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black"><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server1:/exp/br02/brick2<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Status: Transport endpoint is not connected<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black"><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server2:/exp/br02/brick2/<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of entries: 0<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">I have restarted glusterd numerous times.<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">at this time I’m not sure where to go from here… I know that the Server1:/exp/br02/brick2 still has all the data.. and Server3:/exp/br01/brick1 is not complete<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">How do I actually get the brick to replicate?<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060">How can I add Server1:/exp/br02/brick2 back into the trusted pool if I can’t replicate it, or re-add it?<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060">How can I fix this to get it back into a replicated state between the three servers?<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">Thomas<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">----DATA----<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">Gluster volume info at this point<p></p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server1 ~]# gluster volume info<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black"><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Volume Name: Storage1<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Type: Distributed-Replicate<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Volume ID: 9616ce42-48bd-4fe3-883f-decd6c4fcd00<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Status: Started<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of Bricks: 2 x 2 = 4<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Transport-type: tcp<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Bricks:<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick1: Server1:/exp/br01/brick1<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick2: Server2:/exp/br01/brick1<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick3: server3:/exp/br02/brick2<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick4: Server2:/exp/br02/brick2<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Options Reconfigured:<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">diagnostics.brick-log-level: WARNING<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">diagnostics.client-log-level: WARNING<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">cluster.entry-self-heal: off<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">cluster.data-self-heal: off<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">cluster.metadata-self-heal: off<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.cache-size: 1024MB<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.cache-max-file-size: 2MB<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.cache-refresh-timeout: 1<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.stat-prefetch: off<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.read-ahead: on<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.quick-read: off<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.write-behind-window-size: 4MB<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.flush-behind: on<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.write-behind: on<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.io-thread-count: 32<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.io-cache: on<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">network.ping-timeout: 2<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">nfs.addr-namelookup: off<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">performance.strict-write-ordering: on<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@Server1 ~]#<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[root@server3 brick2]# gluster volume heal Storage1 info<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server1:/exp/br01/brick1/<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of entries: 0<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black"><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server2:/exp/br01/brick1/<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of entries: 0<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black"><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server3:/exp/br02/brick2/<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of entries: 0<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black"><p> </p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Brick Server2:/exp/br02/</span><span style="font-size:8.0pt;font-family:"Courier New";color:#002060">brick2/<p></p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:8.0pt;font-family:"Courier New";color:black">Number of entries: 0<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="color:#002060">Gluster LOG ( there are a few errors but I’m not sure how to decipher them)<p></p></span></p>
<p class="MsoNormal"><span style="color:#002060"><p> </p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:29:19.862809] I [MSGID: 106005] [glusterd-handler.c:4142:__glusterd_brick_rpc_notify] 0-management: Brick Server1:/exp/br02/brick2 has disconnected from
glusterd.<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:29:19.862836] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/7565ec897c6454bd3e2f4800250a7221.socket failed (Invalid argument)<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:29:19.862853] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: nfs has disconnected from glusterd.<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:29:19.953762] I [glusterd-pmap.c:227:pmap_registry_bind] 0-pmap: adding brick /exp/br02/brick2 on port 49153<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:31:12.977450] I [glusterd-replace-brick.c:99:__glusterd_handle_replace_brick] 0-management: Received replace brick req<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:31:12.977495] I [glusterd-replace-brick.c:154:__glusterd_handle_replace_brick] 0-management: Received replace brick status request<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:31:13.048852] I [glusterd-replace-brick.c:1412:rb_update_srcbrick_port] 0-: adding src-brick port no<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:31:19.588380] I [glusterd-replace-brick.c:99:__glusterd_handle_replace_brick] 0-management: Received replace brick req<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:31:19.588422] I [glusterd-replace-brick.c:154:__glusterd_handle_replace_brick] 0-management: Received replace brick status request<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:31:19.661101] I [glusterd-replace-brick.c:1412:rb_update_srcbrick_port] 0-: adding src-brick port no<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:31:45.115355] W [glusterd-op-sm.c:4021:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:31:45.118597] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Storage1<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:10.956357] I [glusterd-replace-brick.c:99:__glusterd_handle_replace_brick] 0-management: Received replace brick req<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:10.956385] I [glusterd-replace-brick.c:154:__glusterd_handle_replace_brick] 0-management: Received replace brick commit request<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:11.028472] I [glusterd-replace-brick.c:1412:rb_update_srcbrick_port] 0-: adding src-brick port no<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:12.122552] I [glusterd-utils.c:6276:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV3 successfully<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:12.131836] I [glusterd-utils.c:6281:glusterd_nfs_pmap_deregister] 0-: De-registered MOUNTV1 successfully<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:12.141107] I [glusterd-utils.c:6286:glusterd_nfs_pmap_deregister] 0-: De-registered NFSV3 successfully<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:12.150375] I [glusterd-utils.c:6291:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v4 successfully<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:12.159630] I [glusterd-utils.c:6296:glusterd_nfs_pmap_deregister] 0-: De-registered NLM v1 successfully<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:12.168889] I [glusterd-utils.c:6301:glusterd_nfs_pmap_deregister] 0-: De-registered ACL v3 successfully<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.254689] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.254799] W [socket.c:2992:socket_connect] 0-management: Ignore failed connection attempt on , (No such file or directory)<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.257790] I [rpc-clnt.c:969:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.257908] W [socket.c:2992:socket_connect] 0-management: Ignore failed connection attempt on , (No such file or directory)<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.258031] W [socket.c:611:__socket_rwv] 0-socket.management: writev on 127.0.0.1:1019 failed (Broken pipe)<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.258111] W [socket.c:611:__socket_rwv] 0-socket.management: writev on 127.0.0.1:1021 failed (Broken pipe)<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.258130] W [socket.c:611:__socket_rwv] 0-socket.management: writev on 10.45.16.17:1018 failed (Broken pipe)<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.711948] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=588 max=0 total=0<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.711967] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=124 max=0 total=0<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.712008] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=588 max=0 total=0<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.712021] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=124 max=0 total=0<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.731311] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=588 max=0 total=0<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.731326] I [mem-pool.c:545:mem_pool_destroy] 0-management: size=124 max=0 total=0<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.731356] I [glusterd-pmap.c:271:pmap_registry_remove] 0-pmap: removing brick /exp/br02/brick2 on port 49153<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.823129] I [socket.c:2344:socket_event_handler] 0-transport: disconnecting now<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.840668] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/7565ec897c6454bd3e2f4800250a7221.socket failed (Invalid argument)<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.840693] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: nfs has disconnected from glusterd.<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.840712] W [socket.c:611:__socket_rwv] 0-management: readv on /var/run/ac4c043d3c6a2e5159c86e8c75c51829.socket failed (Invalid argument)<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:13.840728] I [MSGID: 106006] [glusterd-handler.c:4257:__glusterd_nodesvc_rpc_notify] 0-management: glustershd has disconnected from glusterd.<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:14.729667] E [glusterd-rpc-ops.c:1169:__glusterd_commit_op_cbk] 0-management: Received commit RJT from uuid: 294aa603-ec24-44b9-864b-0fe743faa8d9<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:14.743623] E [glusterd-rpc-ops.c:1169:__glusterd_commit_op_cbk] 0-management: Received commit RJT from uuid: 92aabaf4-4b6c-48da-82b6-c465aff2ec6d<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:18.762975] W [glusterd-op-sm.c:4021:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:18.764552] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Storage1<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:18.769051] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not
match. Not aggregating tasks status.<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:18.769070] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from node/brick<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:18.771095] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not
match. Not aggregating tasks status.<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:18.771108] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from node/brick<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:48.570796] W [glusterd-op-sm.c:4021:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:48.572352] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Storage1<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:48.576899] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not
match. Not aggregating tasks status.<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:48.576918] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from node/brick<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:48.578982] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not
match. Not aggregating tasks status.<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:32:48.579001] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from node/brick<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:36:57.840738] W [glusterd-op-sm.c:4021:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:36:57.842370] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Storage1<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:36:57.846919] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not
match. Not aggregating tasks status.<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:36:57.846941] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from node/brick<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:36:57.849026] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not
match. Not aggregating tasks status.<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:36:57.849046] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from node/brick<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:37:20.208081] W [glusterd-op-sm.c:4021:glusterd_op_modify_op_ctx] 0-management: op_ctx modification failed<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:37:20.211279] I [glusterd-handler.c:3803:__glusterd_handle_status_volume] 0-management: Received status volume req for volume Storage1<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:37:20.215792] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not
match. Not aggregating tasks status.<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:37:20.215809] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from node/brick<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:37:20.216295] E [glusterd-utils.c:9955:glusterd_volume_status_aggregate_tasks_status] 0-management: Local tasks count (0) and remote tasks count (1) do not
match. Not aggregating tasks status.<p></p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Courier New";color:black">[2015-02-14 06:37:20.216308] E [glusterd-syncop.c:961:_gd_syncop_commit_op_cbk] 0-management: Failed to aggregate response from node/brick<p></p></span></p>
</div>
<p style="margin-top: 2.5em; margin-bottom: 1em; border-bottom: 1px solid #000"></p><pre class="k9mail"><hr /><br />Gluster-users mailing list<br />Gluster-users@gluster.org<br /><a href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre></blockquote></div><br>
-- <br>
Sent from my Android device with K-9 Mail. Please excuse my brevity.</body></html>