<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1671519470;
mso-list-type:hybrid;
mso-list-template-ids:-1259725934 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have a ~50 node cluster. I configured gluster so that there are 2 volumes: One is configured on top of HDD, and the other one is configured on top of RAM.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@nmIDPP20 ~]# gluster volume info<o:p></o:p></p>
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoNormal">Volume Name: ram<o:p></o:p></p>
<p class="MsoNormal">Type: Distributed-Replicate<o:p></o:p></p>
<p class="MsoNormal">Volume ID: a97fa262-276b-41e9-8f59-40f28451f689<o:p></o:p></p>
<p class="MsoNormal">Status: Started<o:p></o:p></p>
<p class="MsoNormal">Number of Bricks: 5 x 2 = 10<o:p></o:p></p>
<p class="MsoNormal">Transport-type: tcp<o:p></o:p></p>
<p class="MsoNormal">Bricks:<o:p></o:p></p>
<p class="MsoNormal">Brick1: 10.238.0.15:/mnt/ram/data<o:p></o:p></p>
<p class="MsoNormal">Brick2: 10.238.0.16:/mnt/ram/data<o:p></o:p></p>
<p class="MsoNormal">Brick3: 10.238.0.17:/mnt/ram/data<o:p></o:p></p>
<p class="MsoNormal">Brick4: 10.238.0.20:/mnt/ram/data<o:p></o:p></p>
<p class="MsoNormal">Brick5: 10.238.0.19:/mnt/ram/data<o:p></o:p></p>
<p class="MsoNormal">Brick6: 10.238.0.28:/mnt/ram/data<o:p></o:p></p>
<p class="MsoNormal">Brick7: 10.238.0.27:/mnt/ram/data<o:p></o:p></p>
<p class="MsoNormal">Brick8: 10.238.0.21:/mnt/ram/data<o:p></o:p></p>
<p class="MsoNormal">Brick9: 10.238.0.24:/mnt/ram/data<o:p></o:p></p>
<p class="MsoNormal">Brick10: 10.238.0.26:/mnt/ram/data<o:p></o:p></p>
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoNormal">Volume Name: disk<o:p></o:p></p>
<p class="MsoNormal">Type: Replicate<o:p></o:p></p>
<p class="MsoNormal">Volume ID: 9607ae5f-0dbf-4164-b260-5d9ce26d4fc7<o:p></o:p></p>
<p class="MsoNormal">Status: Started<o:p></o:p></p>
<p class="MsoNormal">Number of Bricks: 1 x 3 = 3<o:p></o:p></p>
<p class="MsoNormal">Transport-type: tcp<o:p></o:p></p>
<p class="MsoNormal">Bricks:<o:p></o:p></p>
<p class="MsoNormal">Brick1: 10.238.0.18:/var/cache/gluster/data/options/pp/data<o:p></o:p></p>
<p class="MsoNormal">Brick2: 10.238.0.16:/var/cache/gluster/data/options/pp/data<o:p></o:p></p>
<p class="MsoNormal">Brick3: 10.238.0.17:/var/cache/gluster/data/options/pp/data<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>I’ve bare metaled one of the servers: 10.238.0.22. And, now trying to add it to the pool. So, after
<u>gluster peer probe 10.238.0.22</u> command, we can see that it’s in pool:<o:p></o:p></b></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@nmIDPP20 ~]# gluster pool list<o:p></o:p></p>
<p class="MsoNormal">UUID Hostname State<o:p></o:p></p>
<p class="MsoNormal">baa648a5-ff35-44e0-80ea-a55e43154d12 10.238.0.50 Connected
<o:p></o:p></p>
<p class="MsoNormal">20bb470a-85da-4e3a-a66b-08a935c189ae 10.238.0.26 Connected
<o:p></o:p></p>
<p class="MsoNormal">79dffcf8-8c3a-47b5-926a-39be2c1406da 10.238.0.13 Disconnected
<o:p></o:p></p>
<p class="MsoNormal">7212e375-76a4-46c9-8bac-7470e2e5a910 10.238.0.17 Connected
<o:p></o:p></p>
<p class="MsoNormal">c6080a14-33d7-4012-8940-2d9232752551 10.238.0.14 Connected
<o:p></o:p></p>
<p class="MsoNormal">b553ed3c-21f1-4110-808d-4b08e6ded200 10.238.0.28 Connected
<o:p></o:p></p>
<p class="MsoNormal">5e596931-9151-4f5b-bc57-feb6fe46054f 10.238.0.7 Connected
<o:p></o:p></p>
<p class="MsoNormal">8e1128ed-df07-4747-812e-dcc280fce5c1 10.238.0.16 Connected
<o:p></o:p></p>
<p class="MsoNormal">0b5fae30-e169-42ee-8f39-678d6fc93ac2 10.238.0.19 Connected
<o:p></o:p></p>
<p class="MsoNormal">0f82df55-3994-4561-8a0a-1c1d2e9c3cff 10.238.0.29 Connected
<o:p></o:p></p>
<p class="MsoNormal">446ea1e4-61b9-4881-9073-6aeb9a154710 10.238.0.24 Connected
<o:p></o:p></p>
<p class="MsoNormal">bcf84149-415b-4eb7-8dc1-2b284e135307 10.238.0.27 Connected
<o:p></o:p></p>
<p class="MsoNormal">97dddf9f-0b57-4bb8-86fd-196cb51df4b6 10.238.0.20 Connected
<o:p></o:p></p>
<p class="MsoNormal">b2bf8b3c-890b-423b-b901-f16f1186c3e6 10.238.0.4 Connected
<o:p></o:p></p>
<p class="MsoNormal">878ba732-0fea-4734-b1bc-a08ad7a2c97a 10.238.0.9 Connected
<o:p></o:p></p>
<p class="MsoNormal">51750fb0-c182-4e76-821f-16cee23fdf27 10.238.0.6 Connected
<o:p></o:p></p>
<p class="MsoNormal">b162e108-4301-47df-875f-92151244b694 10.238.0.8 Connected
<o:p></o:p></p>
<p class="MsoNormal">25d29db8-0916-4ef4-80d1-34fbf8aa5d26 10.238.0.21 Connected
<o:p></o:p></p>
<p class="MsoNormal">9acfb879-7df9-4c87-aa1c-eb518b9c668d 10.238.0.12 Connected
<o:p></o:p></p>
<p class="MsoNormal">aacd1fa1-940c-4cec-9b04-1fb49348e764 10.238.0.49 Connected
<o:p></o:p></p>
<p class="MsoNormal">5c36b282-9842-4b85-8d0f-e5101817dfe1 10.238.0.18 Connected
<o:p></o:p></p>
<p class="MsoNormal">a5298a13-144d-46e1-856f-91ade6649840 10.238.0.10 Connected
<o:p></o:p></p>
<p class="MsoNormal">4e7b83bd-367e-419d-aa5b-34947021dbc3 10.238.0.48 Connected
<o:p></o:p></p>
<p class="MsoNormal">6aa7957f-be6f-4bee-a748-32937d3ababd 10.238.0.47 Connected
<o:p></o:p></p>
<p class="MsoNormal">3890ac7d-7959-4565-86de-fc792cc357b0 10.238.0.45 Disconnected
<o:p></o:p></p>
<p class="MsoNormal">4814a743-5b52-44ab-b169-e907082aa229 10.238.0.32 Connected
<o:p></o:p></p>
<p class="MsoNormal">cf735cd8-75e3-413b-88c5-46e5b79f7558 10.238.0.42 Connected
<o:p></o:p></p>
<p class="MsoNormal">b1fa7e22-2e1b-4d07-966e-3096e58e5c78 10.238.0.39 Connected
<o:p></o:p></p>
<p class="MsoNormal">1459fce8-110c-478f-815e-89507225226e 10.238.0.34 Connected
<o:p></o:p></p>
<p class="MsoNormal">a7b21ee9-970b-4d99-9f8f-b7e1cbf4be77 10.238.0.25 Connected
<o:p></o:p></p>
<p class="MsoNormal">dab1a271-4244-41bc-b770-7b13bd6e399d 10.238.0.43 Connected
<o:p></o:p></p>
<p class="MsoNormal">5b483c65-0d04-4188-85a9-77dfbbef78cd 10.238.0.41 Connected
<o:p></o:p></p>
<p class="MsoNormal">1b8cb9d8-ce8f-49aa-b958-705dd09db073 10.238.0.40 Connected
<o:p></o:p></p>
<p class="MsoNormal">4b4f85a0-1310-45df-a613-e33c967cc53d 10.238.0.38 Connected
<o:p></o:p></p>
<p class="MsoNormal">dab043b8-11ba-4fa6-9b82-baa18b41167d 10.238.0.33 Disconnected
<o:p></o:p></p>
<p class="MsoNormal">06cbc4c2-9d79-4689-9ac6-3dbc2250d903 10.238.0.30 Connected
<o:p></o:p></p>
<p class="MsoNormal">f33451c7-e984-495c-8e34-0b2d99a21e1e 10.238.0.31 Connected
<o:p></o:p></p>
<p class="MsoNormal">1873e2ce-1239-4b6d-930f-af14e9c1f13b 10.238.0.5 Connected
<o:p></o:p></p>
<p class="MsoNormal">c85de12f-23e6-4797-adb4-d33b7b4eb5fc 10.238.0.11 Connected
<o:p></o:p></p>
<p class="MsoNormal">4147639d-652e-49a8-aa8b-d77327cca9ca 10.238.0.15 Connected
<o:p></o:p></p>
<p class="MsoNormal"><span style="background:yellow;mso-highlight:yellow">07580a32-c558-449d-b454-044fb679c908 10.238.0.22 Connected</span>
<o:p></o:p></p>
<p class="MsoNormal">d5140e78-498d-4c63-868d-189554aef7d4 localhost Connected<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">But, <u>gluster peer status</u> is giving following output:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@nmIDPP20 ~]# gluster peer status<o:p></o:p></p>
<p class="MsoNormal">Number of Peers: 41<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.50<o:p></o:p></p>
<p class="MsoNormal">Uuid: baa648a5-ff35-44e0-80ea-a55e43154d12<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.26<o:p></o:p></p>
<p class="MsoNormal">Uuid: 20bb470a-85da-4e3a-a66b-08a935c189ae<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.13<o:p></o:p></p>
<p class="MsoNormal">Uuid: 79dffcf8-8c3a-47b5-926a-39be2c1406da<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Disconnected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.17<o:p></o:p></p>
<p class="MsoNormal">Uuid: 7212e375-76a4-46c9-8bac-7470e2e5a910<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.14<o:p></o:p></p>
<p class="MsoNormal">Uuid: c6080a14-33d7-4012-8940-2d9232752551<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.28<o:p></o:p></p>
<p class="MsoNormal">Uuid: b553ed3c-21f1-4110-808d-4b08e6ded200<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.7<o:p></o:p></p>
<p class="MsoNormal">Uuid: 5e596931-9151-4f5b-bc57-feb6fe46054f<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.16<o:p></o:p></p>
<p class="MsoNormal">Uuid: 8e1128ed-df07-4747-812e-dcc280fce5c1<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.19<o:p></o:p></p>
<p class="MsoNormal">Uuid: 0b5fae30-e169-42ee-8f39-678d6fc93ac2<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.29<o:p></o:p></p>
<p class="MsoNormal">Uuid: 0f82df55-3994-4561-8a0a-1c1d2e9c3cff<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.24<o:p></o:p></p>
<p class="MsoNormal">Uuid: 446ea1e4-61b9-4881-9073-6aeb9a154710<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.27<o:p></o:p></p>
<p class="MsoNormal">Uuid: bcf84149-415b-4eb7-8dc1-2b284e135307<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.20<o:p></o:p></p>
<p class="MsoNormal">Uuid: 97dddf9f-0b57-4bb8-86fd-196cb51df4b6<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.4<o:p></o:p></p>
<p class="MsoNormal">Uuid: b2bf8b3c-890b-423b-b901-f16f1186c3e6<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.9<o:p></o:p></p>
<p class="MsoNormal">Uuid: 878ba732-0fea-4734-b1bc-a08ad7a2c97a<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.6<o:p></o:p></p>
<p class="MsoNormal">Uuid: 51750fb0-c182-4e76-821f-16cee23fdf27<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.8<o:p></o:p></p>
<p class="MsoNormal">Uuid: b162e108-4301-47df-875f-92151244b694<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.21<o:p></o:p></p>
<p class="MsoNormal">Uuid: 25d29db8-0916-4ef4-80d1-34fbf8aa5d26<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.12<o:p></o:p></p>
<p class="MsoNormal">Uuid: 9acfb879-7df9-4c87-aa1c-eb518b9c668d<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.49<o:p></o:p></p>
<p class="MsoNormal">Uuid: aacd1fa1-940c-4cec-9b04-1fb49348e764<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.18<o:p></o:p></p>
<p class="MsoNormal">Uuid: 5c36b282-9842-4b85-8d0f-e5101817dfe1<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.10<o:p></o:p></p>
<p class="MsoNormal">Uuid: a5298a13-144d-46e1-856f-91ade6649840<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.48<o:p></o:p></p>
<p class="MsoNormal">Uuid: 4e7b83bd-367e-419d-aa5b-34947021dbc3<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.47<o:p></o:p></p>
<p class="MsoNormal">Uuid: 6aa7957f-be6f-4bee-a748-32937d3ababd<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.45<o:p></o:p></p>
<p class="MsoNormal">Uuid: 3890ac7d-7959-4565-86de-fc792cc357b0<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Disconnected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.32<o:p></o:p></p>
<p class="MsoNormal">Uuid: 4814a743-5b52-44ab-b169-e907082aa229<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.42<o:p></o:p></p>
<p class="MsoNormal">Uuid: cf735cd8-75e3-413b-88c5-46e5b79f7558<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.39<o:p></o:p></p>
<p class="MsoNormal">Uuid: b1fa7e22-2e1b-4d07-966e-3096e58e5c78<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.34<o:p></o:p></p>
<p class="MsoNormal">Uuid: 1459fce8-110c-478f-815e-89507225226e<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.25<o:p></o:p></p>
<p class="MsoNormal">Uuid: a7b21ee9-970b-4d99-9f8f-b7e1cbf4be77<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.43<o:p></o:p></p>
<p class="MsoNormal">Uuid: dab1a271-4244-41bc-b770-7b13bd6e399d<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.41<o:p></o:p></p>
<p class="MsoNormal">Uuid: 5b483c65-0d04-4188-85a9-77dfbbef78cd<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.40<o:p></o:p></p>
<p class="MsoNormal">Uuid: 1b8cb9d8-ce8f-49aa-b958-705dd09db073<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.38<o:p></o:p></p>
<p class="MsoNormal">Uuid: 4b4f85a0-1310-45df-a613-e33c967cc53d<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.33<o:p></o:p></p>
<p class="MsoNormal">Uuid: dab043b8-11ba-4fa6-9b82-baa18b41167d<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Disconnected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.30<o:p></o:p></p>
<p class="MsoNormal">Uuid: 06cbc4c2-9d79-4689-9ac6-3dbc2250d903<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.31<o:p></o:p></p>
<p class="MsoNormal">Uuid: f33451c7-e984-495c-8e34-0b2d99a21e1e<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.5<o:p></o:p></p>
<p class="MsoNormal">Uuid: 1873e2ce-1239-4b6d-930f-af14e9c1f13b<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.11<o:p></o:p></p>
<p class="MsoNormal">Uuid: c85de12f-23e6-4797-adb4-d33b7b4eb5fc<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hostname: 10.238.0.15<o:p></o:p></p>
<p class="MsoNormal">Uuid: 4147639d-652e-49a8-aa8b-d77327cca9ca<o:p></o:p></p>
<p class="MsoNormal">State: Peer in Cluster (Connected)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="background:yellow;mso-highlight:yellow">Hostname: 10.238.0.22<o:p></o:p></span></p>
<p class="MsoNormal"><span style="background:yellow;mso-highlight:yellow">Uuid: 07580a32-c558-449d-b454-044fb679c908<o:p></o:p></span></p>
<p class="MsoNormal"><span style="background:yellow;mso-highlight:yellow">State: Probe Sent to Peer (Connected)</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">And after staying in this state for about 10 min, .22 node disappears from pool list. Also, during peer probe, on node .22 if you do
<u>gluster pool list</u> , it hangs and does not do anything. Only, after few mins it releases the shell, and outputs nothing.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I’ve tried to do couple of things to resolve the issue:<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Disabled firewall -> didn’t help<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Removed mgmt directory from 22, restarted gluster service and glusterfs/d processes -> didn’t help<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">3.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Tried to probe .22 from another server -> didn’t help<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="mso-list:Ignore">4.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Reset uuid of .22 -> didn’t help<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I don’t know what I can do more, so asking for support from you.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Following are logs during probe from .22 and .23:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">10.238.0.22:/<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[2016-05-06 19:45:24.463346] I [glusterd-handler.c:1114:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:46:01.295054] I [glusterd-handler.c:1114:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:46:50.518018] I [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30501<o:p></o:p></p>
<p class="MsoNormal"><span style="background:yellow;mso-highlight:yellow">[2016-05-06 19:46:50.521829] I [glusterd-handler.c:2346:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: d5140e78-498d-4c63-868d-189554aef7d4<o:p></o:p></span></p>
<p class="MsoNormal"><span style="background:yellow;mso-highlight:yellow">[2016-05-06 19:47:10.542419] I [glusterd-handler.c:2374:__glusterd_handle_probe_query] 0-glusterd: Unable to find peerinfo for host: 10.238.0.23 (24007)</span><o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:47:10.548116] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:47:10.548218] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:47:10.548239] I [socket.c:3576:socket_init] 0-management: using system polling thread<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:47:10.553769] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:47:10.553886] I [glusterd-handler.c:2398:__glusterd_handle_probe_query] 0-glusterd: Responded to 10.238.0.23, op_ret: 0, op_errno: 0, ret: 0<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:47:10.554650] I [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: d5140e78-498d-4c63-868d-189554aef7d4<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:50:50.812036] E [glusterd-utils.c:4692:glusterd_brick_start] 0-management: Could not find peer on which brick 10.238.0.15:/mnt/ram/data resides<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">10.238.0.23:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[2016-05-06 19:46:28.982091] I [glusterd-handler.c:1114:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:46:31.930017] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:46:34.934960] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:46:37.916015] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:46:40.947036] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:46:43.950373] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:46:46.961104] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:46:49.966875] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal"><span style="background:yellow;mso-highlight:yellow">[2016-05-06 19:46:50.497510] I [glusterd-handler.c:918:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req 10.238.0.22 24007<o:p></o:p></span></p>
<p class="MsoNormal">[2016-05-06 19:46:50.502555] I [glusterd-handler.c:2931:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: 10.238.0.22 (24007)<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:46:50.511183] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:46:50.511279] I [socket.c:3561:socket_init] 0-management: SSL support is NOT enabled<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:46:50.511300] I [socket.c:3576:socket_init] 0-management: using system polling thread<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:46:50.517005] I [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect returned 0<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:46:52.983838] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:46:55.975533] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:46:58.989536] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:47:01.994423] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:47:04.995025] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:47:07.995849] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal"><span style="background:yellow;mso-highlight:yellow">[2016-05-06 19:47:10.553738] I [glusterd-rpc-ops.c:234:__glusterd_probe_cbk] 0-glusterd: Received probe resp from uuid: 07580a32-c558-449d-b454-044fb679c908, host: 10.238.0.22<o:p></o:p></span></p>
<p class="MsoNormal"><span style="background:yellow;mso-highlight:yellow">[2016-05-06 19:47:10.559641] I [glusterd-rpc-ops.c:306:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req<o:p></o:p></span></p>
<p class="MsoNormal">[2016-05-06 19:47:11.006166] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:47:14.009705] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:47:16.996479] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:47:20.024705] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:47:23.035546] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal">[2016-05-06 19:47:26.041132] E [glusterd-handshake.c:942:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Failed to get handshake ack from remote server<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Please, advice anything to handle this issue.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">Azamat<o:p></o:p></p>
<p class="MsoNormal">Phone: 703-667-8922<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<br><span style="font-size: 8pt; font-family: Arial, sans-serif; color: #003366;">_____________________________________________________<br>This electronic message and any files transmitted with it contains<br>information from iDirect, which may be privileged, proprietary<br>and/or confidential. It is intended solely for the use of the individual<br>or entity to whom they are addressed. If you are not the original<br>recipient or the person responsible for delivering the email to the<br>intended recipient, be advised that you have received this email<br>in error, and that any use, dissemination, forwarding, printing, or<br>copying of this email is strictly prohibited. If you received this email<br>in error, please delete it and immediately notify the sender.<br>_____________________________________________________</span></body>
</html>