<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<meta http-equiv="content-type" content="text/html;
charset=windows-1252">
<blockquote type="cite">
<meta http-equiv="content-type" content="text/html;
charset=windows-1252">
<span style="color: rgb(51, 51, 51); font-family: Verdana, Arial,
Tahoma, Calibri, Geneva, sans-serif; font-size: 13px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: auto;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; display: inline !important;
float: none; background-color: rgb(250, 250, 250);">A-1) shut
down node #1 (the first that is about to be upgraded)</span><br
style="color: rgb(51, 51, 51); font-family: Verdana, Arial,
Tahoma, Calibri, Geneva, sans-serif; font-size: 13px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: auto;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(250, 250,
250);">
<span style="color: rgb(51, 51, 51); font-family: Verdana, Arial,
Tahoma, Calibri, Geneva, sans-serif; font-size: 13px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: auto;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; display: inline !important;
float: none; background-color: rgb(250, 250, 250);">A-2) remove
node #1 from the Proxmox cluster (</span><i style="color:
rgb(51, 51, 51); font-family: Verdana, Arial, Tahoma, Calibri,
Geneva, sans-serif; font-size: 13px; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: left; text-indent: 0px;
text-transform: none; white-space: normal; widows: 1;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(250, 250, 250);">pvevm delnode "metal1"</i><span
style="color: rgb(51, 51, 51); font-family: Verdana, Arial,
Tahoma, Calibri, Geneva, sans-serif; font-size: 13px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: auto;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; display: inline !important;
float: none; background-color: rgb(250, 250, 250);">)</span><br
style="color: rgb(51, 51, 51); font-family: Verdana, Arial,
Tahoma, Calibri, Geneva, sans-serif; font-size: 13px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: auto;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(250, 250,
250);">
<span style="color: rgb(51, 51, 51); font-family: Verdana, Arial,
Tahoma, Calibri, Geneva, sans-serif; font-size: 13px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: auto;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; display: inline !important;
float: none; background-color: rgb(250, 250, 250);">A-3) remove
node #1 from the Gluster volume/cluster (</span><i style="color:
rgb(51, 51, 51); font-family: Verdana, Arial, Tahoma, Calibri,
Geneva, sans-serif; font-size: 13px; font-variant: normal;
font-weight: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: left; text-indent: 0px;
text-transform: none; white-space: normal; widows: 1;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(250, 250, 250);">gluster volume
remove-brick ... && gluster peer detach "metal1"</i><span
style="color: rgb(51, 51, 51); font-family: Verdana, Arial,
Tahoma, Calibri, Geneva, sans-serif; font-size: 13px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: auto;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; display: inline !important;
float: none; background-color: rgb(250, 250, 250);">)</span><br
style="color: rgb(51, 51, 51); font-family: Verdana, Arial,
Tahoma, Calibri, Geneva, sans-serif; font-size: 13px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: auto;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(250, 250,
250);">
<span style="color: rgb(51, 51, 51); font-family: Verdana, Arial,
Tahoma, Calibri, Geneva, sans-serif; font-size: 13px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: auto;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; display: inline !important;
float: none; background-color: rgb(250, 250, 250);">A-4) install
Debian Jessie on node #1, overwriting all data on the HDD -<span
class="Apple-converted-space"> </span></span><b style="color:
rgb(51, 51, 51); font-family: Verdana, Arial, Tahoma, Calibri,
Geneva, sans-serif; font-size: 13px; font-style: normal;
font-variant: normal; letter-spacing: normal; line-height:
normal; orphans: auto; text-align: left; text-indent: 0px;
text-transform: none; white-space: normal; widows: 1;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(250, 250, 250);">with same Network
settings and hostname as before</b><br style="color: rgb(51, 51,
51); font-family: Verdana, Arial, Tahoma, Calibri, Geneva,
sans-serif; font-size: 13px; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing: normal;
line-height: normal; orphans: auto; text-align: left;
text-indent: 0px; text-transform: none; white-space: normal;
widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(250, 250, 250);">
<span style="color: rgb(51, 51, 51); font-family: Verdana, Arial,
Tahoma, Calibri, Geneva, sans-serif; font-size: 13px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: auto;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; display: inline !important;
float: none; background-color: rgb(250, 250, 250);">A-5)<span
class="Apple-converted-space"> </span></span><a
href="https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie"
target="_blank" style="color: rgb(65, 115, 148);
text-decoration: none; font-family: Verdana, Arial, Tahoma,
Calibri, Geneva, sans-serif; font-size: 13px; font-style:
normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: auto;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(250, 250,
250);">install Proxmox 4.0</a><span style="color: rgb(51, 51,
51); font-family: Verdana, Arial, Tahoma, Calibri, Geneva,
sans-serif; font-size: 13px; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing: normal;
line-height: normal; orphans: auto; text-align: left;
text-indent: 0px; text-transform: none; white-space: normal;
widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px;
display: inline !important; float: none; background-color:
rgb(250, 250, 250);"><span class="Apple-converted-space"> </span>on
node #1</span><br style="color: rgb(51, 51, 51); font-family:
Verdana, Arial, Tahoma, Calibri, Geneva, sans-serif; font-size:
13px; font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal; orphans:
auto; text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(250, 250,
250);">
<span style="color: rgb(51, 51, 51); font-family: Verdana, Arial,
Tahoma, Calibri, Geneva, sans-serif; font-size: 13px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: auto;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; display: inline !important;
float: none; background-color: rgb(250, 250, 250);">A-6) install
Gluster on node #1 and add it back to the Gluster volume (</span><i
style="color: rgb(51, 51, 51); font-family: Verdana, Arial,
Tahoma, Calibri, Geneva, sans-serif; font-size: 13px;
font-variant: normal; font-weight: normal; letter-spacing:
normal; line-height: normal; orphans: auto; text-align: left;
text-indent: 0px; text-transform: none; white-space: normal;
widows: 1; word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(250, 250, 250);">gluster volume add-brick
...</i><span style="color: rgb(51, 51, 51); font-family:
Verdana, Arial, Tahoma, Calibri, Geneva, sans-serif; font-size:
13px; font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal; orphans:
auto; text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; display: inline !important;
float: none; background-color: rgb(250, 250, 250);">) =>
shared storage will be complete again (spanning 3.4 and 4.0
nodes)</span><br style="color: rgb(51, 51, 51); font-family:
Verdana, Arial, Tahoma, Calibri, Geneva, sans-serif; font-size:
13px; font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal; orphans:
auto; text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(250, 250,
250);">
<span style="color: rgb(51, 51, 51); font-family: Verdana, Arial,
Tahoma, Calibri, Geneva, sans-serif; font-size: 13px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: auto;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; display: inline !important;
float: none; background-color: rgb(250, 250, 250);">A-7)
configure the Gluster volume as shared storage in Proxmox 4
(node #1)</span><br style="color: rgb(51, 51, 51); font-family:
Verdana, Arial, Tahoma, Calibri, Geneva, sans-serif; font-size:
13px; font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal; orphans:
auto; text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(250, 250,
250);">
<span style="color: rgb(51, 51, 51); font-family: Verdana, Arial,
Tahoma, Calibri, Geneva, sans-serif; font-size: 13px;
font-style: normal; font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: auto;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 1; word-spacing: 0px;
-webkit-text-stroke-width: 0px; display: inline !important;
float: none; background-color: rgb(250, 250, 250);">A-8)
configure the external Backup storage on node #1 (Proxmox 4)</span></blockquote>
<br>
Was the data on the gluster brick deleted as part of step 4? When
you remove the brick, gluster will no longer track pending changes
for that brick. If you add it back in with stale data but matching
gfids, you would have two clean bricks with mismatching data. Did
you have to use "add-brick...force"? <br>
<br>
<br>
<div class="moz-cite-prefix">On 12/09/2015 06:53 AM, Udo Giacomozzi
wrote:<br>
</div>
<blockquote cite="mid:56684057.303@indunet.it" type="cite">
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
<div class="moz-cite-prefix">Am 09.12.2015 um 14:39 schrieb
Lindsay Mathieson:<br>
</div>
<blockquote cite="mid:56682EFC.5070708@gmail.com" type="cite"><br>
Udo, it occurs to me that if your VM's were running on #2 &
#3 and you live migrated them to #1 prior to rebooting #2/3,
then you would indeed rapidly get progressive VM corruption.<br>
<br>
However it wouldn't be due to the heal process, but rather the
live migration with "performance.stat-prefetch" on. This always
leads to qcow2 files becoming corrupted and unusable.</blockquote>
<br>
Nope. All VMs were running on #1, no exception.<br>
Nodes #2 and #3 never had a VM running on them, so they were
pratically idle since their installation.<br>
<br>
Basically I set up node #1, including all VMs. <br>
Then I've installed nodes #2 and #3, configured Proxmox and
Gluster cluster and then waited quite some time until Gluster had
synced up nodes #2 and #3 (healing).<br>
From then on, I've rebooted nodes 2 & 3, but in theory these
nodes never had to do any writes to the Gluster volume at all.<br>
<br>
If you're interested, you can read about my upgrade strategy in
this Proxmox forum post:
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://forum.proxmox.com/threads/24990-Upgrade-3-4-HA-cluster-to-4-0-via-reinstallation-with-minimal-downtime?p=125040#post125040">http://forum.proxmox.com/threads/24990-Upgrade-3-4-HA-cluster-to-4-0-via-reinstallation-with-minimal-downtime?p=125040#post125040</a><br>
<br>
Also, It seems rather strange to me that pratically all ~15 VMs
(!) suffered from data corruption. It's like if Gluster considered
node #2 or #3 to be ahead and it "healed" in the wrong direction.
I don't know..<br>
<br>
BTW, once I understood what was going on, <i>with the problematic
"healing" still in progress</i>, I was able to overwrite the bad
images (still active on #1) by using standard Proxmox
backup-restore and Gluster handled it correctly. <br>
<br>
<br>
Anway, I really love the simplicity of Gluster (setting up and
maintaining a cluster is extremely easy), but these healing issues
are causing some headache to me... ;-)<br>
<br>
Udo<br>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>