<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">On 12/08/2016 06:53 PM, Atin Mukherjee
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAGNCGH1vqHMP=+2ZgR66fVCTwMMYbpk0C2SEr0pLdjZszGSsCw@mail.gmail.com"
      type="cite">
      <div dir="ltr"><br>
        <div class="gmail_extra"><br>
          <div class="gmail_quote">On Thu, Dec 8, 2016 at 6:44 PM, Miloš
            Čučulović - MDPI <span dir="ltr">&lt;<a
                moz-do-not-send="true" href="mailto:cuculovic@mdpi.com"
                target="_blank">cuculovic@mdpi.com</a>&gt;</span> wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">Ah,
              damn! I found the issue. On the storage server, the
              storage2 IP address was wrong, I inversed two digits in
              the /etc/hosts file, sorry for that :(<br>
              <br>
              I was able to add the brick now, I started the heal, but
              still no data transfer visible.<br>
            </blockquote>
          </div>
        </div>
      </div>
    </blockquote>
    1. Are the files getting created on the new brick though?<br>
    2. Can you provide the output of `getfattr -d -m . -e hex
    /data/data-cluster` on both bricks?<br>
    3. Is it possible to attach gdb to the self-heal daemon on the
    original (old) brick and get a backtrace?<br>
      <tt>  `gdb -p &lt;pid of self-heal daemon on the orignal
      brick&gt;`</tt><tt><br>
    </tt><tt>     thread apply all bt</tt><tt>  --&gt;share this output<br>
          quit gdb.<br>
    </tt><br>
    <br>
    -Ravi<br>
    <blockquote
cite="mid:CAGNCGH1vqHMP=+2ZgR66fVCTwMMYbpk0C2SEr0pLdjZszGSsCw@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_extra">
          <div class="gmail_quote">
            <div><br>
            </div>
            <div>@Ravi/Pranith - can you help here?<br>
               <br>
            </div>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <br>
              By doing gluster volume status, I have<span class=""><br>
                <br>
                Status of volume: storage<br>
                Gluster process                       TCP Port  RDMA
                Port  Online  Pid<br>
                ------------------------------<wbr>------------------------------<wbr>------------------<br>
              </span>
              Brick storage2:/data/data-cluster     49152     0         
              Y       23101<br>
              Brick storage:/data/data-cluster      49152     0         
              Y       30773<br>
              Self-heal Daemon on localhost         N/A       N/A       
              Y       30050<br>
              Self-heal Daemon on storage           N/A       N/A       
              Y       30792<br>
              <br>
              <br>
              Any idea?<br>
              <br>
              On storage I have:<br>
              Number of Peers: 1<br>
              <br>
              Hostname: 195.65.194.217<br>
              Uuid: 7c988af2-9f76-4843-8e6f-d94866<wbr>d57bb0<br>
              State: Peer in Cluster (Connected)<span class=""><br>
                <br>
                <br>
                - Kindest regards,<br>
                <br>
                Milos Cuculovic<br>
                IT Manager<br>
                <br>
                ---<br>
                MDPI AG<br>
                Postfach, CH-4020 Basel, Switzerland<br>
                Office: St. Alban-Anlage 66, 4052 Basel, Switzerland<br>
                Tel. +41 61 683 77 35<br>
                Fax +41 61 302 89 18<br>
                Email: <a moz-do-not-send="true"
                  href="mailto:cuculovic@mdpi.com" target="_blank">cuculovic@mdpi.com</a><br>
                Skype: milos.cuculovic.mdpi<br>
                <br>
              </span><span class="">
                On 08.12.2016 13:55, Atin Mukherjee wrote:<br>
              </span>
              <blockquote class="gmail_quote" style="margin:0 0 0
                .8ex;border-left:1px #ccc solid;padding-left:1ex"><span
                  class="">
                  Can you resend the attachment as zip? I am unable to
                  extract the<br>
                  content? We shouldn't have 0 info file. What does
                  gluster peer status<br>
                  output say?<br>
                  <br>
                  On Thu, Dec 8, 2016 at 4:51 PM, Miloš Čučulović - MDPI<br>
                </span><span class="">
                  &lt;<a moz-do-not-send="true"
                    href="mailto:cuculovic@mdpi.com" target="_blank">cuculovic@mdpi.com</a>
                  &lt;mailto:<a moz-do-not-send="true"
                    href="mailto:cuculovic@mdpi.com" target="_blank">cuculovic@mdpi.com</a>&gt;&gt;
                  wrote:<br>
                  <br>
                      I hope you received my last email Atin, thank you!<br>
                  <br>
                      - Kindest regards,<br>
                  <br>
                      Milos Cuculovic<br>
                      IT Manager<br>
                  <br>
                      ---<br>
                      MDPI AG<br>
                      Postfach, CH-4020 Basel, Switzerland<br>
                      Office: St. Alban-Anlage 66, 4052 Basel,
                  Switzerland<br>
                      Tel. +41 61 683 77 35<br>
                      Fax +41 61 302 89 18<br>
                </span>
                    Email: <a moz-do-not-send="true"
                  href="mailto:cuculovic@mdpi.com" target="_blank">cuculovic@mdpi.com</a>
                &lt;mailto:<a moz-do-not-send="true"
                  href="mailto:cuculovic@mdpi.com" target="_blank">cuculovic@mdpi.com</a>&gt;<span
                  class=""><br>
                      Skype: milos.cuculovic.mdpi<br>
                  <br>
                      On 08.12.2016 10:28, Atin Mukherjee wrote:<br>
                  <br>
                  <br>
                          ---------- Forwarded message ----------<br>
                          From: *Atin Mukherjee* &lt;<a
                    moz-do-not-send="true"
                    href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a><br>
                </span><span class="">
                          &lt;mailto:<a moz-do-not-send="true"
                    href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>&gt;
                  &lt;mailto:<a moz-do-not-send="true"
                    href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a><br>
                          &lt;mailto:<a moz-do-not-send="true"
                    href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>&gt;&gt;&gt;<br>
                          Date: Thu, Dec 8, 2016 at 11:56 AM<br>
                          Subject: Re: [Gluster-users] Replica brick not
                  working<br>
                          To: Ravishankar N &lt;<a
                    moz-do-not-send="true"
                    href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a><br>
                </span><span class="">
                          &lt;mailto:<a moz-do-not-send="true"
                    href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a><wbr>&gt;
                  &lt;mailto:<a moz-do-not-send="true"
                    href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a><br>
                          &lt;mailto:<a moz-do-not-send="true"
                    href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a><wbr>&gt;&gt;&gt;<br>
                          Cc: Miloš Čučulović - MDPI &lt;<a
                    moz-do-not-send="true"
                    href="mailto:cuculovic@mdpi.com" target="_blank">cuculovic@mdpi.com</a><br>
                          &lt;mailto:<a moz-do-not-send="true"
                    href="mailto:cuculovic@mdpi.com" target="_blank">cuculovic@mdpi.com</a>&gt;<br>
                </span>
                        &lt;mailto:<a moz-do-not-send="true"
                  href="mailto:cuculovic@mdpi.com" target="_blank">cuculovic@mdpi.com</a>
                &lt;mailto:<a moz-do-not-send="true"
                  href="mailto:cuculovic@mdpi.com" target="_blank">cuculovic@mdpi.com</a>&gt;&gt;&gt;,<span
                  class=""><br>
                          Pranith Kumar Karampuri<br>
                          &lt;<a moz-do-not-send="true"
                    href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>
                  &lt;mailto:<a moz-do-not-send="true"
                    href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>&gt;<br>
                </span>
                        &lt;mailto:<a moz-do-not-send="true"
                  href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>
                &lt;mailto:<a moz-do-not-send="true"
                  href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>&gt;&gt;&gt;<wbr>,<br>
                        gluster-users<br>
                        &lt;<a moz-do-not-send="true"
                  href="mailto:gluster-users@gluster.org"
                  target="_blank">gluster-users@gluster.org</a>
                &lt;mailto:<a moz-do-not-send="true"
                  href="mailto:gluster-users@gluster.org"
                  target="_blank">gluster-users@gluster.<wbr>org</a>&gt;<br>
                        &lt;mailto:<a moz-do-not-send="true"
                  href="mailto:gluster-users@gluster.org"
                  target="_blank">gluster-users@gluster.<wbr>org</a><span
                  class=""><br>
                          &lt;mailto:<a moz-do-not-send="true"
                    href="mailto:gluster-users@gluster.org"
                    target="_blank">gluster-users@gluster.<wbr>org</a>&gt;&gt;&gt;<br>
                  <br>
                  <br>
                  <br>
                  <br>
                          On Thu, Dec 8, 2016 at 11:11 AM, Ravishankar N<br>
                          &lt;<a moz-do-not-send="true"
                    href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>
                  &lt;mailto:<a moz-do-not-send="true"
                    href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a><wbr>&gt;<br>
                </span>
                        &lt;mailto:<a moz-do-not-send="true"
                  href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>
                &lt;mailto:<a moz-do-not-send="true"
                  href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a><wbr>&gt;&gt;&gt;
                <div>
                  <div class="h5"><br>
                            wrote:<br>
                    <br>
                                On 12/08/2016 10:43 AM, Atin Mukherjee
                    wrote:<br>
                    <br>
                                    &gt;From the log snippet:<br>
                    <br>
                                    [2016-12-07 09:15:35.677645] I
                    [MSGID: 106482]<br>
                                    [glusterd-brick-ops.c:442:__gl<wbr>usterd_handle_add_brick]<br>
                                    0-management: Received add brick req<br>
                                    [2016-12-07 09:15:35.677708] I
                    [MSGID: 106062]<br>
                                    [glusterd-brick-ops.c:494:__gl<wbr>usterd_handle_add_brick]<br>
                                    0-management: replica-count is 2<br>
                                    [2016-12-07 09:15:35.677735] E
                    [MSGID: 106291]<br>
                                    [glusterd-brick-ops.c:614:__gl<wbr>usterd_handle_add_brick]<br>
                            0-management:<br>
                    <br>
                                    The last log entry indicates that we
                    hit the code path in<br>
                                    gd_addbr_validate_replica_coun<wbr>t
                    ()<br>
                    <br>
                                                    if (replica_count ==<br>
                            volinfo-&gt;replica_count) {<br>
                                                            if
                    (!(total_bricks %<br>
                                    volinfo-&gt;dist_leaf_count)) {<br>
                                                                    ret
                    = 1;<br>
                                                                    goto
                    out;<br>
                                    }<br>
                                                    }<br>
                    <br>
                    <br>
                                It seems unlikely that this snippet was
                    hit because we print<br>
                            the E<br>
                                [MSGID: 106291] in the above message
                    only if ret==-1.<br>
                                gd_addbr_validate_replica_coun<wbr>t()
                    returns -1 and yet not<br>
                            populates<br>
                                err_str only when in volinfo-&gt;type
                    doesn't match any of the<br>
                            known<br>
                                volume types, so volinfo-&gt;type is
                    corrupted perhaps?<br>
                    <br>
                    <br>
                            You are right, I missed that ret is set to 1
                    here in the above<br>
                            snippet.<br>
                    <br>
                            @Milos - Can you please provide us the
                    volume info file from<br>
                            /var/lib/glusterd/vols/&lt;volnam<wbr>e&gt;/
                    from all the three nodes to<br>
                            continue<br>
                            the analysis?<br>
                    <br>
                    <br>
                    <br>
                                -Ravi<br>
                    <br>
                                    @Pranith, Ravi - Milos was trying to
                    convert a dist (1 X 1)<br>
                                    volume to a replicate (1 X 2) using
                    add brick and hit<br>
                            this issue<br>
                                    where add-brick failed. The cluster
                    is operating with 3.7.6.<br>
                                    Could you help on what scenario this
                    code path can be<br>
                            hit? One<br>
                                    straight forward issue I see here is
                    missing err_str in<br>
                            this path.<br>
                    <br>
                    <br>
                    <br>
                    <br>
                    <br>
                    <br>
                            --<br>
                    <br>
                            ~ Atin (atinm)<br>
                    <br>
                    <br>
                    <br>
                            --<br>
                    <br>
                            ~ Atin (atinm)<br>
                    <br>
                    <br>
                    <br>
                    <br>
                    --<br>
                    <br>
                    ~ Atin (atinm)<br>
                  </div>
                </div>
              </blockquote>
            </blockquote>
          </div>
          <br>
          <br clear="all">
          <br>
          -- <br>
          <div class="gmail_signature" data-smartmail="gmail_signature">
            <div dir="ltr">
              <div>
                <div dir="ltr"><br>
                </div>
                <div>~ Atin (atinm)<br>
                </div>
              </div>
            </div>
          </div>
        </div>
      </div>
    </blockquote>
    <p><br>
    </p>
  </body>
</html>