<div dir="ltr">Hello,<div><br></div><div>We have a 5 node gluster setup with the following configuration;</div><div><br></div><div><br></div><div><div>[root@gb0015nasslow01 Test]# gluster vol info all</div><div> </div><div>Volume Name: data</div><div>Type: Tier</div><div>Volume ID: 702daa3d-b3fa-4e66-bcec-a13f7ec1d47d</div><div>Status: Started</div><div>Number of Bricks: 6</div><div>Transport-type: tcp</div><div>Hot Tier :</div><div>Hot Tier Type : Replicate</div><div>Number of Bricks: 1 x 3 = 3</div><div>Brick1: glusternasbackup01:/bricks/hotbrick/brick</div><div>Brick2: glusternasfast02:/bricks/brick4/brick</div><div>Brick3: glusternasfast01:/bricks/brick3/brick</div><div>Cold Tier:</div><div>Cold Tier Type : Replicate</div><div>Number of Bricks: 1 x 3 = 3</div><div>Brick4: glusternasslow01:/bricks/brick1/brick</div><div>Brick5: glusternasslow02:/bricks/brick2/brick</div><div>Brick6: glusternasbackup01:/bricks/coldbrick/brick</div><div>Options Reconfigured:</div><div>performance.write-behind-window-size: 100mb</div><div>performance.cache-size: 20GB</div><div>storage.batch-fsync-delay-usec: 0</div><div>server.allow-insecure: on</div><div>performance.stat-prefetch: off</div><div>performance.readdir-ahead: on</div><div>features.ctr-enabled: on</div><div>cluster.tier-mode: cache</div></div><div><br></div><div><br></div><div>When we shutdown/reboot/pull the power from 1 of the nodes, the other servers are very laggy when even doing an ls within the glusterfs mounted filesystem (/glusterdata).</div><div><br></div><div><div>df -h</div><div>Filesystem               Size  Used Avail Use% Mounted on</div><div>/dev/mapper/centos-root   50G  4.7G   46G  10% /</div><div>devtmpfs                  48G     0   48G   0% /dev</div><div>tmpfs                     48G     0   48G   0% /dev/shm</div><div>tmpfs                     48G   66M   48G   1% /run</div><div>tmpfs                     48G     0   48G   0% /sys/fs/cgroup</div><div>/dev/sdb1                110T  221M  110T   1% /bricks/brick1</div><div>/dev/sda2                497M  201M  297M  41% /boot</div><div>/dev/sda1                200M  9.5M  191M   5% /boot/efi</div><div>/dev/mapper/centos-home  225G   33M  224G   1% /home</div><div>localhost:data           109T  464G  108T   1% /glusterdata</div><div>tmpfs                    9.5G     0  9.5G   0% /run/user/0</div></div><div><br></div><div>During the time the node is down or coming back up, we not only get the lag described above on the server but file reads and writes to the gluster volume pretty much stall. </div><div><br></div><div><br></div><div><br></div><div><br></div><div>Can anyone give me some much needed advice as surely this should not be the case?</div><div><br></div><div><br></div><div><br></div><div><br></div><div><b><u>packages installed;</u></b></div><div><br></div><div><div>yum list gluster*</div><div><br></div><div>Installed Packages</div><div>glusterfs.x86_64                        3.7.14-1.el7         @glusterfs-3.7-x86_64</div><div>glusterfs-api.x86_64                    3.7.14-1.el7         @glusterfs-3.7-x86_64</div><div>glusterfs-cli.x86_64                    3.7.14-1.el7         @glusterfs-3.7-x86_64</div><div>glusterfs-client-xlators.x86_64         3.7.14-1.el7         @glusterfs-3.7-x86_64</div><div>glusterfs-fuse.x86_64                   3.7.14-1.el7         @glusterfs-3.7-x86_64</div><div>glusterfs-libs.x86_64                   3.7.14-1.el7         @glusterfs-3.7-x86_64</div><div>glusterfs-server.x86_64                 3.7.14-1.el7         @glusterfs-3.7-x86_64</div></div><div><br></div><div><br></div><div><div>yum list samba*</div><div><br></div><div>Installed Packages</div><div>samba.x86_64                                                                        4.2.10-7.el7_2                                                   @centos7-updates-x86_64</div><div>samba-client.x86_64                                                                 4.2.10-7.el7_2                                                   @centos7-updates-x86_64</div><div>samba-client-libs.x86_64                                                            4.2.10-7.el7_2                                                   @centos7-updates-x86_64</div><div>samba-common.noarch                                                                 4.2.10-7.el7_2                                                   @centos7-updates-x86_64</div><div>samba-common-libs.x86_64                                                            4.2.10-7.el7_2                                                   @centos7-updates-x86_64</div><div>samba-common-tools.x86_64                                                           4.2.10-7.el7_2                                                   @centos7-updates-x86_64</div><div>samba-libs.x86_64                                                                   4.2.10-7.el7_2                                                   @centos7-updates-x86_64</div><div>samba-vfs-glusterfs.x86_64                                                          4.2.10-7.el7_2                                                   @centos7-updates-x86_64</div><div>samba-winbind.x86_64                                                                4.2.10-7.el7_2                                                   @centos7-updates-x86_64</div><div>samba-winbind-clients.x86_64                                                        4.2.10-7.el7_2                                                   @centos7-updates-x86_64</div><div>samba-winbind-modules.x86_64                                                        4.2.10-7.el7_2                                                   @centos7-updates-x86_64</div></div><div><br></div><div><div><br></div><div>yum list ctdb*</div><div><br></div><div>Installed Packages</div><div>ctdb.x86_64                                                                 4.2.10-7.el7_2                                                           @centos7-updates-x86_64</div></div><div><br></div><div><br></div><div>Background - We use CTDB for VIP&#39;s and heartbeat and we present our gluster volume via samba to our Windows clients.</div><div><br></div><div>Regards,</div><div>David</div><div><br></div></div>