<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hi, Niels de Vos</div><div class=""><br class=""></div><div class="">I’ve been working for about a year using glusterfs in my company.</div><div class=""><br class=""></div><div class="">Recently, we have tested a Distribute-replicated volume using NFS. </div><div class="">We have used rsync to test.</div><div class=""><br class=""></div><div class="">This is my volume configuration</div><div class="">----------------------------------------------------------------------</div><div class=""><div class="">Volume Name: rep4x2</div><div class="">Type: Distributed-Replicate</div><div class="">Volume ID: 7376a4ad-c50b-40d8-8fe1-ab84111ece26</div><div class="">Status: Started</div><div class="">Number of Bricks: 4 x 2 = 8</div><div class="">Transport-type: tcp</div><div class="">Bricks:</div><div class="">Brick1: 10.10.1.151:/volume/rep4x2</div><div class="">Brick2: 10.10.1.152:/volume/rep4x2</div><div class="">Brick3: 10.10.1.153:/volume/rep4x2</div><div class="">Brick4: 10.10.1.154:/volume/rep4x2</div><div class="">Brick5: 10.10.1.155:/volume/rep4x2</div><div class="">Brick6: 10.10.1.156:/volume/rep4x2</div><div class="">Brick7: 10.10.1.157:/volume/rep4x2</div><div class="">Brick8: 10.10.1.158:/volume/rep4x2</div><div class="">Options Reconfigured:</div><div class="">nfs.disable: false</div><div class="">server.root-squash: off</div><div class="">nfs.volume-access: read-write</div><div class="">nfs.rpc-auth-allow: *</div><div class="">nfs.ports-insecure: off</div><div class="">server.allow-insecure: on</div><div class="">diagnostics.brick-sys-log-level: WARNING</div><div class="">diagnostics.client-sys-log-level: WARNING</div><div class="">network.ping-timeout: 5</div><div class="">performance.readdir-ahead: on</div></div><div class="">----------------------------------------------------------------------</div><div class=""><br class=""></div><div class="">plus, I have a node as client</div><div class=""><br class=""></div><div class="">I did below on client</div><div class=""><br class=""></div><div class="">10.10.2.151 is the other address to access Brick1 node.</div><div class=""><br class=""></div><div class=""><b class="">Mount</b></div><div class=""><div class="">Filesystem Type Size Used Avail Use% Mounted on</div><div class="">/dev/sda1 ext3 5.0G 3.1G 1.7G 66% /</div><div class="">/dev/sda2 ext3 5.0G 992M 3.7G 21% /var</div><div class="">none tmpfs 5.9G 120K 5.9G 1% /dev/shm</div><div class="">/dev/mapper/LD-PlugDISK_DB xfs 50G 2.7G 48G 6% /PlugDISK_DB</div><div class="">/dev/mapper/LD-LV xfs 8.2T 4.2T 4.0T 52% /LV</div><div class=""><u class="">10.10.2.151:/rep4x2 nfs 2.3T 35G 2.3T 2% /mnt/ac2-8node</u></div></div><div class=""><br class=""></div><div class=""># rsync -a /source /mnt/ac2-8node</div><div class=""><br class=""></div><div class="">When I did above, rsync generate and write temporary files which look like “.<FILENAME>.[hash]”</div><div class=""><br class=""></div><div class="">original: /PATH/CentOS_release_6.5-i686.tar</div><div class="">temporary file: /PATH/.CentOS_release_6.5-i686.tar.3UUsQ6</div><div class=""><br class=""></div><div class="">and some files raise I/O error while writing.</div><div class=""><br class=""></div><div class="">but when I did same act into a path which is mounted as fuse, It worked properly.</div><div class=""><br class=""></div><div class="">/var/log/glusterfs/nfs.log</div><div class="">: <a href="https://gist.github.com/Alghost/86a4b6c9f26c18a8e3af26628571a2df" class="">https://gist.github.com/Alghost/86a4b6c9f26c18a8e3af26628571a2df</a></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">volume status is below</div><div class=""><br class=""></div><div class=""><div class="">Status of volume: rep4x2</div><div class="">Gluster process TCP Port RDMA Port Online Pid</div><div class="">------------------------------------------------------------------------------</div><div class="">Brick 10.10.1.151:/volume/rep4x2 49153 0 Y 2401 </div><div class="">Brick 10.10.1.152:/volume/rep4x2 49153 0 Y 16285</div><div class="">Brick 10.10.1.153:/volume/rep4x2 49153 0 Y 24049</div><div class="">Brick 10.10.1.154:/volume/rep4x2 49152 0 Y 6470 </div><div class="">Brick 10.10.1.155:/volume/rep4x2 49152 0 Y 31469</div><div class="">Brick 10.10.1.156:/volume/rep4x2 49152 0 Y 25676</div><div class="">Brick 10.10.1.157:/volume/rep4x2 49152 0 Y 20197</div><div class="">Brick 10.10.1.158:/volume/rep4x2 49152 0 Y 12305</div><div class="">NFS Server on localhost 2049 0 Y 2447 </div><div class="">Self-heal Daemon on localhost N/A N/A Y 2457 </div><div class="">NFS Server on 10.10.1.153 2049 0 Y 24185</div><div class="">Self-heal Daemon on 10.10.1.153 N/A N/A Y 24193</div><div class="">NFS Server on 10.10.1.158 2049 0 Y 12325</div><div class="">Self-heal Daemon on 10.10.1.158 N/A N/A Y 12355</div><div class="">NFS Server on 10.10.1.152 2049 0 Y 16321</div><div class="">Self-heal Daemon on 10.10.1.152 N/A N/A Y 16329</div><div class="">NFS Server on 10.10.1.155 2049 0 Y 31490</div><div class="">Self-heal Daemon on 10.10.1.155 N/A N/A Y 31598</div><div class="">NFS Server on 10.10.1.156 2049 0 Y 25696</div><div class="">Self-heal Daemon on 10.10.1.156 N/A N/A Y 25704</div><div class="">NFS Server on 10.10.1.154 2049 0 Y 6490 </div><div class="">Self-heal Daemon on 10.10.1.154 N/A N/A Y 6520 </div><div class="">NFS Server on 10.10.1.157 2049 0 Y 20360</div><div class="">Self-heal Daemon on 10.10.1.157 N/A N/A Y 20369</div><div class=""> </div><div class="">Task Status of Volume rep4x2</div><div class="">------------------------------------------------------------------------------</div><div class="">There are no active volume tasks</div></div><div class=""><br class=""></div><div class="">I guess, It would be related to DHT trick which is mentioned on GlusterSummit</div><div class="">: <a href="https://twitter.com/raghavendra_t/status/784310769491914752" class="">https://twitter.com/raghavendra_t/status/784310769491914752</a></div><div class=""><br class=""></div><div class="">If you want me to test something or give you information, </div><div class="">you can email me anytime!</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Regards,</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-ligatures: normal; font-variant-position: normal; font-variant-caps: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><span class="Apple-style-span" style="border-collapse: separate; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: normal; border-spacing: 0px;"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-ligatures: normal; font-variant-position: normal; font-variant-caps: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-stroke-width: 0px;"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-ligatures: normal; font-variant-position: normal; font-variant-caps: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-stroke-width: 0px;"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant-ligatures: normal; font-variant-position: normal; font-variant-caps: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; border-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-stroke-width: 0px;"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">-----------------------------------------</div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">이 태 화<br class="">Taehwa Lee<br class="">Gluesys Co.,Ltd.<br class=""><a href="mailto:alghost.lee@gmail.com" class="">alghost.lee@gmail.com</a><br class="">010-3420-6114, 070-8785-6591</div><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">-----------------------------------------</div></span></div></span></div></span></div></span></div></div></div>
</div>
<br class=""></body></html>