<div dir="ltr">I have a 3x2 hot tier on NVMe drives with a 3x2 cold tier on RAID6 drives.<div><br></div><div><div># gluster vol info 1nvme-distrep3x2</div><div> </div><div>Volume Name: 1nvme-distrep3x2</div><div>Type: Tier</div><div>Volume ID: 21e3fc14-c35c-40c5-8e46-c258c1302607</div><div>Status: Started</div><div>Number of Bricks: 12</div><div>Transport-type: tcp</div><div>Hot Tier :</div><div>Hot Tier Type : Distributed-Replicate</div><div>Number of Bricks: 3 x 2 = 6</div><div>Brick1: n5:/rhgs/hotbricks/1nvme-distrep3x2-hot</div><div>Brick2: n4:/rhgs/hotbricks/1nvme-distrep3x2-hot</div><div>Brick3: n3:/rhgs/hotbricks/1nvme-distrep3x2-hot</div><div>Brick4: n2:/rhgs/hotbricks/1nvme-distrep3x2-hot</div><div>Brick5: n1:/rhgs/hotbricks/1nvme-distrep3x2-hot</div><div>Brick6: n0:/rhgs/hotbricks/1nvme-distrep3x2-hot</div><div>Cold Tier:</div><div>Cold Tier Type : Distributed-Replicate</div><div>Number of Bricks: 3 x 2 = 6</div><div>Brick7: n0:/rhgs/coldbricks/1nvme-distrep3x2</div><div>Brick8: n1:/rhgs/coldbricks/1nvme-distrep3x2</div><div>Brick9: n2:/rhgs/coldbricks/1nvme-distrep3x2</div><div>Brick10: n3:/rhgs/coldbricks/1nvme-distrep3x2</div><div>Brick11: n4:/rhgs/coldbricks/1nvme-distrep3x2</div><div>Brick12: n5:/rhgs/coldbricks/1nvme-distrep3x2</div><div>Options Reconfigured:</div><div>cluster.tier-mode: cache</div><div>features.ctr-enabled: on</div><div>performance.readdir-ahead: on</div><div><br></div><div><br></div><div>I am attempting to run the 'smallfile' benchmark tool on this volume. The 'smallfile' tool creates a starting gate directory and files in a shared filesystem location. The first run (write) works as expected.</div><div><br></div><div># smallfile_cli.py --threads 12 --file-size 4096 --files 300 --top /rhgs/client/1nvme-distrep3x2 --host-set c0,c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11 --prefix test1 --stonewall Y --network-sync-dir /rhgs/client/1nvme-distrep3x2/smf1 --operation create<br></div><div><br></div><div>For the second run (read), I believe that smallfile attempts first to 'rm -rf' the "network-sync-dir" path, which fails with ENOTEMPTY, causing the run to fail</div><div><br></div><div># smallfile_cli.py --threads 12 --file-size 4096 --files 300 --top /rhgs/client/1nvme-distrep3x2 --host-set c0,c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11 --prefix test1 --stonewall Y --network-sync-dir /rhgs/client/1nvme-distrep3x2/smf1 --operation create<br></div><div>...</div><div><div>Traceback (most recent call last):</div><div> File "/root/bin/smallfile_cli.py", line 280, in <module></div><div> run_workload()</div><div> File "/root/bin/smallfile_cli.py", line 270, in run_workload</div><div> return run_multi_host_workload(params)</div><div> File "/root/bin/smallfile_cli.py", line 62, in run_multi_host_workload</div><div> sync_files.create_top_dirs(master_invoke, True)</div><div> File "/root/bin/sync_files.py", line 27, in create_top_dirs</div><div> shutil.rmtree(master_invoke.network_dir)</div><div> File "/usr/lib64/python2.7/shutil.py", line 256, in rmtree</div><div> onerror(os.rmdir, path, sys.exc_info())</div><div> File "/usr/lib64/python2.7/shutil.py", line 254, in rmtree</div><div> os.rmdir(path)</div><div>OSError: [Errno 39] Directory not empty: '/rhgs/client/1nvme-distrep3x2/smf1'</div></div><div><br></div><div><br></div><div>From the client perspective, the directory is clearly empty.</div><div><br></div><div><div># ls -a /rhgs/client/1nvme-distrep3x2/smf1/</div><div>. ..</div></div><div><br></div><div><br></div><div>And a quick search on the bricks shows that the hot tier on the last replica pair is the offender.</div><div><br></div><div><div># for i in {0..5}; do ssh n$i "hostname; ls /rhgs/coldbricks/1nvme-distrep3x2/smf1 | wc -l; ls /rhgs/hotbricks/1nvme-distrep3x2-hot/smf1 | wc -l"; donerhosd0</div><div>0</div><div>0</div><div>rhosd1</div><div>0</div><div>0</div><div>rhosd2</div><div>0</div><div>0</div><div>rhosd3</div><div>0</div><div>0</div><div>rhosd4</div><div>0</div><div>1</div><div>rhosd5</div><div>0</div><div>1</div></div><div><br></div><div><br></div><div>(For the record, multiple runs of this reproducer show that it is consistently the hot tier that is to blame, but it is not always the same replica pair.)</div><div><br></div><div><br></div><div>Can someone try recreating this scenario to see if the problem is consistent? Please reach out if you need me to provide any further details.</div><div><br></div><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><br><div dir="ltr" style="font-size:small">Dustin Black, RHCA </div><div dir="ltr" style="font-size:small">Senior Architect, Software-Defined Storage</div><div dir="ltr" style="font-size:small">Red Hat, Inc.<br>(o) +1.212.510.4138 (m) +1.215.821.7423 <br><a href="mailto:dustin@redhat.com" style="color:rgb(17,85,204)" target="_blank">dustin@redhat.com</a> <br> </div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div></div>