<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 2, 2016 at 9:54 AM, Serkan Çoban <span dir="ltr"><<a href="mailto:cobanserkan@gmail.com" target="_blank">cobanserkan@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">+1 for "no-rewinddir-support" option in DHT.<br>
We are seeing very slow directory listing specially with 1500+ brick<br>
volume, 'ls' takes 20+ second with 1000+ files.<br></blockquote><div><br></div><div style="">If its not clear, I would like to point out that serialized readdir is not the sole issue that's causing slowness. If directories are _HUGE_ then I don't expect too much of benefit from parallelizing. Also, as others have been pointing out (in various in-person discussions) there are other scalability limits like number of messages, memory consumed etc to wind calls parallely. I'll probably do a rough POC in next couple of months to see whether this idea has any substance or not and post the results.</div><div style=""><br></div><div style=""><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div class="h5"><br>
On Wed, Nov 2, 2016 at 7:08 AM, Raghavendra Gowdappa<br>
<<a href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a>> wrote:<br>
><br>
><br>
> ----- Original Message -----<br>
>> From: "Keiviw" <<a href="mailto:keiviw@163.com">keiviw@163.com</a>><br>
>> To: <a href="mailto:gluster-devel@gluster.org">gluster-devel@gluster.org</a><br>
>> Sent: Tuesday, November 1, 2016 12:41:02 PM<br>
>> Subject: [Gluster-devel] A question of GlusterFS dentries!<br>
>><br>
>> Hi,<br>
>> In GlusterFS distributed volumes, listing a non-empty directory was slow.<br>
>> Then I read the dht codes and found the reasons. But I was confused that<br>
>> GlusterFS dht travesed all the bricks(in the volume) sequentially,why not<br>
>> use multi-thread to read dentries from multiple bricks simultaneously.<br>
>> That's a question that's always puzzled me, Couly you please tell me<br>
>> something about this???<br>
><br>
> readdir across subvols is sequential mostly because we have to support rewinddir(3). We need to maintain the mapping of offset and dentry across multiple invocations of readdir. In other words if someone did a rewinddir to an offset corresponding to earlier dentry, subsequent readdirs should return same set of dentries what the earlier invocation of readdir returned. For example, in an hypothetical scenario, readdir returned following dentries:<br>
><br>
> 1. a, off=10<br>
> 2. b, off=2<br>
> 3. c, off=5<br>
> 4. d, off=15<br>
> 5. e, off=17<br>
> 6. f, off=13<br>
><br>
> Now if we did rewinddir to off 5 and issue readdir again we should get following dentries:<br>
> (c, off=5), (d, off=15), (e, off=17), (f, off=13)<br>
><br>
> Within a subvol backend filesystem provides rewinddir guarantee for the dentries present on that subvol. However, across subvols it is the responsibility of DHT to provide the above guarantee. Which means we should've some well defined order in which we send readdir calls (Note that order is not well defined if we do a parallel readdir across all subvols). So, DHT has sequential readdir which is a well defined order of reading dentries.<br>
><br>
> To give an example if we have another subvol - subvol2 - (in addiction to the subvol above - say subvol1) with following listing:<br>
> 1. g, off=16<br>
> 2. h, off=20<br>
> 3. i, off=3<br>
> 4. j, off=19<br>
><br>
> With parallel readdir we can have many ordering like - (a, b, g, h, i, c, d, e, f, j), (g, h, a, b, c, i, j, d, e, f) etc. Now if we do (with readdir done parallely):<br>
><br>
> 1. A complete listing of the directory (which can be any one of 10P1 = 10 ways - I hope math is correct here).<br>
> 2. Do rewinddir (20)<br>
><br>
> We cannot predict what are the set of dentries that come _after_ offset 20. However, if we do a readdir sequentially across subvols there is only one directory listing i.e, (a, b, c, d, e, f, g, h, i, j). So, its easier to support rewinddir.<br>
><br>
> If there is no POSIX requirement for rewinddir support, I think a parallel readdir can easily be implemented (which improves performance too). But unfortunately rewinddir is still a POSIX requirement. This also opens up another possibility of a "no-rewinddir-support" option in DHT, which if enabled results in parallel readdirs across subvols. What I am not sure is how many users still use rewinddir? If there is a critical mass which wants performance with a tradeoff of no rewinddir support this can be a good feature.<br>
><br>
> +gluster-users to get an opinion on this.<br>
><br>
> regards,<br>
> Raghavendra<br>
><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> ______________________________<wbr>_________________<br>
>> Gluster-devel mailing list<br>
>> <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
>> <a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br>
> ______________________________<wbr>_________________<br>
</div></div>> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
<div class="HOEnZb"><div class="h5">______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">Raghavendra G<br></div>
</div></div>