<div dir="ltr"><div>Serkan<br><br></div>I'd be interested to know how your disks are attached (SAS?)? Do you use any hardware RAID, or zfs and do you have and SSDs in there?<br></div><div class="gmail_extra"><br><div class="gmail_quote">On 9 November 2016 at 06:17, Serkan Çoban <span dir="ltr"><<a href="mailto:cobanserkan@gmail.com" target="_blank">cobanserkan@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi, I am using 26x8TB disks per server. There are 60 servers in gluster cluster.<br>
Each disk is a brick and configuration is 16+4 EC, 9PB single volume.<br>
Clients are using fuse mounts.<br>
Even with 1-2K files in a directory, ls from clients takes ~60 secs.<br>
So If you are sensitive to metadata operations, I suggest another approach...<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
On Wed, Nov 9, 2016 at 1:05 PM, Frank Rothenstein<br>
<<a href="mailto:f.rothenstein@bodden-kliniken.de">f.rothenstein@bodden-<wbr>kliniken.de</a>> wrote:<br>
> As you said you want to have 3 or 4 replicas, so i would use the zfs<br>
> knowledge and build 1 zpool per node with whatever config you know is<br>
> fastest on this kind of hardware and as safe as you need (stripe,<br>
> mirror, raidz1..3 - resilvering zfs is faster than healing gluster, I<br>
> think) . 1 node -> 1 brick (per gluster volume).<br>
><br>
> Frank<br>
> Am Dienstag, den 08.11.2016, 19:19 +0000 schrieb Thomas Wakefield:<br>
>> We haven’t decided how the JBODS would be configured. They would<br>
>> likely be SAS attached without a raid controller for improved<br>
>> performance. I run large ZFS arrays this way, but only in single<br>
>> server NFS setups right now.<br>
>> Mounting each hard drive as it’s own brick would probably give the<br>
>> most usable space, but would need scripting to manage building all<br>
>> the bricks. But does Gluster handle 1000’s of small bricks?<br>
>><br>
>><br>
>><br>
>> > On Nov 8, 2016, at 9:18 AM, Frank Rothenstein <f.rothenstein@bodden<br>
>> > -<a href="http://kliniken.de" rel="noreferrer" target="_blank">kliniken.de</a>> wrote:<br>
>> ><br>
>> > Hi Thomas,<br>
>> ><br>
>> > thats a huge storage.<br>
>> > What I can say from my usecase - dont use Gluster directly if the<br>
>> > files<br>
>> > are small. I dont know, if the file count matters, but if the files<br>
>> > are<br>
>> > small (few KiB), Gluster takes ages to remove for example. Doing<br>
>> > the<br>
>> > same in a VM with e.g. ext4 disk on the very same Gluster gives a<br>
>> > big<br>
>> > speedup.<br>
>> > There are many options for a new Gluster volume, like Lindsay<br>
>> > mentioned.<br>
>> > And there are other options, like Ceph, OrangeFS.<br>
>> > How do you want to use the JBODs? I dont think you would use every<br>
>> > single drive as a brick... How are these connected to the servers?<br>
>> ><br>
>> > Im only dealing with about 10TiB Gluster volumes, so by far not at<br>
>> > your<br>
>> > planned level, but I really would like to see some results, if you<br>
>> > go<br>
>> > for Gluster!<br>
>> ><br>
>> > Frank<br>
>> ><br>
>> ><br>
>> > Am Dienstag, den 08.11.2016, 13:49 +0000 schrieb Thomas Wakefield:<br>
>> > > I think we are leaning towards erasure coding with 3 or 4<br>
>> > > copies. But open to suggestions.<br>
>> > ><br>
>> > ><br>
>> > > > On Nov 8, 2016, at 8:43 AM, Lindsay Mathieson <lindsay.mathieso<br>
>> > > > n@gm<br>
>> > > > <a href="http://ail.com" rel="noreferrer" target="_blank">ail.com</a>> wrote:<br>
>> > > ><br>
>> > > > On 8/11/2016 11:38 PM, Thomas Wakefield wrote:<br>
>> > > > > High Performance Computing, we have a small cluster on campus<br>
>> > > > > of<br>
>> > > > > about 50 linux compute servers.<br>
>> > > > ><br>
>> > > ><br>
>> > > > D'oh! I should have thought of that.<br>
>> > > ><br>
>> > > ><br>
>> > > > Are you looking at replication (2 or 3)/disperse or pure<br>
>> > > > disperse?<br>
>> > > ><br>
>> > > > --<br>
>> > > > Lindsay Mathieson<br>
>> > > ><br>
>> > ><br>
>> > > ______________________________<wbr>_________________<br>
>> > > Gluster-users mailing list<br>
>> > > <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
>> > > <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> > ______________________________<wbr>______________________________<wbr>_______<br>
>> > ___________<br>
>> > BODDEN-KLINIKEN Ribnitz-Damgarten GmbH<br>
>> > Sandhufe 2<br>
>> > 18311 Ribnitz-Damgarten<br>
>> ><br>
>> > Telefon: 03821-700-0<br>
>> > Fax: 03821-700-240<br>
>> ><br>
>> > E-Mail: <a href="mailto:info@bodden-kliniken.de">info@bodden-kliniken.de</a> Internet: <a href="http://www.bodden-klini" rel="noreferrer" target="_blank">http://www.bodden-klini</a><br>
>> > <a href="http://ken.de" rel="noreferrer" target="_blank">ken.de</a><br>
>> ><br>
>> > Sitz: Ribnitz-Damgarten, Amtsgericht: Stralsund, HRB 2919, Steuer-<br>
>> > Nr.: 079/133/40188<br>
>> > Aufsichtsratsvorsitzende: Carmen Schröter, Geschäftsführer: Dr.<br>
>> > Falko Milski<br>
>> ><br>
>> > Der Inhalt dieser E-Mail ist ausschließlich für den bezeichneten<br>
>> > Adressaten bestimmt. Wenn Sie nicht der vorge-<br>
>> > sehene Adressat dieser E-Mail oder dessen Vertreter sein sollten,<br>
>> > beachten Sie bitte, dass jede Form der Veröf-<br>
>> > fentlichung, Vervielfältigung oder Weitergabe des Inhalts dieser E-<br>
>> > Mail unzulässig ist. Wir bitten Sie, sofort den<br>
>> > Absender zu informieren und die E-Mail zu löschen.<br>
>> ><br>
>> ><br>
>> > Bodden-Kliniken Ribnitz-Damgarten GmbH 2016<br>
>> > *** Virenfrei durch Kerio Mail Server und Sophos Antivirus ***<br>
>> ><br>
>><br>
>><br>
><br>
><br>
><br>
><br>
><br>
> ______________________________<wbr>______________________________<wbr>__________________<br>
> BODDEN-KLINIKEN Ribnitz-Damgarten GmbH<br>
> Sandhufe 2<br>
> 18311 Ribnitz-Damgarten<br>
><br>
> Telefon: 03821-700-0<br>
> Fax: 03821-700-240<br>
><br>
> E-Mail: <a href="mailto:info@bodden-kliniken.de">info@bodden-kliniken.de</a> Internet: <a href="http://www.bodden-kliniken.de" rel="noreferrer" target="_blank">http://www.bodden-kliniken.de</a><br>
><br>
> Sitz: Ribnitz-Damgarten, Amtsgericht: Stralsund, HRB 2919, Steuer-Nr.: 079/133/40188<br>
> Aufsichtsratsvorsitzende: Carmen Schröter, Geschäftsführer: Dr. Falko Milski<br>
><br>
> Der Inhalt dieser E-Mail ist ausschließlich für den bezeichneten Adressaten bestimmt. Wenn Sie nicht der vorge-<br>
> sehene Adressat dieser E-Mail oder dessen Vertreter sein sollten, beachten Sie bitte, dass jede Form der Veröf-<br>
> fentlichung, Vervielfältigung oder Weitergabe des Inhalts dieser E-Mail unzulässig ist. Wir bitten Sie, sofort den<br>
> Absender zu informieren und die E-Mail zu löschen.<br>
><br>
><br>
> Bodden-Kliniken Ribnitz-Damgarten GmbH 2016<br>
> *** Virenfrei durch Kerio Mail Server und Sophos Antivirus ***<br>
><br>
> ______________________________<wbr>_________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/<wbr>mailman/listinfo/gluster-users</a></div></div></blockquote></div><br></div>