<html><body><div style="font-family: garamond,new york,times,serif; font-size: 12pt; color: #000000"><div><br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Vijay Bellur" <vbellur@redhat.com><br><b>To: </b>"Krutika Dhananjay" <kdhananj@redhat.com><br><b>Cc: </b>"Gluster Devel" <gluster-devel@gluster.org><br><b>Sent: </b>Tuesday, February 24, 2015 11:35:28 AM<br><b>Subject: </b>Re: [Gluster-devel] Sharding - Inode write fops - recoverability from failures - design<br><div><br></div>On 02/24/2015 10:36 AM, Krutika Dhananjay wrote:<br>><br>><br>> ------------------------------------------------------------------------<br>><br>> *From: *"Vijay Bellur" <vbellur@redhat.com><br>> *To: *"Krutika Dhananjay" <kdhananj@redhat.com>, "Gluster Devel"<br>> <gluster-devel@gluster.org><br>> *Sent: *Monday, February 23, 2015 5:25:57 PM<br>> *Subject: *Re: [Gluster-devel] Sharding - Inode write fops -<br>> recoverability from failures - design<br>><br>> On 02/22/2015 06:08 PM, Krutika Dhananjay wrote:<br>> > Hi,<br>> ><br>> > Please find the design doc for one of the problems in sharding which<br>> > Pranith and I are trying to solve and its solution @<br>> > http://review.gluster.org/#/c/9723/1.<br>> > Reviews and feedback are much appreciated.<br>> ><br>><br>> Can this feature be made optional? I think there are use cases like<br>> virtual machine image storage, hdfs etc. where the number of metadata<br>> queries might not be very high. It would be an acceptable tradeoff in<br>> such cases to not be very efficient for answering metadata queries but<br>> be very efficient for data operations.<br>><br>> IOW, can we have two possible modes of operation for the sharding<br>> translator to answer metadata queries?<br>><br>> 1. One that behaves like a regular filesystem where we expect a mix of<br>> data and metadata operations. Your document seems to cover that part<br>> well. We can look at optimizing behavior for multi-threaded single<br>> writer use cases after an initial implementation is in place.<br>> Techniques<br>> like eager locking can be applied here.<br>><br>> 2. Another mode where we do not expect a lot of metadata queries. In<br>> this mode, we can visit all nodes where we have shards to answer these<br>> queries.<br>><br>> But for sharding translator to be able to visit all shards, it is<br>> required to know the last shard number.<br>> Without this, it will never know when to stop looking up the different<br>> shards. For this to happen, we<br>> still need to maintain the size attribute for each file.<br>><br><div><br></div>Wouldn't maintaining the total number of shards in the metadata shard be <br>sufficient?</blockquote><div>Maintaining the correctness of "total number of shards" would again incur the same cost as maintaining size or any other metadata attribute if a client/brick crashes in the middle of a write fop before the attribute is committed to disk.</div><div>In other words, we will again need to maintain a "dirty" and "committed" copy of the shard_count to ensure its correctness.</div><div><br></div><div>-Krutika</div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><br><div><br></div>-Vijay<br><div><br></div></blockquote><div><br></div></div></body></html>