The problem you describe is very specific to glfs_new(), and not to gfapi in general. I guess we can handle this in glfs_new by initializing an appropriate value into THIS (save old_THIS and restore it before returning from glfs_new). That should avoid the need for all those new macros?<br><br>Thanks<br><br><div class="gmail_quote">On Wed, Jan 21, 2015, 23:37 Raghavendra Bhat <<a href="mailto:rabhat@redhat.com">rabhat@redhat.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Hi,<br>
<br>
In glusterfs at the time of the process coming up, it creates 5 pthread<br>
keys (for saving THIS, syncop, uuid buf, lkowner buf and syncop ctx).<br>
Even gfapi does the same thing in its glfs_new function. But with User<br>
Serviceable Snapshots (where a glusterfs process spawns multiple gfapi<br>
instances for a snapshot) this will lead to more and more consumption of<br>
pthread keys. In fact the old keys are lost (as same variables are used<br>
for creating the keys) and eventually the process will run out of<br>
pthread keys after 203 snapshots (maximum allowed number is 1024 per<br>
process.). So to avoid it, pthread keys creation can be done only once<br>
(using pthread_once in which, the globals_init function is called).<br>
<br>
But now a new problem arises. Say, from snapview-server xlator glfs_new<br>
was called (or glfs_init etc). Now gfapi wants calls THIS for some of<br>
its operations such properly accounting the memory within the xlator<br>
while allocating a new structure. But when gfapi calls THIS it gets<br>
snapview-server's pointer. Since snapview-server does not know about<br>
gfapi's internal structure, it asserts at the time of allocating.<br>
<br>
For now, a patch has been sent to handle the issue by turning off<br>
memory-accounting for snapshot daemon.<br>
(<a href="http://review.gluster.org/#/c/9430" target="_blank">http://review.gluster.org/#/<u></u>c/9430</a>).<br>
<br>
But if memory-accounting has to be turned on for snapshot daemon, then<br>
the above problem has to be fixed.<br>
2 ways that can be used for fixing the issue are:<br>
<br>
1) Add the datastructures that are used by gfapi to libglusterfs (and<br>
hence their mem-types as well), so that any xlator that is calling gfapi<br>
functions (such as snapview-server as of now) will be aware of the<br>
memory types used by gfapi and hence will not cause problems, when<br>
memory accounting has to be done as part of allocations and frees.<br>
<br>
OR<br>
<br>
2) Properly manage THIS by introducing a new macro similar to STACK_WIND<br>
(for now it can be called STACK_API_WIND). The macro will be much<br>
simpler than STACK_WIND as it need not create new frames before handing<br>
over the call to the next layer. Before handing over the call to gfapi<br>
(any call, such as glfs_new or fops such as glfs_h_open), saves THIS in<br>
a variable and calls the gfapi function given as an argument. After the<br>
function returns it again sets THIS back the value before the gfapi<br>
function was called.<br>
<br>
Ex:<br>
<br>
STACK_API_WIND (this, fn, ret, params....)<br>
do {<br>
xlator_t *old_THIS = NULL;<br>
<br>
old_THIS = this;<br>
ret = fn (params);<br>
THIS = old_THIS;<br>
} while (0);<br>
<br>
a caller (as of now snapview-server xlator) would call the macro like this<br>
STACK_API_WIND (this, glfs_h_open, glfd, fs, object, flags);<br>
<br>
<br>
Please provide feedback and any suggestions or solutions to handle the<br>
mentioned issue are welcome.<br>
<br>
Regards,<br>
Raghavendra Bhat<br>
<br>
______________________________<u></u>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-devel" target="_blank">http://www.gluster.org/<u></u>mailman/listinfo/gluster-devel</a><br>
</blockquote></div>