<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 12pt;
font-family:Calibri
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>Hi,<div><br></div><div>Kindly check the following link for all 7 bricks logs;</div><div><br></div><div><a href="https://db.tt/YP5qTGXk" target="_blank">https://db.tt/YP5qTGXk</a><br><br><br>
<div class="ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br><br></font><font color="#3366ff"></font></div><br><br><div><hr id="stopSpelling">From: kdhananj@redhat.com<br>Date: Thu, 4 Aug 2016 13:00:43 +0530<br>Subject: Re: [Gluster-users] Gluster 3.7.13 NFS Crash<br>To: mahdi.adnan@outlook.com<br>CC: gluster-users@gluster.org<br><br><div dir="ltr"><div>Could you also attach the brick logs please?<br><br></div>-Krutika<br></div><div class="ecxgmail_extra"><br><div class="ecxgmail_quote">On Thu, Aug 4, 2016 at 12:48 PM, Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br><blockquote class="ecxgmail_quote" style="border-left:1px #ccc solid;padding-left:1ex;">
<div><div dir="ltr">appreciate your help,<div><br></div><div>(gdb) frame 2</div><span><div>#2 0x00007f195deb1787 in shard_common_inode_write_do (frame=0x7f19699f1164, this=0x7f195802ac10) at shard.c:3716</div></span><div>3716<span style="white-space:pre-wrap;"> </span> anon_fd = fd_anonymous (local->inode_list[i]);</div><div>(gdb) p local->inode_list[0]</div><div>$4 = (inode_t *) 0x7f195c532b18</div><div>(gdb) p local->inode_list[1]</div><div>$5 = (inode_t *) 0x0</div><div>(gdb) <span><br><br><br>
<div>-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br><br></font><font color="#3366ff"></font></div><br><br></span><div><hr>From: <a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a><br>Date: Thu, 4 Aug 2016 12:43:10 +0530<div><div class="h5"><br>Subject: Re: [Gluster-users] Gluster 3.7.13 NFS Crash<br>To: <a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a><br>CC: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br><br><div dir="ltr"><div><div><div><div><div>OK.<br></div>Could you also print the values of the following variables from the original core:<br></div>i. i<br></div>ii. local->inode_list[0]<br></div>iii. local->inode_list[1]<br><br></div>-Krutika<br></div><div><br><div>On Wed, Aug 3, 2016 at 9:01 PM, Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br><blockquote style="border-left:1px #ccc solid;padding-left:1ex;">
<div><div dir="ltr">Hi,<div><br></div><div>Unfortunately no, but i can setup a test bench and see if it gets the same results.<span><br><br>
<div>-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br></font><font color="#3366ff"><br></font></div><br><br></span><div><hr>From: <a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a><br>Date: Wed, 3 Aug 2016 20:59:50 +0530<div><div><br>Subject: Re: [Gluster-users] Gluster 3.7.13 NFS Crash<br>To: <a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a><br>CC: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br><br><div dir="ltr"><div>Do you have a test case that consistently recreates this problem?<br><br></div>-Krutika<br></div><div><br><div>On Wed, Aug 3, 2016 at 8:32 PM, Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br><blockquote style="border-left:1px #ccc solid;padding-left:1ex;">
<div><div dir="ltr"><div><span style="font-size:12pt;">Hi,</span></div><div><span style="font-size:12pt;"><br></span></div><div> So i have updated to 3.7.14 and i still have the same issue with NFS.</div><div>based on what i have provided so far from logs and dumps do you think it's an NFS issue ? should i switch to nfs-ganesha ?<br>the problem is, the current setup is used in a production environment, and switching the mount point of +50 VMs from native nfs to nfs-ganesha is not going to be smooth and without downtime, so i really appreciate your thoughts on this matter.<span><br><br>
<div>-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br><br></font><font color="#3366ff"></font></div><br><br></span><div><span><hr>From: <a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a><br>To: <a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a><br></span>Date: Tue, 2 Aug 2016 08:44:16 +0300<div><div><br>CC: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>Subject: Re: [Gluster-users] Gluster 3.7.13 NFS Crash<br><br>
<div dir="ltr">Hi,<div><br></div><div>The NFS just crashed again, latest bt;</div><div><br></div><div><div>(gdb) bt</div><div>#0 0x00007f0b71a9f210 in pthread_spin_lock () from /lib64/libpthread.so.0</div><div>#1 0x00007f0b72c6fcd5 in fd_anonymous (inode=0x0) at fd.c:804</div><div>#2 0x00007f0b64ca5787 in shard_common_inode_write_do (frame=0x7f0b707c062c, this=0x7f0b6002ac10) at shard.c:3716</div><div>#3 0x00007f0b64ca5a53 in shard_common_inode_write_post_<wbr>lookup_shards_handler (frame=<optimized out>, this=<optimized out>) at shard.c:3769</div><div>#4 0x00007f0b64c9eff5 in shard_common_lookup_shards_cbk (frame=0x7f0b707c062c, cookie=<optimized out>, this=0x7f0b6002ac10, op_ret=0, </div><div> op_errno=<optimized out>, inode=<optimized out>, buf=0x7f0b51407640, xdata=0x7f0b72f57648, postparent=0x7f0b514076b0) at shard.c:1601</div><div>#5 0x00007f0b64efe141 in dht_lookup_cbk (frame=0x7f0b7075fcdc, cookie=<optimized out>, this=<optimized out>, op_ret=0, op_errno=0, inode=0x7f0b5f1d1f58, </div><div> stbuf=0x7f0b51407640, xattr=0x7f0b72f57648, postparent=0x7f0b514076b0) at dht-common.c:2174</div><div>#6 0x00007f0b651871f3 in afr_lookup_done (frame=frame@entry=0x7f0b7079a<wbr>4c8, this=this@entry=0x7f0b60023ba0<wbr>) at afr-common.c:1825</div><div>#7 0x00007f0b65187b84 in afr_lookup_metadata_heal_check (frame=frame@entry=0x7f0b7079a<wbr>4c8, this=0x7f0b60023ba0, this@entry=0xca0bd88259f5a800)</div><div> at afr-common.c:2068</div><div>#8 0x00007f0b6518834f in afr_lookup_entry_heal (frame=frame@entry=0x7f0b7079a<wbr>4c8, this=0xca0bd88259f5a800, this@entry=0x7f0b60023ba0) at afr-common.c:2157</div><div>#9 0x00007f0b6518867d in afr_lookup_cbk (frame=0x7f0b7079a4c8, cookie=<optimized out>, this=0x7f0b60023ba0, op_ret=<optimized out>, </div><div> op_errno=<optimized out>, inode=<optimized out>, buf=0x7f0b564e9940, xdata=0x7f0b72f708c8, postparent=0x7f0b564e99b0) at afr-common.c:2205</div><div>#10 0x00007f0b653d6e42 in client3_3_lookup_cbk (req=<optimized out>, iov=<optimized out>, count=<optimized out>, myframe=0x7f0b7076354c)</div><div> at client-rpc-fops.c:2981</div><div>#11 0x00007f0b72a00a30 in rpc_clnt_handle_reply (clnt=clnt@entry=0x7f0b603393c<wbr>0, pollin=pollin@entry=0x7f0b50c1<wbr>c2d0) at rpc-clnt.c:764</div><div>#12 0x00007f0b72a00cef in rpc_clnt_notify (trans=<optimized out>, mydata=0x7f0b603393f0, event=<optimized out>, data=0x7f0b50c1c2d0) at rpc-clnt.c:925</div><div>#13 0x00007f0b729fc7c3 in rpc_transport_notify (this=this@entry=0x7f0b6034904<wbr>0, event=event@entry=RPC_TRANSPOR<wbr>T_MSG_RECEIVED, data=data@entry=0x7f0b50c1c2d0<wbr>)</div><div> at rpc-transport.c:546</div><div>#14 0x00007f0b678c39a4 in socket_event_poll_in (this=this@entry=0x7f0b6034904<wbr>0) at socket.c:2353</div><div>#15 0x00007f0b678c65e4 in socket_event_handler (fd=fd@entry=29, idx=idx@entry=17, data=0x7f0b60349040, poll_in=1, poll_out=0, poll_err=0) at socket.c:2466</div><div>#16 0x00007f0b72ca0f7a in event_dispatch_epoll_handler (event=0x7f0b564e9e80, event_pool=0x7f0b7349bf20) at event-epoll.c:575</div><div>#17 event_dispatch_epoll_worker (data=0x7f0b60152d40) at event-epoll.c:678</div><div>#18 0x00007f0b71a9adc5 in start_thread () from /lib64/libpthread.so.0</div><div>#19 0x00007f0b713dfced in clone () from /lib64/libc.so.6</div><div><br></div><div><br></div>
<div>-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br></font></div><br><div><hr>From: <a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a><br>To: <a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a><br>Date: Mon, 1 Aug 2016 16:31:50 +0300<br>CC: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>Subject: Re: [Gluster-users] Gluster 3.7.13 NFS Crash<br><br>
<div dir="ltr">Many thanks,<div><br></div><div>here's the results;</div><div><br></div><div><br></div><div><div>(gdb) p cur_block</div><div>$15 = 4088</div><div>(gdb) p last_block</div><div>$16 = 4088</div><div>(gdb) p local->first_block</div><div>$17 = 4087</div><div>(gdb) p odirect</div><div>$18 = _gf_false</div><div>(gdb) p fd->flags</div><div>$19 = 2</div><div>(gdb) p local->call_count</div><div>$20 = 2</div><div><br></div><div><br></div><div>If you need more core dumps, i have several files i can upload.</div><br>
<div>-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br><br></font><font color="#3366ff"></font></div><br><br><div><hr>From: <a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a><br>Date: Mon, 1 Aug 2016 18:39:27 +0530<br>Subject: Re: [Gluster-users] Gluster 3.7.13 NFS Crash<br>To: <a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a><br>CC: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br><br><div dir="ltr"><div><div><div><div><div><div><div>Sorry I didn't make myself clear. The reason I asked YOU to do it is because i tried it on my system and im not getting the backtrace (it's all question marks).<br><br>Attach the core to gdb.<br></div>At the gdb prompt, go to frame 2 by typing<br></div>(gdb) f 2<br><br></div>There, for each of the variables i asked you to get the values of, type p followed by the variable name.<br></div>For instance, to get the value of the variable 'odirect', do this:<br><br></div>(gdb) p odirect<br><br></div>and gdb will print its value for you in response.<br><br></div>-Krutika<br></div><div><br><div>On Mon, Aug 1, 2016 at 4:55 PM, Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br><blockquote style="border-left:1px #ccc solid;padding-left:1ex;">
<div><div dir="ltr">Hi,<div><br></div><div>How to get the results of the below variables ? i cant get the results from gdb.<br><br><br>
<div>-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br></font><font color="#3366ff"><br></font></div><br><br><div><hr>From: <a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a><br>Date: Mon, 1 Aug 2016 15:51:38 +0530<br>Subject: Re: [Gluster-users] Gluster 3.7.13 NFS Crash<br>To: <a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a><br>CC: <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><div><div><br><br><div dir="ltr"><div><div><div><div><div><div>Could you also print and share the values of the following variables from the backtrace please:<br><br></div>i. cur_block<br></div>ii. last_block<br></div><div>iii. local->first_block<br></div>iv. odirect<br></div>v. fd->flags<br></div>vi. local->call_count<br><br></div>-Krutika<br></div><div><br><div>On Sat, Jul 30, 2016 at 5:04 PM, Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br><blockquote style="border-left:1px #ccc solid;padding-left:1ex;">
<div><div dir="ltr"><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;">Hi,</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;">I really appreciate if someone can help me fix my nfs crash, its happening a lot and it's causing lots of issues to my VMs;</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;">the problem is every few hours the native nfs crash and the volume become unavailable from the affected node unless i restart glusterd.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;">the volume is used by vmware esxi as a datastore for it's VMs with the following options;</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;">OS: CentOS 7.2</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;">Gluster: 3.7.13</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;"><div>Volume Name: vlm01</div><div>Type: Distributed-Replicate</div><div>Volume ID: eacd8248-dca3-4530-9aed-7714a5<wbr>a114f2</div><div>Status: Started</div><div>Number of Bricks: 7 x 3 = 21</div><div>Transport-type: tcp</div><div>Bricks:</div><div>Brick1: gfs01:/bricks/b01/vlm01</div><div>Brick2: gfs02:/bricks/b01/vlm01</div><div>Brick3: gfs03:/bricks/b01/vlm01</div><div>Brick4: gfs01:/bricks/b02/vlm01</div><div>Brick5: gfs02:/bricks/b02/vlm01</div><div>Brick6: gfs03:/bricks/b02/vlm01</div><div>Brick7: gfs01:/bricks/b03/vlm01</div><div>Brick8: gfs02:/bricks/b03/vlm01</div><div>Brick9: gfs03:/bricks/b03/vlm01</div><div>Brick10: gfs01:/bricks/b04/vlm01</div><div>Brick11: gfs02:/bricks/b04/vlm01</div><div>Brick12: gfs03:/bricks/b04/vlm01</div><div>Brick13: gfs01:/bricks/b05/vlm01</div><div>Brick14: gfs02:/bricks/b05/vlm01</div><div>Brick15: gfs03:/bricks/b05/vlm01</div><div>Brick16: gfs01:/bricks/b06/vlm01</div><div>Brick17: gfs02:/bricks/b06/vlm01</div><div>Brick18: gfs03:/bricks/b06/vlm01</div><div>Brick19: gfs01:/bricks/b07/vlm01</div><div>Brick20: gfs02:/bricks/b07/vlm01</div><div>Brick21: gfs03:/bricks/b07/vlm01</div><div>Options Reconfigured:</div><div>performance.readdir-ahead: off</div><div>performance.quick-read: off</div><div>performance.read-ahead: off</div><div>performance.io-cache: off</div><div>performance.stat-prefetch: off</div><div>cluster.eager-lock: enable</div><div>network.remote-dio: enable</div><div>cluster.quorum-type: auto</div><div>cluster.server-quorum-type: server</div><div>performance.strict-write-order<wbr>ing: on</div><div>performance.write-behind: off</div><div>cluster.data-self-heal-algorit<wbr>hm: full</div><div>cluster.self-heal-window-size: 128</div><div>features.shard-block-size: 16MB</div><div>features.shard: on</div><div>auth.allow: 192.168.221.50,192.168.221.51,<wbr>192.168.221.52,192.168.221.56,<wbr>192.168.208.130,192.168.208.13<wbr>1,192.168.208.132,192.168.208.<wbr>89,192.168.208.85,192.168.208.<wbr>208.86</div><div>network.ping-timeout: 10</div></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;">latest bt;</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;"><div><br></div><div><br></div><div>(gdb) bt </div><div>#0 0x00007f196acab210 in pthread_spin_lock () from /lib64/libpthread.so.0</div><div>#1 0x00007f196be7bcd5 in fd_anonymous (inode=0x0) at fd.c:804</div><div>#2 0x00007f195deb1787 in shard_common_inode_write_do (frame=0x7f19699f1164, this=0x7f195802ac10) at shard.c:3716</div><div>#3 0x00007f195deb1a53 in shard_common_inode_write_post_<wbr>lookup_shards_handler (frame=<optimized out>, this=<optimized out>) at shard.c:3769</div><div>#4 0x00007f195deaaff5 in shard_common_lookup_shards_cbk (frame=0x7f19699f1164, cookie=<optimized out>, this=0x7f195802ac10, op_ret=0, </div><div> op_errno=<optimized out>, inode=<optimized out>, buf=0x7f194970bc40, xdata=0x7f196c15451c, postparent=0x7f194970bcb0) at shard.c:1601</div><div>#5 0x00007f195e10a141 in dht_lookup_cbk (frame=0x7f196998e7d4, cookie=<optimized out>, this=<optimized out>, op_ret=0, op_errno=0, inode=0x7f195c532b18, </div><div> stbuf=0x7f194970bc40, xattr=0x7f196c15451c, postparent=0x7f194970bcb0) at dht-common.c:2174</div><div>#6 0x00007f195e3931f3 in afr_lookup_done (frame=frame@entry=0x7f196997f<wbr>8a4, this=this@entry=0x7f1958022a20<wbr>) at afr-common.c:1825</div><div>#7 0x00007f195e393b84 in afr_lookup_metadata_heal_check (frame=frame@entry=0x7f196997f<wbr>8a4, this=0x7f1958022a20, this@entry=0xe3a929e0b67fa500)</div><div> at afr-common.c:2068</div><div>#8 0x00007f195e39434f in afr_lookup_entry_heal (frame=frame@entry=0x7f196997f<wbr>8a4, this=0xe3a929e0b67fa500, this@entry=0x7f1958022a20) at afr-common.c:2157</div><div>#9 0x00007f195e39467d in afr_lookup_cbk (frame=0x7f196997f8a4, cookie=<optimized out>, this=0x7f1958022a20, op_ret=<optimized out>, </div><div> op_errno=<optimized out>, inode=<optimized out>, buf=0x7f195effa940, xdata=0x7f196c1853b0, postparent=0x7f195effa9b0) at afr-common.c:2205</div><div>#10 0x00007f195e5e2e42 in client3_3_lookup_cbk (req=<optimized out>, iov=<optimized out>, count=<optimized out>, myframe=0x7f196999952c)</div><div> at client-rpc-fops.c:2981</div><div>#11 0x00007f196bc0ca30 in rpc_clnt_handle_reply (clnt=clnt@entry=0x7f19583adaf<wbr>0, pollin=pollin@entry=0x7f195907<wbr>f930) at rpc-clnt.c:764</div><div>#12 0x00007f196bc0ccef in rpc_clnt_notify (trans=<optimized out>, mydata=0x7f19583adb20, event=<optimized out>, data=0x7f195907f930) at rpc-clnt.c:925</div><div>#13 0x00007f196bc087c3 in rpc_transport_notify (this=this@entry=0x7f19583bd77<wbr>0, event=event@entry=RPC_TRANSPOR<wbr>T_MSG_RECEIVED, data=data@entry=0x7f195907f930<wbr>)</div><div> at rpc-transport.c:546</div><div>#14 0x00007f1960acf9a4 in socket_event_poll_in (this=this@entry=0x7f19583bd77<wbr>0) at socket.c:2353</div><div>#15 0x00007f1960ad25e4 in socket_event_handler (fd=fd@entry=25, idx=idx@entry=14, data=0x7f19583bd770, poll_in=1, poll_out=0, poll_err=0) at socket.c:2466</div><div>#16 0x00007f196beacf7a in event_dispatch_epoll_handler (event=0x7f195effae80, event_pool=0x7f196dbf5f20) at event-epoll.c:575</div><div>#17 event_dispatch_epoll_worker (data=0x7f196dc41e10) at event-epoll.c:678</div><div>#18 0x00007f196aca6dc5 in start_thread () from /lib64/libpthread.so.0</div><div>#19 0x00007f196a5ebced in clone () from /lib64/libc.so.6</div></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;"><br></div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:10pt;background-color:rgb(255,255,255);">nfs logs and the core dump can be found in the dropbox link below;</span><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;"><a href="https://db.tt/rZrC9d7f" target="_blank">https://db.tt/rZrC9d7f</a><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;"><br></div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;">thanks in advance.</div><div style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;"><br clear="all"><div><div><div dir="ltr"><font color="#3366ff"><font color="#000000">Respectfully<b><br></b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br><br></font></div></div></div></div> </div></div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a><br></blockquote></div><br></div></div></div></div></div> </div></div>
</blockquote></div><br></div></div></div> </div>
<br>______________________________<wbr>_________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a></div></div> </div>
<br>______________________________<wbr>_________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman<wbr>/listinfo/gluster-users</a></div></div></div></div> </div></div>
</blockquote></div><br></div></div></div></div></div> </div></div>
</blockquote></div><br></div></div></div></div></div> </div></div>
</blockquote></div><br></div></div></div> </div></body>
</html>