<div dir="ltr">Hi,<div><br></div><div>Sorry for the delay, it took a long time to reproduce. But currently we have the same issue again. This was happened afrer reseting all nodes. Quorum is enabled. Logs and details below.</div><div><br></div><div><div>gluster> volume heal vm_storage_volume info split-brain</div><div>Gathering list of split brain entries on volume vm_storage_volume has been successful </div><div><br></div><div>Brick svm1:/srv/vol</div><div>Number of entries: 0</div><div><br></div><div>Brick svm2:/srv/vol</div><div>Number of entries: 0</div><div><br></div><div>Brick svm3:/srv/vol</div><div>Number of entries: 0</div><div><br></div><div>Brick svm4:/srv/vol</div><div>Number of entries: 0</div><div><br></div><div>Brick svm5:/srv/vol</div><div>Number of entries: 0</div><div><br></div><div>Brick svm6:/srv/vol</div><div>Number of entries: 2</div><div>at path on brick</div><div>-----------------------------------</div><div>2015-01-22 09:08:47 /vm_images_and_config/vm9.img</div><div>2015-01-22 09:11:52 /vm_images_and_config/vm20.img</div><div><br></div><div>Brick svm7:/srv/vol</div><div>Number of entries: 0</div></div><div><br></div><div><br></div><div><div>gluster> volume heal vm_storage_volume statistics</div><div>Gathering crawl statistics on volume vm_storage_volume has been successful </div><div>------------------------------------------------</div><div><br></div><div>Crawl statistics for brick no 0</div><div>Hostname of brick svm1</div><div><br></div><div>Starting time of crawl: Thu Jan 22 12:53:28 2015</div><div><br></div><div>Crawl is in progress</div><div>Type of crawl: INDEX</div><div>No. of entries healed: 1</div><div>No. of entries in split-brain: 0</div><div>No. of heal failed entries: 2</div><div>------------------------------------------------</div><div><br></div><div>Crawl statistics for brick no 1</div><div>Hostname of brick svm2</div><div><br></div><div>Starting time of crawl: Thu Jan 22 13:02:44 2015</div><div><br></div><div>Crawl is in progress</div><div>Type of crawl: INDEX</div><div>No. of entries healed: 1</div><div>No. of entries in split-brain: 0</div><div>No. of heal failed entries: 3</div><div>------------------------------------------------</div><div><br></div><div>Crawl statistics for brick no 2</div><div>Hostname of brick svm3</div><div><br></div><div>Starting time of crawl: Thu Jan 22 13:11:17 2015</div><div><br></div><div>Crawl is in progress</div><div>Type of crawl: INDEX</div><div>No. of entries healed: 0</div><div>No. of entries in split-brain: 0</div><div>No. of heal failed entries: 0</div><div>------------------------------------------------</div><div><br></div><div>Crawl statistics for brick no 3</div><div>Hostname of brick svm4</div><div><br></div><div>Starting time of crawl: Thu Jan 22 13:11:48 2015</div><div><br></div><div>Crawl is in progress</div><div>Type of crawl: INDEX</div><div>No. of entries healed: 0</div><div>No. of entries in split-brain: 0</div><div>No. of heal failed entries: 1</div><div>------------------------------------------------</div><div><br></div><div>Crawl statistics for brick no 4</div><div>Hostname of brick svm5</div><div><br></div><div>Starting time of crawl: Thu Jan 22 12:55:52 2015</div><div><br></div><div>Crawl is in progress</div><div>Type of crawl: INDEX</div><div>No. of entries healed: 0</div><div>No. of entries in split-brain: 0</div><div>No. of heal failed entries: 3</div><div>------------------------------------------------</div><div><br></div><div>Crawl statistics for brick no 5</div><div>Hostname of brick svm6</div><div><br></div><div>Starting time of crawl: Thu Jan 22 12:53:23 2015</div><div><br></div><div>Crawl is in progress</div><div>Type of crawl: INDEX</div><div>No. of entries healed: 0</div><div>No. of entries in split-brain: 2</div><div>No. of heal failed entries: 2</div><div>------------------------------------------------</div><div><br></div><div>Crawl statistics for brick no 6</div><div>Hostname of brick svm7</div><div><br></div><div>Starting time of crawl: Thu Jan 22 13:24:08 2015</div><div><br></div><div>Crawl is in progress</div><div>Type of crawl: INDEX</div><div>No. of entries healed: 0</div><div>No. of entries in split-brain: 0</div><div>No. of heal failed entries: 1</div></div><div><br></div><div><br></div><div><div>[2015-01-22 09:11:51.316542] I [rpc-clnt.c:1729:rpc_clnt_reconfig] 0-vm_storage_volume-client-3: changing port to 49216 (from 0)</div><div>[2015-01-22 09:11:51.317179] I [client-handshake.c:1677:select_server_supported_programs] 0-vm_storage_volume-client-3: Using Program GlusterFS 3.3, Num (1298437), Version (330)</div><div>[2015-01-22 09:11:51.317459] I [client-handshake.c:1462:client_setvolume_cbk] 0-vm_storage_volume-client-3: Connected to <a href="http://11.2.1.204:49216">11.2.1.204:49216</a>, attached to remote volume '/srv/vol'.</div><div>[2015-01-22 09:11:51.317493] I [client-handshake.c:1474:client_setvolume_cbk] 0-vm_storage_volume-client-3: Server and Client lk-version numbers are not same, reopening the fds</div><div>[2015-01-22 09:11:51.317528] I [client-handshake.c:1314:client_post_handshake] 0-vm_storage_volume-client-3: 1 fds open - Delaying child_up until they are re-opened</div><div>[2015-01-22 09:11:51.352698] I [client-handshake.c:936:client_child_up_reopen_done] 0-vm_storage_volume-client-3: last fd open'd/lock-self-heal'd - notifying CHILD-UP</div><div>[2015-01-22 09:11:51.355238] I [client-handshake.c:450:client_set_lk_version_cbk] 0-vm_storage_volume-client-3: Server lk version = 1</div><div>[2015-01-22 09:11:51.357918] I [afr-self-heald.c:1690:afr_dir_exclusive_crawl] 0-vm_storage_volume-replicate-0: Another crawl is in progress for vm_storage_volume-client-5</div><div>[2015-01-22 09:11:52.299413] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-vm_storage_volume-replicate-0: Unable to self-heal contents of '<gfid:f7b77d22-9606-4141-943c-b738aa2a21fc>' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 2451 650 0 2 1452 5405 ] [ 12 1 64 1 3 1453 3551 ] [ 0 0 0 0 0 0 0 ] [ 0 0 0 0 0 0 0 ] [ 11 2441 650 0 0 1452 5403 ] [ 12 2442 651 1 3 1 3953 ] [ 0 0 0 0 0 0 0 ] ]</div></div><div><br></div><div><br></div><div><div>[2015-01-22 09:08:47.105262] E [client-rpc-fops.c:1533:client3_3_inodelk_cbk] 0-vm_storage_volume-client-2: remote operation failed: Transport endpoint is not connected</div><div>[2015-01-22 09:08:47.105309] E [client-rpc-fops.c:1533:client3_3_inodelk_cbk] 0-vm_storage_volume-client-3: remote operation failed: Transport endpoint is not connected</div><div>[2015-01-22 09:08:47.105654] W [client-rpc-fops.c:4243:client3_3_flush] 0-vm_storage_volume-client-2: (00000000-0000-0000-0000-000000000000) remote_fd is -1. EBADFD</div><div>[2015-01-22 09:08:47.105686] E [afr-self-heal-data.c:97:afr_sh_data_flush_cbk] 0-vm_storage_volume-replicate-0: flush failed on <gfid:84b645df-774d-49ab-b7c6-4fd44318fd34> on subvolume vm_storage_volume-client-2: File descriptor in bad state</div><div>[2015-01-22 09:08:47.105713] W [client-rpc-fops.c:4243:client3_3_flush] 0-vm_storage_volume-client-3: (00000000-0000-0000-0000-000000000000) remote_fd is -1. EBADFD</div><div>[2015-01-22 09:08:47.105727] E [afr-self-heal-data.c:97:afr_sh_data_flush_cbk] 0-vm_storage_volume-replicate-0: flush failed on <gfid:84b645df-774d-49ab-b7c6-4fd44318fd34> on subvolume vm_storage_volume-client-3: File descriptor in bad state</div><div>[2015-01-22 09:08:47.110715] E [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-vm_storage_volume-replicate-0: Unable to self-heal contents of '<gfid:87673d99-7651-47b1-8239-afefe8e4f320>' (possible split-brain). Please delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0 2183 678 0 23 11 0 ] [ 9 0 189 0 0 0 0 ] [ 0 0 0 0 0 0 0 ] [ 0 0 0 0 0 0 0 ] [ 9 478 667 0 0 0 0 ] [ 9 478 667 0 0 0 0 ] [ 0 0 0 0 0 0 0 ] ]</div><div>[2015-01-22 09:09:00.042738] E [socket.c:2244:socket_connect_finish] 0-vm_storage_volume-client-3: connection to <a href="http://172.16.0.204:24007">172.16.0.204:24007</a> failed (No route to host)</div></div><div><br></div><div><br></div><div><div>gluster> volume heal vm_storage_volume info</div><div>*** glibc detected *** /usr/sbin/glfsheal: malloc(): memory corruption (fast): 0x00007f3964971f30 ***</div><div>======= Backtrace: =========</div><div>/lib/x86_64-linux-gnu/libc.so.6(+0x78db6)[0x7f395fa86db6]</div><div>/lib/x86_64-linux-gnu/libc.so.6(+0x7af81)[0x7f395fa88f81]</div><div>/lib/x86_64-linux-gnu/libc.so.6(__libc_calloc+0xd0)[0x7f395fa8ba10]</div><div>/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(__gf_calloc+0xbe)[0x7f396162002e]</div><div>/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(iobref_new+0x15)[0x7f39616226f5]</div><div>/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/protocol/client.so(client_submit_request+0x329)[0x7f395b712de9]</div><div>/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/protocol/client.so(client3_3_inodelk+0x387)[0x7f395b722297]</div><div>/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/protocol/client.so(client_inodelk+0x9b)[0x7f395b70e64b]</div><div>/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(+0x4967f)[0x7f395b4d167f]</div><div>/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(afr_unlock+0x81)[0x7f395b4d1831]</div><div>/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(afr_sh_data_unlock+0x6b)[0x7f395b4bb2ab]</div><div>/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(afr_sh_data_finish+0x9d)[0x7f395b4bb3fd]</div><div>/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(afr_sh_data_fxattrop_fstat_done+0x281)[0x7f395b4bcf21]</div><div>/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/cluster/replicate.so(afr_sh_data_fstat_cbk+0x183)[0x7f395b4bd1e3]</div><div>/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/protocol/client.so(client3_3_fstat_cbk+0x426)[0x7f395b72bcd6]</div><div>/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_handle_reply+0xa4)[0x7f3960b2cae4]</div><div>/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0xcd)[0x7f3960b2ce6d]</div><div>/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_transport_notify+0x23)[0x7f3960b29173]</div><div>/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/rpc-transport/socket.so(+0x85a4)[0x7f395c1b75a4]</div><div>/usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/rpc-transport/socket.so(+0xad9c)[0x7f395c1b9d9c]</div><div>/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x6935a)[0x7f396164935a]</div><div>/usr/lib/x86_64-linux-gnu/libgfapi.so.0(+0x75b4)[0x7f39609095b4]</div><div>/lib/x86_64-linux-gnu/libpthread.so.0(+0x7db4)[0x7f3960185db4]</div><div>/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f395fae64fd]</div><div>======= Memory map: ========</div><div>7f3950000000-7f3950021000 rw-p 00000000 00:00 0 </div><div>7f3950021000-7f3954000000 ---p 00000000 00:00 0 </div><div>7f3957bd4000-7f395a3ca000 rw-p 00000000 00:00 0 </div><div>7f395a3ca000-7f395a3e3000 r-xp 00000000 fc:00 5395225 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/debug/io-stats.so</div><div>7f395a3e3000-7f395a5e2000 ---p 00019000 fc:00 5395225 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/debug/io-stats.so</div><div>7f395a5e2000-7f395a5e3000 r--p 00018000 fc:00 5395225 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/debug/io-stats.so</div><div>7f395a5e3000-7f395a5e5000 rw-p 00019000 fc:00 5395225 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/debug/io-stats.so</div><div>7f395a5e5000-7f395a5f2000 r-xp 00000000 fc:00 5395204 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/md-cache.so</div><div>7f395a5f2000-7f395a7f2000 ---p 0000d000 fc:00 5395204 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/md-cache.so</div><div>7f395a7f2000-7f395a7f3000 r--p 0000d000 fc:00 5395204 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/md-cache.so</div><div>7f395a7f3000-7f395a7f4000 rw-p 0000e000 fc:00 5395204 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/md-cache.so</div><div>7f395a7f4000-7f395a7fb000 r-xp 00000000 fc:00 5395206 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/open-behind.so</div><div>7f395a7fb000-7f395a9fb000 ---p 00007000 fc:00 5395206 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/open-behind.so</div><div>7f395a9fb000-7f395a9fc000 r--p 00007000 fc:00 5395206 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/open-behind.so</div><div>7f395a9fc000-7f395a9fd000 rw-p 00008000 fc:00 5395206 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/open-behind.so</div><div>7f395a9fd000-7f395aa04000 r-xp 00000000 fc:00 5395210 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/quick-read.so</div><div>7f395aa04000-7f395ac03000 ---p 00007000 fc:00 5395210 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/quick-read.so</div><div>7f395ac03000-7f395ac04000 r--p 00006000 fc:00 5395210 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/quick-read.so</div><div>7f395ac04000-7f395ac05000 rw-p 00007000 fc:00 5395210 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/quick-read.so</div><div>7f395ac05000-7f395ac15000 r-xp 00000000 fc:00 5395213 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/io-cache.so</div><div>7f395ac15000-7f395ae15000 ---p 00010000 fc:00 5395213 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/io-cache.so</div><div>7f395ae15000-7f395ae16000 r--p 00010000 fc:00 5395213 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/io-cache.so</div><div>7f395ae16000-7f395ae18000 rw-p 00011000 fc:00 5395213 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/io-cache.so</div><div>7f395ae18000-7f395ae23000 r-xp 00000000 fc:00 5395209 /usr/lib/x86_64-linux-gnu/glusterfs/3.5.3/xlator/performance/read-ahead.soBrick svm1:/srv/vol/</div><div>/vm_images_and_config/vm17.img - Possibly undergoing heal</div><div>/vm_images_and_config/vm12.img - Possibly undergoing heal</div><div>/vm_images_and_config/vm1.img - Possibly undergoing heal</div><div>/vm_images_and_config/vm9.img - Possibly undergoing heal</div><div>/vm_images_and_config/vm20.img - Possibly undergoing heal</div><div>/vm_images_and_config/vm15.img - Possibly undergoing heal</div><div>/vm_images_and_config/vm5.img - Possibly undergoing heal</div><div>/vm_images_and_config/vm7.img - Possibly undergoing heal</div><div>/vm_images_and_config/vm13.img - Possibly undergoing heal</div><div>/users/admvs/.mozilla/firefox/admvs.default</div><div>/vm_images_and_config/vm2.img - Possibly undergoing heal</div><div>/vm_images_and_config/vm19.img - Possibly undergoing heal</div><div>/vm_images_and_config/vm-clone.img - Possibly undergoing heal</div><div>/vm_images_and_config/vm8.img - Possibly undergoing heal</div><div>/vm_images_and_config/vm18.img - Possibly undergoing heal</div><div>/vm_images_and_config/vm16.img - Possibly undergoing heal</div><div>/vm_images_and_config/vm6.img - Possibly undergoing heal</div></div><div><br></div><div><br></div><div>Best regards,</div><div>Alexey</div></div><div class="gmail_extra"><br><div class="gmail_quote">2014-12-28 5:51 GMT+03:00 Pranith Kumar Karampuri <span dir="ltr"><<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><div><div class="h5">
<br>
<div>On 12/25/2014 08:05 PM, Alexey wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi all,
<div><br>
</div>
<div>
<div>We are using glusterfs setup with a quorum turned on and
the configuration as the follows:</div>
<div><br>
</div>
<div>Nodes: 3</div>
<div>Type: Replicate</div>
<div>Number of Bricks: 1 x 3 = 3</div>
<div>cluster.quorum-type: fixed</div>
<div>cluster.quorum-count: 2</div>
<div>cluster.data-self-heal-algorithm: diff</div>
<div>cluster.server-quorum-ratio: 51%</div>
<div>glusterfs version: 3.5.3</div>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div dir="ltr">
<div>
<div><br>
</div>
<div>
<div>Despite on the quorum is turned on sometimes we are
still encounter a split-brain occurrence after shutting
down one node or all nodes together/</div>
<div><br>
</div>
</div>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>Is this is a normal behavior? What conditions could
lead to this and how to prevent split-brain occurence?</div>
</div>
</div>
</div>
</blockquote></div></div>
Could you please describe what is the kind of split-brain that
happened?<br>
<br>
Pranith<br>
<blockquote type="cite">
<div dir="ltr">
<div><br>
</div>
<div>BR,</div>
<div>Alexey</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</div>
</blockquote></div><br></div>