<div dir="ltr">Yes, you are right. I mixed up the logs. I just ran the add-brick command again after cleaning up fs4 and re-installing gluster. This is the complete fs4 data-brick.log.<div><br></div><div><div>[root@fs1 ~]# gluster volume add-brick cees-data fs4:/data/brick</div><div>volume add-brick: failed: Commit failed on fs4. Please check log file for details.</div><div><br></div><div><div>[root@fs4 bricks]# pwd</div><div>/var/log/glusterfs/bricks</div><div>[root@fs4 bricks]# cat data-brick.log </div><div>[2016-09-27 16:16:28.095661] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.7.14 (args: /usr/sbin/glusterfsd -s fs4 --volfile-id cees-data.fs4.data-brick -p /var/lib/glusterd/vols/cees-data/run/fs4-data-brick.pid -S /var/run/gluster/5203ab38be21e1d37c04f6bdfee77d4a.socket --brick-name /data/brick -l /var/log/glusterfs/bricks/data-brick.log --xlator-option *-posix.glusterd-uuid=f04b231e-63f8-4374-91ae-17c0c623f165 --brick-port 49152 --xlator-option cees-data-server.transport.rdma.listen-port=49153 --xlator-option cees-data-server.listen-port=49152 --volfile-server-transport=socket,rdma)</div><div>[2016-09-27 16:16:28.101547] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1</div><div>[2016-09-27 16:16:28.104637] I [graph.c:269:gf_add_cmdline_options] 0-cees-data-server: adding option 'listen-port' for volume 'cees-data-server' with value '49152'</div><div>[2016-09-27 16:16:28.104646] I [graph.c:269:gf_add_cmdline_options] 0-cees-data-server: adding option 'transport.rdma.listen-port' for volume 'cees-data-server' with value '49153'</div><div>[2016-09-27 16:16:28.104662] I [graph.c:269:gf_add_cmdline_options] 0-cees-data-posix: adding option 'glusterd-uuid' for volume 'cees-data-posix' with value 'f04b231e-63f8-4374-91ae-17c0c623f165'</div><div>[2016-09-27 16:16:28.104808] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/data/brick: skip format check for non-addr auth option auth.login./data/brick.allow</div><div>[2016-09-27 16:16:28.104814] I [MSGID: 115034] [server.c:403:_check_for_auth_option] 0-/data/brick: skip format check for non-addr auth option auth.login.18ddaf4c-ad98-4155-9372-717eae718b4c.password</div><div>[2016-09-27 16:16:28.104883] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2</div><div>[2016-09-27 16:16:28.105479] I [rpcsvc.c:2196:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64</div><div>[2016-09-27 16:16:28.105532] W [MSGID: 101002] [options.c:957:xl_opt_validate] 0-cees-data-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction</div><div>[2016-09-27 16:16:28.109456] W [socket.c:3665:reconfigure] 0-cees-data-quota: NBIO on -1 failed (Bad file descriptor)</div><div>[2016-09-27 16:16:28.489255] I [MSGID: 121050] [ctr-helper.c:259:extract_ctr_options] 0-gfdbdatastore: CTR Xlator is disabled.</div><div>[2016-09-27 16:16:28.489272] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-cees-data-changetimerecorder: Failed to retrieve sql-db-pagesize from params.Assigning default value: 4096</div><div>[2016-09-27 16:16:28.489278] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-cees-data-changetimerecorder: Failed to retrieve sql-db-journalmode from params.Assigning default value: wal</div><div>[2016-09-27 16:16:28.489284] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-cees-data-changetimerecorder: Failed to retrieve sql-db-sync from params.Assigning default value: off</div><div>[2016-09-27 16:16:28.489288] W [MSGID: 101105] [gfdb_sqlite3.h:239:gfdb_set_sql_params] 0-cees-data-changetimerecorder: Failed to retrieve sql-db-autovacuum from params.Assigning default value: none</div><div>[2016-09-27 16:16:28.490431] I [trash.c:2412:init] 0-cees-data-trash: no option specified for 'eliminate', using NULL</div><div>[2016-09-27 16:16:28.672814] W [graph.c:357:_log_if_unknown_option] 0-cees-data-server: option 'rpc-auth.auth-glusterfs' is not recognized</div><div>[2016-09-27 16:16:28.672854] W [graph.c:357:_log_if_unknown_option] 0-cees-data-server: option 'rpc-auth.auth-unix' is not recognized</div><div>[2016-09-27 16:16:28.672872] W [graph.c:357:_log_if_unknown_option] 0-cees-data-server: option 'rpc-auth.auth-null' is not recognized</div><div>[2016-09-27 16:16:28.672924] W [graph.c:357:_log_if_unknown_option] 0-cees-data-quota: option 'timeout' is not recognized</div><div>[2016-09-27 16:16:28.672955] W [graph.c:357:_log_if_unknown_option] 0-cees-data-trash: option 'brick-path' is not recognized</div><div>Final graph:</div><div>+------------------------------------------------------------------------------+</div><div> 1: volume cees-data-posix</div><div> 2: type storage/posix</div><div> 3: option glusterd-uuid f04b231e-63f8-4374-91ae-17c0c623f165</div><div> 4: option directory /data/brick</div><div> 5: option volume-id 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2</div><div> 6: option update-link-count-parent on</div><div> 7: end-volume</div><div> 8: </div><div> 9: volume cees-data-trash</div><div> 10: type features/trash</div><div> 11: option trash-dir .trashcan</div><div> 12: option brick-path /data/brick</div><div> 13: option trash-internal-op off</div><div> 14: subvolumes cees-data-posix</div><div> 15: end-volume</div><div> 16: </div><div> 17: volume cees-data-changetimerecorder</div><div> 18: type features/changetimerecorder</div><div> 19: option db-type sqlite3</div><div> 20: option hot-brick off</div><div> 21: option db-name brick.db</div><div> 22: option db-path /data/brick/.glusterfs/</div><div> 23: option record-exit off</div><div> 24: option ctr_link_consistency off</div><div> 25: option ctr_lookupheal_link_timeout 300</div><div> 26: option ctr_lookupheal_inode_timeout 300</div><div> 27: option record-entry on</div><div> 28: option ctr-enabled off</div><div> 29: option record-counters off</div><div> 30: option ctr-record-metadata-heat off</div><div> 31: option sql-db-cachesize 1000</div><div> 32: option sql-db-wal-autocheckpoint 1000</div><div> 33: subvolumes cees-data-trash</div><div> 34: end-volume</div><div> 35: </div><div> 36: volume cees-data-changelog</div><div> 37: type features/changelog</div><div> 38: option changelog-brick /data/brick</div><div> 39: option changelog-dir /data/brick/.glusterfs/changelogs</div><div> 40: option changelog-barrier-timeout 120</div><div> 41: subvolumes cees-data-changetimerecorder</div><div> 42: end-volume</div><div> 43: </div><div> 44: volume cees-data-bitrot-stub</div><div> 45: type features/bitrot-stub</div><div> 46: option export /data/brick</div><div> 47: subvolumes cees-data-changelog</div><div> 48: end-volume</div><div> 49: </div><div> 50: volume cees-data-access-control</div><div> 51: type features/access-control</div><div> 52: subvolumes cees-data-bitrot-stub</div><div> 53: end-volume</div><div> 54: </div><div> 55: volume cees-data-locks</div><div> 56: type features/locks</div><div> 57: subvolumes cees-data-access-control</div><div> 58: end-volume</div><div> 59: </div><div> 60: volume cees-data-upcall</div><div> 61: type features/upcall</div><div> 62: option cache-invalidation off</div><div> 63: subvolumes cees-data-locks</div><div> 64: end-volume</div><div> 65: </div><div> 66: volume cees-data-io-threads</div><div> 67: type performance/io-threads</div><div> 68: subvolumes cees-data-upcall</div><div> 69: end-volume</div><div> 70: </div><div> 71: volume cees-data-marker</div><div> 72: type features/marker</div><div> 73: option volume-uuid 27d2a59c-bdac-4f66-bcd8-e6124e53a4a2</div><div> 74: option timestamp-file /var/lib/glusterd/vols/cees-data/marker.tstamp</div><div> 75: option quota-version 1</div><div> 76: option xtime off</div><div> 77: option gsync-force-xtime off</div><div> 78: option quota on</div><div> 79: option inode-quota on</div><div> 80: subvolumes cees-data-io-threads</div><div> 81: end-volume</div><div> 82: </div><div> 83: volume cees-data-barrier</div><div> 84: type features/barrier</div><div> 85: option barrier disable</div><div> 86: option barrier-timeout 120</div><div> 87: subvolumes cees-data-marker</div><div> 88: end-volume</div><div> 89: </div><div> 90: volume cees-data-index</div><div> 91: type features/index</div><div> 92: option index-base /data/brick/.glusterfs/indices</div><div> 93: subvolumes cees-data-barrier</div><div> 94: end-volume</div><div> 95: </div><div> 96: volume cees-data-quota</div><div> 97: type features/quota</div><div> 98: option transport.socket.connect-path /var/run/gluster/quotad.socket</div><div> 99: option transport-type socket</div><div>100: option transport.address-family unix</div><div>101: option volume-uuid cees-data</div><div>102: option server-quota on</div><div>103: option timeout 0</div><div>104: option deem-statfs on</div><div>105: subvolumes cees-data-index</div><div>106: end-volume</div><div>107: </div><div>108: volume cees-data-worm</div><div>109: type features/worm</div><div>110: option worm off</div><div>111: subvolumes cees-data-quota</div><div>112: end-volume</div><div>113: </div><div>114: volume cees-data-read-only</div><div>115: type features/read-only</div><div>116: option read-only off</div><div>117: subvolumes cees-data-worm</div><div>118: end-volume</div><div>119: </div><div>120: volume /data/brick</div><div>121: type debug/io-stats</div><div>122: option log-level INFO</div><div>123: option latency-measurement off</div><div>124: option count-fop-hits off</div><div>125: subvolumes cees-data-read-only</div><div>126: end-volume</div><div>127: </div><div>128: volume cees-data-server</div><div>129: type protocol/server</div><div>130: option transport.socket.listen-port 49152</div><div>131: option rpc-auth.auth-glusterfs on</div><div>132: option rpc-auth.auth-unix on</div><div>133: option rpc-auth.auth-null on</div><div>134: option rpc-auth-allow-insecure on</div><div>135: option transport.rdma.listen-port 49153</div><div>136: option transport-type tcp,rdma</div><div>137: option auth.login./data/brick.allow 18ddaf4c-ad98-4155-9372-717eae718b4c</div><div>138: option auth.login.18ddaf4c-ad98-4155-9372-717eae718b4c.password 9e913e92-7de0-47f9-94ed-d08cbb130d23</div><div>139: option auth.addr./data/brick.allow *</div><div>140: subvolumes /data/brick</div><div>141: end-volume</div><div>142: </div><div>+------------------------------------------------------------------------------+</div><div>[2016-09-27 16:16:30.079541] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c</div><div>[2016-09-27 16:16:30.079567] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs3-12560-2016/09/27-16:16:30:47674-cees-data-client-3-0-0 (version: 3.7.14)</div><div>[2016-09-27 16:16:30.081487] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c</div><div>[2016-09-27 16:16:30.081505] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs2-11709-2016/09/27-16:16:30:50047-cees-data-client-3-0-0 (version: 3.7.14)</div><div>[2016-09-27 16:16:30.111091] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c</div><div>[2016-09-27 16:16:30.111113] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs2-11701-2016/09/27-16:16:29:24060-cees-data-client-3-0-0 (version: 3.7.14)</div><div>[2016-09-27 16:16:30.112822] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c</div><div>[2016-09-27 16:16:30.112836] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs3-12552-2016/09/27-16:16:29:23041-cees-data-client-3-0-0 (version: 3.7.14)</div><div>[2016-09-27 16:16:31.950978] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c</div><div>[2016-09-27 16:16:31.950998] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs1-6721-2016/09/27-16:16:26:939991-cees-data-client-3-0-0 (version: 3.7.14)</div><div>[2016-09-27 16:16:31.981977] I [login.c:81:gf_auth] 0-auth/login: allowed user names: 18ddaf4c-ad98-4155-9372-717eae718b4c</div><div>[2016-09-27 16:16:31.981994] I [MSGID: 115029] [server-handshake.c:690:server_setvolume] 0-cees-data-server: accepted client from fs1-6729-2016/09/27-16:16:27:971228-cees-data-client-3-0-0 (version: 3.7.14)</div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 27, 2016 at 8:46 AM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span class="">
<div>On 09/27/2016 09:06 PM, Dennis Michael
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Yes, the brick log
/var/log/glusterfs/bricks/<wbr>data-brick.log is created on fs4, and
the snippets showing the errors were from that log.<br>
<br>
</div>
</blockquote></span>
Unless I'm missing something, the snippet below is from glusterd's
log and not the brick's as is evident from the function names.<br>
-Ravi<span class=""><br>
<blockquote type="cite">
<div dir="ltr">
Dennis <br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Sep 26, 2016 at 5:58 PM,
Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On 09/27/2016 05:25 AM, Dennis Michael wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
[2016-09-26 22:44:39.254921] E [MSGID: 106005]
[glusterd-utils.c:4771:gluster<wbr>d_brick_start]
0-management: Unable to start brick fs4:/data/brick<br>
[2016-09-26 22:44:39.254949] E [MSGID: 106074]
[glusterd-brick-ops.c:2372:glu<wbr>sterd_op_add_brick]
0-glusterd: Unable to add bricks<br>
</blockquote>
<br>
</span>
Is the brick log created on fs4? Does it contain
warnings/errors?<br>
<br>
-Ravi<br>
<br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<p><br>
</p>
</span></div>
</blockquote></div><br></div>