<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
But now I have strange issue:
<div class=""><br class="">
</div>
<div class="">After creating the geo-rep session and starting it (from nodeB):</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">[root@nodeB]# gluster vol geo-replication myvol slaveA::myvol status detail</div>
<div class=""><br class="">
</div>
<div class="">MASTER NODE MASTER VOL MASTER BRICK SLAVE STATUS CHECKPOINT STATUS CRAWL STATUS FILES SYNCD FILES PENDING BYTES PENDING DELETES PENDING FILES SKIPPED</div>
<div class="">-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------</div>
<div class="">nodeB myvol /export/raid/myvol slaveA::myvol Passive N/A N/A 0 0 0 0 0</div>
<div class="">nodeA myvol /export/raid/myvol slaveA::myvol Passive N/A N/A 0 0 0 0 0</div>
<div class="">nodeC myvol /export/raid/myvol slaveA::myvol Active N/A Hybrid Crawl 0 8191 0 0 0</div>
<div class=""><br class="">
</div>
<div class="">[root@nodeB]# gluster vol geo-replication myvol slaveA::myvol status detail</div>
<div class=""><br class="">
</div>
<div class="">MASTER NODE MASTER VOL MASTER BRICK SLAVE STATUS CHECKPOINT STATUS CRAWL STATUS FILES SYNCD FILES PENDING BYTES PENDING DELETES PENDING FILES SKIPPED</div>
<div class="">-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------</div>
<div class="">nodeB myvol /export/raid/myvol slaveA::myvol Passive N/A N/A 0 0 0 0 0</div>
<div class="">nodeA myvol /export/raid/myvol slaveA::myvol Active N/A Hybrid Crawl 0 8191 0 0 0</div>
<div class="">nodeC myvol /export/raid/myvol slaveA::myvol Passive N/A N/A 0 0 0 0 0</div>
<div class=""><br class="">
</div>
<div class="">[root@nodeB]# gluster vol geo-replication myvol slaveA::myvol status detail</div>
<div class=""><br class="">
</div>
<div class="">MASTER NODE MASTER VOL MASTER BRICK SLAVE STATUS CHECKPOINT STATUS CRAWL STATUS FILES SYNCD FILES PENDING BYTES PENDING DELETES PENDING FILES SKIPPED</div>
<div class="">-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------</div>
<div class="">nodeB myvol /export/raid/myvol slaveA::myvol Active N/A Hybrid Crawl 0 8191 0 0 0</div>
<div class="">nodeA myvol /export/raid/myvol slaveA::myvol Passive N/A N/A 0 0 0 0 0</div>
<div class="">nodeC myvol /export/raid/myvol slaveA::myvol Passive N/A N/A 0 0 0 0 0</div>
<div class=""><br class="">
</div>
<div class="">So.</div>
<div class=""><br class="">
</div>
<div class=""> 1/Why there is 3 masters nodes ??? nodeB should be the master node only</div>
<div class=""> 2/Why it kept changing turn by turn from active to passive?</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">Thanks</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class="">-- <br class="">
Cyril Peponnet </div>
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Feb 2, 2015, at 10:40 AM, PEPONNET, Cyril N (Cyril) <<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">cyril.peponnet@alcatel-lucent.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
For the record, after adding
<div class=""><br class="">
</div>
<div class="">operating-version=2</div>
<div class=""><br class="">
</div>
<div class="">on every nodes (ABC) AND slave node, the commands are working<br class="">
<div class="">-- <br class="">
Cyril Peponnet </div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Feb 2, 2015, at 9:46 AM, PEPONNET, Cyril N (Cyril) <<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">cyril.peponnet@alcatel-lucent.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class="">More informations here:</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
I update the state of the peer in the uid file located in /v/l/g/peers from state 10 to state 3 (as it is on other node) and now the node is in cluster.
<div class=""><br class="">
</div>
<div class="">gluster system:: execute gsec_create now create a proper file from master node with every node’s key in it.</div>
<div class=""><br class="">
</div>
<div class="">Now from there I try to create my georeplication between master nodeB and slaveA</div>
<div class=""><br class="">
</div>
<div class="">gluster vol geo myvol slave::myvol create push-pem force</div>
<div class=""><br class="">
</div>
<div class="">From slaveA I got this error message logs:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">[2015-02-02 17:19:04.754809] E [glusterd-geo-rep.c:1686:glusterd_op_stage_copy_file] 0-: Op Version not supported.</div>
<div class="">[2015-02-02 17:19:04.754890] E [glusterd-syncop.c:912:gd_stage_op_phase] 0-management: Staging of operation 'Volume Copy File' failed on localhost : One or more nodes do not support the required op version.</div>
<div class="">[2015-02-02 17:19:07.513547] E [glusterd-geo-rep.c:1620:glusterd_op_stage_sys_exec] 0-: Op Version not supported.</div>
<div class="">[2015-02-02 17:19:07.513632] E [glusterd-geo-rep.c:1658:glusterd_op_stage_sys_exec] 0-: One or more nodes do not support the required op version.</div>
<div class="">[2015-02-02 17:19:07.513660] E [glusterd-syncop.c:912:gd_stage_op_phase] 0-management: Staging of operation 'Volume Execute system commands' failed on localhost : One or more nodes do not support the required op version.</div>
</div>
<div class=""><br class="">
</div>
<div class="">On slaveA I have the common pem file transfered in /v/l/g/geo/ with my 3 nodes from source site.</div>
<div class=""><br class="">
</div>
<div class="">But the /root/.ssh/authorized_keys is not populated with this file.</div>
<div class=""><br class="">
</div>
<div class="">From the log I saw that there is a call to a script</div>
<div class=""><br class="">
</div>
<div class="">/var/lib/glusterd/hooks/1/gsync-create/post/S56glusterd-geo-rep-create-post.sh —volname=myvol is_push_pem=1 pub_file=/var/lib/glusterd/geo-replication/common_secret.pem.pub slave_ip=salveA</div>
<div class=""><br class="">
</div>
<div class="">In this script the following is done:</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">```</div>
<div class="">
<div class=""> scp $pub_file $slave_ip:$pub_file_tmp</div>
<div class=""> ssh $slave_ip "mv $pub_file_tmp $pub_file"</div>
<div class=""> ssh $slave_ip "gluster system:: copy file /geo-replication/common_secret.pem.pub > /dev/null"</div>
<div class=""> ssh $slave_ip “gluster system:: execute add_secret_pub > /dev/null"</div>
</div>
<div class="">```</div>
<div class=""><br class="">
</div>
<div class="">The first 2 lines passed, the third fail so the fourth is never executed.</div>
<div class=""><br class="">
</div>
<div class="">Third command on slaveA</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">#gluster system:: copy file /geo-replication/common_secret.pem.pub</div>
<div class="">One or more nodes do not support the required op version.</div>
<div class=""><br class="">
</div>
<div class=""># gluster peer status</div>
<div class="">Number of Peers: 0</div>
</div>
<div class=""><br class="">
</div>
<div class="">from logs:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">==> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log <==</div>
<div class="">[2015-02-02 17:43:29.242524] E [glusterd-geo-rep.c:1686:glusterd_op_stage_copy_file] 0-: Op Version not supported.</div>
<div class="">[2015-02-02 17:43:29.242610] E [glusterd-syncop.c:912:gd_stage_op_phase] 0-management: Staging of operation 'Volume Copy File' failed on localhost : One or more nodes do not support the required op version.</div>
<div class="">One or more nodes do not support the required op version.</div>
</div>
<div class=""><br class="">
</div>
<div class="">I have for now only one node on my remote site.</div>
<div class=""><br class="">
</div>
<div class="">Any way, as this step is done to copy the file accros all the cluster member I can deal without</div>
<div class=""><br class="">
</div>
<div class="">The fourth command is not working:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">[root@slaveA geo-replication]# gluster system:: execute add_secret_pub</div>
<div class="">[2015-02-02 17:44:49.123326] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled</div>
<div class="">[2015-02-02 17:44:49.123381] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread</div>
<div class="">[2015-02-02 17:44:49.123568] I [socket.c:3561:socket_init] 0-glusterfs: SSL support is NOT enabled</div>
<div class="">[2015-02-02 17:44:49.123588] I [socket.c:3576:socket_init] 0-glusterfs: using system polling thread</div>
<div class="">[2015-02-02 17:44:49.306482] I [socket.c:2238:socket_event_handler] 0-transport: disconnecting now</div>
<div class=""><br class="">
</div>
<div class="">==> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log <==</div>
<div class="">[2015-02-02 17:44:49.307921] E [glusterd-geo-rep.c:1620:glusterd_op_stage_sys_exec] 0-: Op Version not supported.</div>
<div class="">[2015-02-02 17:44:49.308009] E [glusterd-geo-rep.c:1658:glusterd_op_stage_sys_exec] 0-: One or more nodes do not support the required op version.</div>
<div class="">[2015-02-02 17:44:49.308038] E [glusterd-syncop.c:912:gd_stage_op_phase] 0-management: Staging of operation 'Volume Execute system commands' failed on localhost : One or more nodes do not support the required op version.</div>
<div class="">One or more nodes do not support the required op version.</div>
<div class=""><br class="">
</div>
<div class="">==> /var/log/glusterfs/cli.log <==</div>
<div class="">[2015-02-02 17:44:49.308493] I [input.c:36:cli_batch] 0-: Exiting with: -1</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">I have only one node… I don’t understand the meaning of the errror: One or more nodes do not support the required op version.</div>
<div class=""><br class="">
<div apple-content-edited="true" class=""> -- <br class="">
Cyril Peponnet </div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Feb 2, 2015, at 8:49 AM, PEPONNET, Cyril N (Cyril) <<a href="mailto:cyril.peponnet@alcatel-lucent.com" class="">cyril.peponnet@alcatel-lucent.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div class="">Every node is connected:</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">root@nodeA geo-replication]# gluster peer status</div>
<div class="">Number of Peers: 2</div>
<div class=""><br class="">
</div>
<div class="">Hostname: nodeB</div>
<div class="">Uuid: 6a9da7fc-70ec-4302-8152-0e61929a7c8b</div>
<div class="">State: Peer in Cluster (Connected)</div>
<div class=""><br class="">
</div>
<div class="">Hostname: nodeC</div>
<div class="">Uuid: c12353b5-f41a-4911-9329-fee6a8d529de</div>
<div class="">State: Peer in Cluster (Connected)</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">[root@nodeB ~]# gluster peer status</div>
<div class="">Number of Peers: 2</div>
<div class=""><br class="">
</div>
<div class="">Hostname: nodeC</div>
<div class="">Uuid: c12353b5-f41a-4911-9329-fee6a8d529de</div>
<div class="">State: Peer in Cluster (Connected)</div>
<div class=""><br class="">
</div>
<div class="">Hostname: nodeA</div>
<div class="">Uuid: 2ac172bb-a2d0-44f1-9e09-6b054dbf8980</div>
<div class=""><b class="">State: Peer is connected and Accepted (Connected)</b></div>
</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">[root@nodeC geo-replication]# gluster peer status</div>
<div class="">Number of Peers: 2</div>
<div class=""><br class="">
</div>
<div class="">Hostname: nodeA</div>
<div class="">Uuid: 2ac172bb-a2d0-44f1-9e09-6b054dbf8980</div>
<div class="">State: Peer in Cluster (Connected)</div>
<div class=""><br class="">
</div>
<div class="">Hostname: nodeB</div>
<div class="">Uuid: 6a9da7fc-70ec-4302-8152-0e61929a7c8b</div>
<div class="">State: Peer in Cluster (Connected)</div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">The only difference is <b class="">State: Peer is connected and Accepted (Connected)
</b> from nodeB about nodeA</div>
<div class=""><br class="">
</div>
<div class="">When I execute gluster system from node A or C, I have the 3 nodes keys in common pem file. But from nodeB, I only have keys for nodeB and node C. This is infortunate as I try to launch the georeplication job from nodeB (master).</div>
<div class=""><br class="">
</div>
<br class="">
<div class="">-- <br class="">
Cyril Peponnet </div>
<br class="">
<div class="">
<blockquote type="cite" class="">
<div class="">On Feb 2, 2015, at 2:07 AM, Aravinda <<a href="mailto:avishwan@redhat.com" class="">avishwan@redhat.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Looks
like node C is in diconnected state. Please let us know the output of `gluster peer status` from all the master nodes and slave nodes.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">--</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">regards</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Aravinda</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">On
01/22/2015 12:27 AM, PEPONNET, Cyril N (Cyril) wrote:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
So,<br class="">
<br class="">
On master node of my 3 node setup:<br class="">
<br class="">
1) gluster system:: execute gsec_create<br class="">
<br class="">
in /var/lib/glusterd/geo-replication/common_secret.pub I have pem pub key from master node A and node B (not node C).<br class="">
<br class="">
On node C in don’t have anything in /v/l/g/geo/ except the gsync template config.<br class="">
<br class="">
So here I have an issue.<br class="">
<br class="">
The only error I saw on node C is:<br class="">
<br class="">
[2015-01-21 18:36:41.179601] E [rpc-clnt.c:208:call_bail]<br class="">
0-management: bailing out frame type(Peer mgmt) op(—(2)) xid =<br class="">
0x23 sent = 2015-01-21 18:26:33.031937. timeout = 600 for<br class="">
xx.xx.xx.xx:24007<br class="">
<br class="">
<br class="">
On node A, the cli.log looks like:<br class="">
<br class="">
[2015-01-21 18:49:49.878905] I [socket.c:3561:socket_init]<br class="">
0-glusterfs: SSL support is NOT enabled<br class="">
[2015-01-21 18:49:49.878947] I [socket.c:3576:socket_init]<br class="">
0-glusterfs: using system polling thread<br class="">
[2015-01-21 18:49:49.879085] I [socket.c:3561:socket_init]<br class="">
0-glusterfs: SSL support is NOT enabled<br class="">
[2015-01-21 18:49:49.879095] I [socket.c:3576:socket_init]<br class="">
0-glusterfs: using system polling thread<br class="">
[2015-01-21 18:49:49.951835] I<br class="">
[socket.c:2238:socket_event_handler] 0-transport: disconnecting now<br class="">
[2015-01-21 18:49:49.972143] I [input.c:36:cli_batch] 0-: Exiting<br class="">
with: 0<br class="">
<br class="">
<br class="">
If I run gluster system:: execute gsec_create on node C or node B, the common pem key file contains my 3 nodes pem puk keys. So in some way node A is unable to get the key from node C.<br class="">
<br class="">
So let’s try to fix this one before going further.<br class="">
<br class="">
--<br class="">
Cyril Peponnet<br class="">
<br class="">
<blockquote type="cite" class="">On Jan 20, 2015, at 9:38 PM, Aravinda <<a href="mailto:avishwan@redhat.com" class="">avishwan@redhat.com</a><span class="Apple-converted-space"> </span><<a href="mailto:avishwan@redhat.com" class="">mailto:avishwan@redhat.com</a>>>
wrote:<br class="">
<br class="">
On 01/20/2015 11:01 PM, PEPONNET, Cyril N (Cyril) wrote:<br class="">
<blockquote type="cite" class="">Hi,<br class="">
<br class="">
I’m ready for new testing, I delete the geo-rep session between master and slace, remove the lines in authorized keys file on slave.<br class="">
I also remove the common secret pem from slave, and from master. There is only the gsyncd_template.conf in /var/lib/gluster now.<br class="">
<br class="">
Here is our setup:<br class="">
<br class="">
Site A: gluster 3 nodes<br class="">
Site B: gluster 1 node (for now, a second will come).<br class="">
<br class="">
I can issue<br class="">
<br class="">
gluster systen:: execute gsec_create<br class="">
<br class="">
what to check?<br class="">
</blockquote>
common_secret.pem.pub is created in /var/lib/glusterd/geo-replication/common_secret.pem.pub, which should contain public keys from all Master nodes(Site A). Should match with contents of /var/lib/glusterd/geo-replication/secret.pem.pub and /var/lib/glusterd/geo-replication/tar_ssh.pem.pub.<br class="">
<br class="">
<blockquote type="cite" class=""><br class="">
then<br class="">
<br class="">
gluster geo vol geo_test slave::geo_test create push-pem force (force is needed because the slave vol is smaller than the master vol).<br class="">
<br class="">
What to check ?<br class="">
</blockquote>
Check for any errors in, /var/log/glusterfs/etc-glusterfs-glusterd.vol.log in rpm installation or in /var/log/glusterfs/usr-local-etc-glusterfs-glusterd.vol.log if source installation. In case of any errors related to hook execution, run directly the hook command
copied from the log. From your previous mail I understand their is some issue while executing hook script. I will look into the issue in hook script.<br class="">
<blockquote type="cite" class=""><br class="">
I want to use change_detector changelog and not rsync btw.<br class="">
</blockquote>
change_detector is crawling mecanism. Available options are: changelog and xsync. xsync is FS Crawl.<br class="">
sync mecanisms available are: rsync and tarssh.<br class="">
<blockquote type="cite" class=""><br class="">
Can you guide me to setup this but also debug why it’s not working out of the box ?<br class="">
<br class="">
If needed I can get in touch with you through IRC.<br class="">
</blockquote>
Sure. IRC nickname is aravindavk.<br class="">
<blockquote type="cite" class=""><br class="">
Thanks for your help.<br class="">
<br class="">
</blockquote>
<br class="">
--<br class="">
regards<br class="">
Aravinda<br class="">
<a href="http://aravindavk.in/" class="">http://aravindavk.in</a></blockquote>
</blockquote>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>