<p dir="ltr">Hi,</p>
<p dir="ltr">I am still getting the same issue, and looking further into this, I have modified syncdutils.py slightly on the slave node to print more information on the nature of this OSError:</p>
<p dir="ltr">[2015-09-30 12:19:25.8531] I [gsyncd(slave):649:main_i] <top>: syncing: gluster://localhost:homegs<br>
[2015-09-30 12:19:26.79925] I [resource(slave):844:service_loop] GLUSTER: slave listening<br>
[2015-09-30 12:19:30.723542] W [syncdutils(slave):481:errno_wrap] <top>: about to raise exception for <built-in function rename>(['.gfid/a780729e-3602-4c30-9966-f31fa5964843/CMakeCache.txt.tmp', '.gfid/a780729e-3602-4c30-9966-f31fa5964843/CMakeCache.txt']) from /...[Errno 16] Device or resource busy<br>
[2015-09-30 12:19:30.723845] I [syncdutils(slave):483:errno_wrap] <top>: we could try deleting '.glusterfs/a7/80/a780729e-3602-4c30-9966-f31fa5964843/CMakeCache.txt.tmp'<br>
[2015-09-30 12:19:30.723974] E [repce(slave):117:worker] <top>: call failed:<br>
Traceback (most recent call last):<br>
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 113, in worker<br>
res = getattr(self.obj, rmeth)(*in_data[2:])<br>
File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 725, in entry_ops<br>
[ENOENT, EEXIST], [ESTALE])<br>
File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 475, in errno_wrap<br>
return call(*arg)<br>
OSError: [Errno 16] Device or resource busy<br>
[2015-09-30 12:19:30.770679] I [repce(slave):92:service_loop] RepceServer: terminating on reaching EOF.<br>
[2015-09-30 12:19:30.770944] I [syncdutils(slave):220:finalize] <top>: exiting.</p>
<p dir="ltr">Below is a list of errors:</p>
<p dir="ltr">W [syncdutils(slave):481:errno_wrap] <top>: about to raise exception for <built-in function rename>(['.gfid/93b6282a-a47f-40fb-b83e-efec5b9c1a4e/.viminfo.tmp', '.gfid/93b6282a-a47f-40fb-b83e-efec5b9c1a4e/.viminfo']) from /...[Errno 16] Device or resource busy<br>
W [syncdutils(slave):481:errno_wrap] <top>: about to raise exception for <built-in function rename>(['.gfid/9c4f77d0-0855-4607-8d65-1a52b3cdbbce/gr_rsync_homes_new.sh', '.gfid/9c4f77d0-0855-4607-8d65-1a52b3cdbbce/gr_rsync_homes_new.sh~']) from /...[Errno 16] Device or resource busy<br>
W [syncdutils(slave):481:errno_wrap] <top>: about to raise exception for <built-in function rename>(['.gfid/a780729e-3602-4c30-9966-f31fa5964843/CMakeCache.txt.tmp', '.gfid/a780729e-3602-4c30-9966-f31fa5964843/CMakeCache.txt']) from /...[Errno 16] Device or resource busy<br>
W [syncdutils(slave):481:errno_wrap] <top>: about to raise exception for <built-in function rename>(['.gfid/a78e4abc-1de4-4f0d-a5e2-1244a0e3b708/depend.internal.tmp', '.gfid/a78e4abc-1de4-4f0d-a5e2-1244a0e3b708/depend.internal']) from /...[Errno 16] Device or resource busy<br>
W [syncdutils(slave):481:errno_wrap] <top>: about to raise exception for <built-in function rename>(['.gfid/f5491794-32e2-46f4-8c50-7ca31e34111f/<a href="http://gluster-volume-info.debug.pl/">gluster-volume-info.debug.pl</a>', '.gfid/f5491794-32e2-46f4-8c50-7ca31e34111f/<a href="http://gluster-volume-info.debug.pl/">gluster-volume-info.debug.pl</a>~'])...[Errno 16] Device or resource busy<br>
W [syncdutils(slave):481:errno_wrap] <top>: about to raise exception for <built-in function rename>(['.gfid/f5491794-32e2-46f4-8c50-7ca31e34111f/<a href="http://gluster-volume-info.debug.pl/">gluster-volume-info.debug.pl</a>', '.gfid/f5491794-32e2-46f4-8c50-7ca31e34111f/<a href="http://gluster-volume-info.debug.pl/">gluster-volume-info.debug.pl</a>~']) from /...[Errno 16] Device or resource busy</p>
<p dir="ltr">I am not entirely clear why rename() would return this ‘Device or resource busy’. Also I’m not sure what are these ‘.gfid’ paths exactly: is this something transient? Where does it live exactly? </p>
<p dir="ltr">I tried deleting the affected files (e.g. a78e4abc-1de4-4f0d-a5e2-1244a0e3b708/depend.internal.tmp) in .glusterfs/, and so far that allowed the geo-replication to resume… until it hits another one of these.</p>
<p dir="ltr">In some cases I found that this gfid points to a directory that doesn’t exist, e.g.:</p>
<p dir="ltr">ls-alh /gluster/homegs-brick-1/brick/.glusterfs/a7/8e/a78e4abc-1de4-4f0d-a5e2-1244a0e3b708</p>
<p dir="ltr">lrwxrwxrwx 1 root root 71 Sep 30 11:26 /gluster/homegs-brick-1/brick/.glusterfs/a7/8e/a78e4abc-1de4-4f0d-a5e2-1244a0e3b708 -> ../../0d/be/0dbe0730-5ba9-45d5-91e5-aeb069ab08bf/watchdog_unit_test.dir</p>
<p dir="ltr">but the link is broken.</p>
<p dir="ltr">In that case deleting the /gluster/homegs-brick-1/brick/.glusterfs/a7/8e/a78e4abc-1de4-4f0d-a5e2-1244a0e3b708 allowed the geo-replication to carry on.</p>
<p dir="ltr">Obviously it doesn’t work if I have to do this 50 times a day… </p>
<p dir="ltr">Any idea what could cause this and how to fix it properly?</p>
<p dir="ltr">Thx,<br>
Thib.</p>
<div class="gmail_quote">On 26 Aug 2015 12:58 pm, "Thibault Godouet" <<a href="mailto:tibo92@godouet.net">tibo92@godouet.net</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Hi,</p>
<p dir="ltr">Given that this is an 'OSError', I should probably have said we use Redhat 6.6 64bit and XFS as the brick filesystem.</p>
<p dir="ltr">Has anyone any input on how to troubleshoot this?</p>
<p dir="ltr">Thanks,<br>
Thibault.</p>
<div class="gmail_quote">On 24 Aug 2015 4:03 pm, "Thibault Godouet" <<a href="mailto:Thibault.Godouet@gresearch.co.uk" target="_blank">Thibault.Godouet@gresearch.co.uk</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
<p></p>
<p>I have had multiple issues with geo-replication. It seems to work OK initially, the replica gets up to date, and not long after (e.g. a couple of days), the replication goes into a faulty state and won't get out of it.</p>
<p><br>
I have tried a few times now, and last attempt I re-created the slave volume and setup the replication again. Same symptoms again.</p>
<p> </p>
<p>I use Gluster 3.7.3, and you will find my setup and log messages at the bottom of the email.</p>
<p> </p>
Any idea what could cause this and how to fix it?
<p></p>
<p>Thanks,<br>
Thibault.<br>
</p>
<p>ps: my setup and log messages:</p>
<p> </p>
<p>Master:</p>
<p> </p>
<p>Volume Name: home<br>
Type: Replicate<br>
Volume ID: 2299a204-a1dc-449d-8556-bc65197373c7<br>
Status: Started<br>
Number of Bricks: 1 x 2 = 2<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: server4.uberit.net:/gluster/home-brick-1<br>
Brick2: server5.uberit.net:/gluster/home-brick-1<br>
Options Reconfigured:<br>
performance.readdir-ahead: on<br>
geo-replication.indexing: on<br>
geo-replication.ignore-pid-check: on<br>
changelog.changelog: on</p>
<p> </p>
<p>Slave: </p>
<p>Volume Name: homegs<br>
Type: Distribute<br>
Volume ID: 746dfdc3-650d-4468-9fdd-d621dd215b94<br>
Status: Started<br>
Number of Bricks: 1<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: remoteserver1.uberit.net:/gluster/homegs-brick-1/brick<br>
Options Reconfigured:<br>
performance.readdir-ahead: on</p>
<p> </p>
<p>The geo-replication status and config (I think I ended up with only defaults values) are:</p>
<p> </p>
<p># gluster volume geo-replication home ssh://remoteserver1::homegs status</p>
<p>MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED<br>
---------------------------------------------------------------------------------------------------------------------------------------------------------<br>
server5 home /gluster/home-brick-1 root ssh://remoteserver1::homegs N/A Faulty N/A N/A<br>
server4 home /gluster/home-brick-1 root ssh://remoteserver1::homegs N/A Faulty N/A N/A</p>
<p> </p>
<p># gluster volume geo-replication home ssh://remoteserver1::homegs config<br>
special_sync_mode: partial<br>
state_socket_unencoded: /var/lib/glusterd/geo-replication/home_remoteserver1_homegs/ssh%3A%2F%2Froot%40IPADDRESS%3Agluster%3A%2F%2F127.0.0.1%3Ahomegs.socket<br>
gluster_log_file: /var/log/glusterfs/geo-replication/home/ssh%3A%2F%2Froot%40IPADDRESS%3Agluster%3A%2F%2F127.0.0.1%3Ahomegs.gluster.log<br>
ssh_command: ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem<br>
ignore_deletes: false<br>
change_detector: changelog<br>
gluster_command_dir: /usr/sbin/<br>
georep_session_working_dir: /var/lib/glusterd/geo-replication/home_remoteserver1_homegs/<br>
state_file: /var/lib/glusterd/geo-replication/home_remoteserver1_homegs/ssh%3A%2F%2Froot%40IPADDRESS%3Agluster%3A%2F%2F127.0.0.1%3Ahomegs.status<br>
remote_gsyncd: /nonexistent/gsyncd<br>
session_owner: 2299a204-a1dc-449d-8556-bc65197373c7<br>
changelog_log_file: /var/log/glusterfs/geo-replication/home/ssh%3A%2F%2Froot%40IPADDRESS%3Agluster%3A%2F%2F127.0.0.1%3Ahomegs-changes.log<br>
socketdir: /var/run/gluster<br>
working_dir: /var/lib/misc/glusterfsd/home/ssh%3A%2F%2Froot%40IPADDRESS%3Agluster%3A%2F%2F127.0.0.1%3Ahomegs<br>
state_detail_file: /var/lib/glusterd/geo-replication/home_remoteserver1_homegs/ssh%3A%2F%2Froot%40IPADDRESS%3Agluster%3A%2F%2F127.0.0.1%3Ahomegs-detail.status<br>
ssh_command_tar: ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/tar_ssh.pem<br>
pid_file: /var/lib/glusterd/geo-replication/home_remoteserver_homegs/ssh%3A%2F%2Froot%40IPADDRESS%3Agluster%3A%2F%2F127.0.0.1%3Ahomegs.pid<br>
log_file: /var/log/glusterfs/geo-replication/home/ssh%3A%2F%2Froot%40IPADDRESS%3Agluster%3A%2F%2F127.0.0.1%3Ahomegs.log<br>
gluster_params: aux-gfid-mount acl<br>
volume_id: 2299a204-a1dc-449d-8556-bc65197373c7</p>
<p> </p>
<p>The logs look like on the master on server1:</p>
<p> </p>
<p>[2015-08-24 15:21:07.955600] I [monitor(monitor):221:monitor] Monitor: ------------------------------------------------------------<br>
[2015-08-24 15:21:07.955883] I [monitor(monitor):222:monitor] Monitor: starting gsyncd worker<br>
[2015-08-24 15:21:08.69528] I [gsyncd(/gluster/home-brick-1):649:main_i] <top>: syncing: gluster://localhost:home -> ssh://root@vivlinuxinfra1.uberit.net:gluster://localhost:homegs<br>
[2015-08-24 15:21:08.70938] I [changelogagent(agent):75:__init__] ChangelogAgent: Agent listining...<br>
[2015-08-24 15:21:11.255237] I [master(/gluster/home-brick-1):83:gmaster_builder] <top>: setting up xsync change detection mode<br>
[2015-08-24 15:21:11.255532] I [master(/gluster/home-brick-1):404:__init__] _GMaster: using 'rsync' as the sync engine<br>
[2015-08-24 15:21:11.256570] I [master(/gluster/home-brick-1):83:gmaster_builder] <top>: setting up changelog change detection mode<br>
[2015-08-24 15:21:11.256726] I [master(/gluster/home-brick-1):404:__init__] _GMaster: using 'rsync' as the sync engine<br>
[2015-08-24 15:21:11.257345] I [master(/gluster/home-brick-1):83:gmaster_builder] <top>: setting up changeloghistory change detection mode<br>
[2015-08-24 15:21:11.257534] I [master(/gluster/home-brick-1):404:__init__] _GMaster: using 'rsync' as the sync engine<br>
[2015-08-24 15:21:13.333628] I [master(/gluster/home-brick-1):1212:register] _GMaster: xsync temp directory: /var/lib/misc/glusterfsd/home/ssh%3A%2F%2Froot%40172.18.0.169%3Agluster%3A%2F%2F127.0.0.1%3Ahomegs/62d98e8cc00a34eb85b4fe6d6fd3ba33/xsync<br>
[2015-08-24 15:21:13.333870] I [resource(/gluster/home-brick-1):1432:service_loop] GLUSTER: Register time: 1440426073<br>
[2015-08-24 15:21:13.401132] I [master(/gluster/home-brick-1):523:crawlwrap] _GMaster: primary master with volume id 2299a204-a1dc-449d-8556-bc65197373c7 ...<br>
[2015-08-24 15:21:13.412795] I [master(/gluster/home-brick-1):532:crawlwrap] _GMaster: crawl interval: 1 seconds<br>
[2015-08-24 15:21:13.427340] I [master(/gluster/home-brick-1):1127:crawl] _GMaster: starting history crawl... turns: 1, stime: (1440411353, 0)<br>
[2015-08-24 15:21:14.432327] I [master(/gluster/home-brick-1):1156:crawl] _GMaster: slave's time: (1440411353, 0)<br>
[2015-08-24 15:21:14.890889] E [repce(/gluster/home-brick-1):207:__call__] RepceClient: call 20960:140215190427392:1440426074.56 (entry_ops) failed on peer with OSError<br>
[2015-08-24 15:21:14.891124] E [syncdutils(/gluster/home-brick-1):276:log_raise_exception] <top>: FAIL:<br>
Traceback (most recent call last):<br>
File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 165, in main<br>
main_i()<br>
File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 659, in main_i<br>
local.service_loop(*[r for r in [remote] if r])<br>
File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1438, in service_loop<br>
g3.crawlwrap(oneshot=True)<br>
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 584, in crawlwrap<br>
self.crawl()<br>
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1165, in crawl<br>
self.changelogs_batch_process(changes)<br>
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1074, in changelogs_batch_process<br>
self.process(batch)<br>
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 952, in process<br>
self.process_change(change, done, retry)<br>
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 907, in process_change<br>
failures = self.slave.server.entry_ops(entries)<br>
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 226, in __call__<br>
return self.ins(self.meth, *a)<br>
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 208, in __call__<br>
raise res<br>
OSError: [Errno 5] Input/output error<br>
[2015-08-24 15:21:14.892291] I [syncdutils(/gluster/home-brick-1):220:finalize] <top>: exiting.<br>
[2015-08-24 15:21:14.893665] I [repce(agent):92:service_loop] RepceServer: terminating on reaching EOF.<br>
[2015-08-24 15:21:14.893879] I [syncdutils(agent):220:finalize] <top>: exiting.<br>
[2015-08-24 15:21:15.259360] I [monitor(monitor):282:monitor] Monitor: worker(/gluster/home-brick-1) died in startup phase</p>
<p> </p>
<p>and on master server2:</p>
<p> </p>
<p>[2015-08-24 15:21:07.650707] I [monitor(monitor):221:monitor] Monitor: ------------------------------------------------------------<br>
[2015-08-24 15:21:07.651144] I [monitor(monitor):222:monitor] Monitor: starting gsyncd worker<br>
[2015-08-24 15:21:07.764817] I [gsyncd(/gluster/home-brick-1):649:main_i] <top>: syncing: gluster://localhost:home -> ssh://root@remoteserver1.uberit.net:gluster://localhost:homegs<br>
[2015-08-24 15:21:07.768552] I [changelogagent(agent):75:__init__] ChangelogAgent: Agent listining...<br>
[2015-08-24 15:21:11.9820] I [master(/gluster/home-brick-1):83:gmaster_builder] <top>: setting up xsync change detection mode<br>
[2015-08-24 15:21:11.10199] I [master(/gluster/home-brick-1):404:__init__] _GMaster: using 'rsync' as the sync engine<br>
[2015-08-24 15:21:11.10946] I [master(/gluster/home-brick-1):83:gmaster_builder] <top>: setting up changelog change detection mode<br>
[2015-08-24 15:21:11.11115] I [master(/gluster/home-brick-1):404:__init__] _GMaster: using 'rsync' as the sync engine<br>
[2015-08-24 15:21:11.11744] I [master(/gluster/home-brick-1):83:gmaster_builder] <top>: setting up changeloghistory change detection mode<br>
[2015-08-24 15:21:11.11933] I [master(/gluster/home-brick-1):404:__init__] _GMaster: using 'rsync' as the sync engine<br>
[2015-08-24 15:21:13.59192] I [master(/gluster/home-brick-1):1212:register] _GMaster: xsync temp directory: /var/lib/misc/glusterfsd/home/ssh%3A%2F%2Froot%40IPADDRESS%3Agluster%3A%2F%2F127.0.0.1%3Ahomegs/62d98e8cc00a34eb85b4fe6d6fd3ba33/xsync<br>
[2015-08-24 15:21:13.59454] I [resource(/gluster/home-brick-1):1432:service_loop] GLUSTER: Register time: 1440426073<br>
[2015-08-24 15:21:13.113203] I [master(/gluster/home-brick-1):523:crawlwrap] _GMaster: primary master with volume id 2299a204-a1dc-449d-8556-bc65197373c7 ...<br>
[2015-08-24 15:21:13.122018] I [master(/gluster/home-brick-1):532:crawlwrap] _GMaster: crawl interval: 1 seconds<br>
[2015-08-24 15:21:23.209912] E [repce(/gluster/home-brick-1):207:__call__] RepceClient: call 1561:140164806457088:1440426083.11 (keep_alive) failed on peer with OSError<br>
[2015-08-24 15:21:23.210119] E [syncdutils(/gluster/home-brick-1):276:log_raise_exception] <top>: FAIL:<br>
Traceback (most recent call last):<br>
File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 306, in twrap<br>
tf(*aa)<br>
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 438, in keep_alive<br>
cls.slave.server.keep_alive(vi)<br>
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 226, in __call__<br>
return self.ins(self.meth, *a)<br>
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 208, in __call__<br>
raise res<br>
OSError: [Errno 22] Invalid argument<br>
[2015-08-24 15:21:23.210975] I [syncdutils(/gluster/home-brick-1):220:finalize] <top>: exiting.<br>
[2015-08-24 15:21:23.212455] I [repce(agent):92:service_loop] RepceServer: terminating on reaching EOF.<br>
[2015-08-24 15:21:23.212707] I [syncdutils(agent):220:finalize] <top>: exiting.<br>
[2015-08-24 15:21:24.23336] I [monitor(monitor):282:monitor] Monitor: worker(/gluster/home-brick-1) died in startup phase</p>
<p> </p>
<p>and on the slave (in a different timezone, one hour behind):</p>
<p> </p>
<p>[2015-08-24 14:22:02.923098] I [resource(slave):844:service_loop] GLUSTER: slave listening<br>
[2015-08-24 14:22:07.606774] E [repce(slave):117:worker] <top>: call failed:<br>
Traceback (most recent call last):<br>
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 113, in worker<br>
res = getattr(self.obj, rmeth)(*in_data[2:])<br>
File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 731, in entry_ops<br>
[ESTALE, EINVAL])<br>
File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 475, in errno_wrap<br>
return call(*arg)<br>
File "/usr/libexec/glusterfs/python/syncdaemon/libcxattr.py", line 78, in lsetxattr<br>
cls.raise_oserr()<br>
File "/usr/libexec/glusterfs/python/syncdaemon/libcxattr.py", line 37, in raise_oserr<br>
raise OSError(errn, os.strerror(errn))<br>
OSError: [Errno 5] Input/output error<br>
[2015-08-24 14:22:07.652092] I [repce(slave):92:service_loop] RepceServer: terminating on reaching EOF.<br>
[2015-08-24 14:22:07.652364] I [syncdutils(slave):220:finalize] <top>: exiting.</p>
<p><br>
</p>
<span style="font-family:arial,helvetica,sans-serif;font-size:x-small">--------------</span><br style="font-family:arial,helvetica,sans-serif;font-size:x-small"><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">G-RESEARCH believes the information provided herein is reliable. While every care has been taken to ensure accuracy, the information is furnished to the recipients with no warranty as to the completeness and accuracy of its contents and on condition that any errors or omissions shall not be made the basis of any claim, demand or cause of action.<br></span><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">The information in this email is intended only for the named recipient. If you are not the intended recipient please notify us immediately and do not copy, distribute or take action based on this e-mail.</span><br style="font-family:arial,helvetica,sans-serif;font-size:x-small"><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">All messages sent to and from this e-mail address will be logged by G-RESEARCH and are subject to archival storage, monitoring, review and disclosure.</span><br style="font-family:arial,helvetica,sans-serif;font-size:x-small"><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">G-RESEARCH is the trading name of Trenchant Limited, 5th Floor, Whittington House, 19-30 Alfred Place, London WC1E 7EA.</span><br style="font-family:arial,helvetica,sans-serif;font-size:x-small"><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">Trenchant Limited is a company registered in England with company number 08127121.</span><br style="font-family:arial,helvetica,sans-serif;font-size:x-small"><span style="font-family:arial,helvetica,sans-serif;font-size:x-small">--------------</span>
</div>
</blockquote></div>
</blockquote></div>