<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
I also had problems getting geo-replication working correctly and
eventually gave it up due to project time constraints.<br>
<br>
What version of gluster?<br>
What is the topology of x, xx, and xxx/xxy/xxz?<br>
<br>
I tried a 2x2 stripe-replica with geo-replication to a 2x1 stripe
using 3.7.4. Starting replication with 32 GB of small files never
completed, it failed several times. Starting replication with an
empty volume then filling it with a rate limit of 2000k/s managed to
keep sync until completion but could not handle the rate of change
under normal usage.<br>
<div class="moz-signature">
<style type="text/css"> body {font-family: sans-serif;}</style><br>
</div>
<div class="moz-cite-prefix">On 5/11/2015 3:30 AM, Brian Ericson
wrote:<br>
</div>
<blockquote cite="mid:563A40CF.8010300@ptc.com" type="cite">tl;dr --
geo-replication of ~200,000 CHANGELOG files is killing me... Help!
<br>
<br>
I have about 125G spread over just shy of 5000 files that I'm
replicating with
<br>
geo-replication to nodes around the world. The content is fairly
stable and
<br>
probably hasn't changed at all since I initially established the
GlusterFS
<br>
nodes/network, which looks as follows:
<br>
x -> xx -> [xxx, xxy] (x geo-replicates to xx, xx
geo-replicates to xxx/xxy)
<br>
<br>
Latency & throughput are markedly different (x -> xx is the
fastest, xx -> xxx
<br>
the slowest (at about 1G/hour)). That said, all nodes were synced
with 5 days
<br>
of setting up the network.
<br>
<br>
I have since added another node, xxz, which is also geo-replicated
from xx (xx
<br>
-> xxz). Its latency/throughput is clearly better than xx ->
xxx's, but over 5
<br>
days later, I'm still replicating CHANGELOGs and haven't gotten to
any real
<br>
content (the replicated volumes' mounted filesystems are empty).
<br>
<br>
Starting with x, you can see I have a "reasonable" number of
CHANGELOGs:
<br>
x # find /bricks/*/.glusterfs/changelogs -name CHANGELOG\* | wc -l
<br>
186
<br>
<br>
However, xxz's source is xx, and I've got a real problem with xx:
<br>
xx # find /bricks/*/.glusterfs/changelogs -name CHANGELOG\* | wc
-l
<br>
193450
<br>
<br>
5+ days into this, and I've hardly managed to dent this on xxz:
<br>
xxz # find /bricks/*/.glusterfs/changelogs -name CHANGELOG\* | wc
-l
<br>
43211
<br>
<br>
On top of that, xx is generating new CHANGELOGs at a rate of
~6/minute (two
<br>
volumes at ~3/minute each), so chasing CHANGELOGs is a (quickly)
moving target.
<br>
<br>
And these files are small! The "I'm alive" file is 92 bytes long,
I've also
<br>
seen them also average about 4k. Demonstrating latency/throughput,
you can see
<br>
that small files (for me) are a real killer:
<br>
### x -> xx (fastest route)
<br>
# for i in 1 10 100 1000; do file="$( dd if=/dev/urandom bs=1024
count=$((4000/i)) 2> /dev/null )"; echo "$i ($(( $( echo -n
"$file" | wc -c )/1024 ))k): $( ( time for i in $( seq 1 $i ); do
echo -n "$file" | ssh xx 'cat > /dev/null'; done ) |& awk
'/^real/{ print $2 }' )"; done
<br>
1 $i ); do echo -n "$file" | ssh $location 'cat > /dev/null';
done ) |& awk '/^real/{ print $2 }' )"; done
<br>
1 (3984k): 0m4.777s
<br>
10 (398k): 0m10.737s
<br>
100 (39k): 0m53.286s
<br>
1000 (3k): 7m21.493s
<br>
<br>
### xx -> xxx (slowest route)
<br>
# for i in 1 10 100 1000; do file="$( dd if=/dev/urandom bs=1024
count=$((4000/i)) 2> /dev/null )"; echo "$i ($(( $( echo -n
"$file" | wc -c )/1024 ))k): $( ( time for i in $( seq 1 $i ); do
echo -n "$file" | ssh xxx 'cat > /dev/null'; done ) |& awk
'/^real/{ print $2 }' )"; done
<br>
1 (3984k): 0m11.065s
<br>
10 (398k): 0m41.007s
<br>
100 (39k): 4m52.814s
<br>
1000 (3k): 39m23.009s
<br>
<br>
### xx -> xxz (the route I've added and am trying to sync)
<br>
# for i in 1 10 100 1000; do file="$( dd if=/dev/urandom bs=1024
count=$((4000/i)) 2> /dev/null )"; echo "$i ($(( $( echo -n
"$file" | wc -c )/1024 ))k): $( ( time for i in $( seq 1 $i ); do
echo -n "$file" | ssh xxz 'cat > /dev/null'; done ) |& awk
'/^real/{ print $2 }' )"; done
<br>
1 (3984k): 0m2.673s
<br>
10 (398k): 0m16.333s
<br>
100 (39k): 2m0.676s
<br>
1000 (3k): 17m28.265s
<br>
<br>
What you're looking at is the cost of transferring a total of
4000k: 1 transfer
<br>
at 4000k, 10@400k, 100@40k, and 1000@4k. With 1 transfer at under
3s and 1000
<br>
transfers at nearly 17 1/2 minutes for xx -> xxz and for the
same total
<br>
transfer size, it's really a killer to transfer CHANGELOGs,
especially almost
<br>
200,000 of them.
<br>
<br>
And, 92 byte files doesn't improve this:
<br>
### x -> xx (fastest route)
<br>
# file="$( dd if=/dev/urandom bs=92 count=1 2> /dev/null )";
i=100; echo "$i ($(( $( echo -n "$file" | wc -c ) ))): $( ( time
for i in $( seq 1 $i ); do echo -n "$file" | ssh xx 'cat >
/dev/null'; done ) |& awk '/^real/{ print $2 }' )"
<br>
100 (92): 0m34.164s
<br>
<br>
### xx -> xxx (slowest route)
<br>
# file="$( dd if=/dev/urandom bs=92 count=1 2> /dev/null )";
i=100; echo "$i ($(( $( echo -n "$file" | wc -c ) ))): $( ( time
for i in $( seq 1 $i ); do echo -n "$file" | ssh xxx 'cat >
/dev/null'; done ) |& awk '/^real/{ print $2 }' )"
<br>
100 (92): 3m53.388s
<br>
<br>
### xx -> xxz (the route I've added and am trying to sync)
<br>
# file="$( dd if=/dev/urandom bs=92 count=1 2> /dev/null )";
i=100; echo "$i ($(( $( echo -n "$file" | wc -c ) ))): $( ( time
for i in $( seq 1 $i ); do echo -n "$file" | ssh xxz 'cat >
/dev/null'; done ) |& awk '/^real/{ print $2 }' )"
<br>
100 (92): 1m43.389s
<br>
<br>
Questions...:
<br>
o Why so many CHANGELOGs?
<br>
<br>
o Why so slow (in 5 days, I've transferred 43211 CHANGELOGs, so
43211/5/24/60=6
<br>
implies a real transfer rate of about 6 CHANGELOG files per
minute, which
<br>
brings me back to xx's generating new ones at about that
rate...)?
<br>
<br>
o What can I do to "fix" this?
<br>
<br>
_______________________________________________
<br>
Gluster-users mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<br>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a>
<br>
</blockquote>
<br>
</body>
</html>