undefined | Better HN

0 pointsotterley13y ago0 comments

Of course you'll have to have presence in multiple DCs. I didn't mean to imply that one would suffice.

I'm not sure whether you've analyzed the bandwidth of the replication stream, but my experience is that if you've got a good circuit provider, you should have adequate bandwidth for 30Mbps+ streams, except under extremely unusual conditions, even over the open Internet.

0 comments

1 comments · 1 top-level

solso13y ago

The replication stream is about 20-25 Mbps for the current throughput of 25K-30K req/s, our peak to valley ratio is low.

It seems that we have found this unusual conditions you mention :-) Before compression our replication was often lagging (when below the required bandwidth), which is very dangerous, even if after a minute bandwidth goes well above the threshold.

The case of replication is kind of worst case, it's no good to have an average of 30 Mbps if it stays a substantial fraction of the time below the threshold. For every hours at 75% memory grew 2GB, but even one minute below the threshold is bad... if the replication queue builds up and there is a crash, consistency goes out of the window. So better to over-dimension capacity.

We even did a quick test with Hetzner instead of Rackspace, it was worse. The average over 4 hours was about 15 Mbps.

j / k navigate · click thread line to collapse