undefined | Better HN

0 pointshalbritt7y ago0 comments

Couple things:

Cross-region networking. To this day, I'm flabbergasted this isn't something that AWS has. All the other networking bits seem more sensible to me as well.

Another, is 2Gbps per core up to 8 cores which is way more throughput than I've seen on any AWS or Azure instance.

On workloads where I care about Network IO:CPU ratio, I'm using 8 core nodes and seeing 16gbps throughput between them consistently.

0 comments

3 comments · 2 top-level

ti_ranger7y ago· 1 in thread

> Cross-region networking. To this day, I'm flabbergasted this isn't something that AWS has. All the other networking bits seem more sensible to me as well.

AWS tries to avoid letting customers create multi-region or global failure modes, like:

* https://status.cloud.google.com/incident/compute/18005 22-hour incident with GLOBAL impact resulting in many VMs getting duplicate IPs (this no network connectivity) on 2018-06-15

* https://status.cloud.google.com/incident/cloud-networking/18... 3-hour (seemingly GLOBAL) impact to load-balancers (no updates or creations) on 2018-01-03

* https://status.cloud.google.com/incident/compute/16007 - 18-minute GLOBAL outage on 2016-04-11

* https://status.cloud.google.com/incident/compute/15055 - 5-minute GLOBAL outage on 2015-08-04

* https://groups.google.com/forum/#!topic/gce-operations/fynnX... 43-minute multi-region packet loss due to multi-region configuration deployment on 2015-03-07

* https://groups.google.com/forum/#!topic/gce-operations/1uw-q... unscheduled reboot of 28% of instances in multiple regions in ~1.5h on 2014-09-17

There have been recent changes in many AWS services (e.g. DynamoDB, S3, Aurora) to allow cross-region replication without introducing any multi-region dependency, precisely to make it easier to implement multi-region infrastrucure to tolerate a single-region failure (however rare that is on AWS compared to global outages on GCP).

> Another, is 2Gbps per core up to 8 cores which is way more throughput than I've seen on any AWS or Azure instance.

Which instance types did use on AWS? Many recent instance types (r4, i3, c5, r5, z1) support up to 10Gbps with once core (2 vCPU instances). However, you may need to use placement groups ( https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placemen... ) in large regions in order to get the full throughput on low TCP connection numbers. The only reason I can think of that would explain why GCP doesn't have this problem is they don't have any regions anywhere near as large as large AWS regions ...

Edit: white-space only formatting changes

halbrittOP7y ago

Your point about global failure modes is well-taken. Even still, one could reasonably expect inter-region networking not to be so difficult.

As for the instances I tested, they were m4 and r4. To get full 10GE out of either, I needed to use m4.10xl and r4.8xl, both roughly around $900 per month.

By comparison an n1-standard-8 is about $60/mo and the n1-highcpu-8 is $50/mo. I've tested the former and got something like 15.8 or 15.9gbps using iperf.

IMHO, that's pretty significant.

gazoakley7y ago

You can do inter region VPC peering: https://aws.amazon.com/about-aws/whats-new/2017/11/announcin...

j / k navigate · click thread line to collapse