undefined | Better HN

0 pointssam0x174y ago0 comments

Most startups I've worked at literally have a script to deploy their whole setup to a new region when desired. Then you just need latency-based routing running on top of it to ensure people are processed in the closest region to them. Really not expensive. You can do this with under $200/month in terms of complexity and the bandwidth + database costs are going to be roughly the same as they normally are because you're splitting your load between regions. Now if you stupidly just duplicate your current infrastructure entirely, yes it would be expensive because you'd be massively overpaying on DB.

In theory the only additional cost should be the latency-based routing itself, which is $50/month. Other than that, you'll probably save money if you choose the right regions.

0 comments

8 comments · 5 top-level

bradly4y ago· 2 in thread

Are the same services available in all regions?

Are the same instance sizes available in all regions?

Are there enough instances of the sizes you need?

Do you have reserved instances in the other region?

Are your increased quotas applied to all regions?

What region are your S3 assets in? Are you going to migrate those as well?

Is it acceptable for all user sessions to be terminated?

Have you load tested the other region?

How often are you going to test the region fail over? Yearly? Quarterly? With every code change?

What is the acceptable RTO and RPO with executives and board-members?

And all of that is without thinking about cache warming, database migration/mirror/replication, solr indexing (are you going to migrate the index or rebuild? Do you know how long it takes to rebuild your solr index?).

The startups you worked at probably had different needs the Roblox. I was the tech leach on a Rails app that was embedded in TurboTax and QuickBooks and was rendered on each TT screen transition and reading your comment in that context shows a lot of inexperience in large, production systems.

sam0x17OP4y ago

A lot of this can also be mitigated by going all in on API gateway + Lambda, like we have at Arist. We only need to worry about DB scaling and a few considerations with S3 (that are themselves mitigated by using CloudFront).

bradly4y ago

Are you implying that Roblox should move their entire system to the API Gateway + Lambda to solve their availability problems?

Seriously though, what is your RTO and RPO? We are talking systems that when they are down you are on the news. Systems where minutes of downtime are millions of dollars. I encourage you to setup some time with your CTO at Arist and talk through these questions.

1 more reply

aprdm4y ago· 1 in thread

You have no idea what you're talking about when comparing their setup to "most startups you've worked with"

sam0x17OP4y ago

This was in reply to OP who said deploying to a new region is insanely complicated. In general it is not. For Roblox, if they are manually doing stuff in EC2, it could be quite complicated.

e4e78a064y ago

Correctly handling failure edge cases in a active-active multi-region distributed database requires work. SaaS DBs do a lot of the heavy lifting but they are still highly configurable and you need to understand the impact of the config you use. Not to mention your scale-up runbooks need to be established so a stampede from a failure in one region doesn't cause the other region to go down. You also need to avoid cross-region traffic even though you might have stateful services that aren't replicated across regions. That might mean changes in config or business logic across all your services.

It is absolutely not as simple as spinning up a cluster on AWS at Roblox's scale.

Twirrim4y ago

Roblox is not a startup, and has a significant sized footprint (18,000 servers isn't something that's just available, even within clouds. They're not magically scalable places, capacity tends to land just ahead of demand). It's not even remotely a simple case of just "run a script and wee we have redundancy" There are lots of things to consider.

18k servers is also not cheap, at all. They suggest at least some of their clusters are running on 64 cores, some on 128. I'm guessing they probably have a fair spread of cores.

Just to give a sense of cost, AWS's calculator estimates 18,0000 32 core instances would set you back $9m per month. That's just the EC2 cost, and assuming a lower core count is used by other components in the platform. 64 core would bump that to $18m. Per month. Doing nothing but sitting waiting ready. That's not considering network bandwidth costs, load balancers etc. etc.

When you're talking infrastructure on that scale, you have to contact cloud companies in advance, and work with them around capacity requirements, or you'll find you're barely started on provisioning and you won't find capacity available (you'll want to on that scale anyway because you'll get discounts but it's still going to be very expensive)

Symbiote4y ago

So Roblox need a button to press to (re)deploy 18,000 servers and 170,000 containers? They already have multiple core data centres, as well as many edge locations.

You will note the problem was with the software provided and supported by Hashicorp.

j / k navigate · click thread line to collapse

0 comments

8 comments · 5 top-level

bradly4y ago· 2 in thread

Are the same services available in all regions?

Are the same instance sizes available in all regions?

Are there enough instances of the sizes you need?

Do you have reserved instances in the other region?

Are your increased quotas applied to all regions?

What region are your S3 assets in? Are you going to migrate those as well?

Is it acceptable for all user sessions to be terminated?

Have you load tested the other region?

How often are you going to test the region fail over? Yearly? Quarterly? With every code change?

What is the acceptable RTO and RPO with executives and board-members?

sam0x17OP4y ago

bradly4y ago

Are you implying that Roblox should move their entire system to the API Gateway + Lambda to solve their availability problems?

1 more reply

aprdm4y ago· 1 in thread

You have no idea what you're talking about when comparing their setup to "most startups you've worked with"

sam0x17OP4y ago

This was in reply to OP who said deploying to a new region is insanely complicated. In general it is not. For Roblox, if they are manually doing stuff in EC2, it could be quite complicated.

e4e78a064y ago

It is absolutely not as simple as spinning up a cluster on AWS at Roblox's scale.

Twirrim4y ago

18k servers is also not cheap, at all. They suggest at least some of their clusters are running on 64 cores, some on 128. I'm guessing they probably have a fair spread of cores.

Symbiote4y ago

So Roblox need a button to press to (re)deploy 18,000 servers and 170,000 containers? They already have multiple core data centres, as well as many edge locations.

You will note the problem was with the software provided and supported by Hashicorp.

j / k navigate · click thread line to collapse