Or you need to restore your Postgres database and you find out that the backups didn't work.
And finally you have a brilliant idea of hiring a second $150k/year dev ops admin so that at least one is always working and they can check each other's work. Suddenly, you're spending $300k on two dev ops admins alone and the cost savings of using cheaper dedicated servers are completely gone.
Or you need to debug why your Lambda function is throttling and you find out that the CloudWatch logs were never properly configured and you’ve been flying blind for three months.
And finally you have a brilliant idea of hiring a second $150k/year AWS solutions architect so that at least one person can actually understand the bill and they can check each other’s Terraform configs. Suddenly, you’re spending $300k on two cloud wizards alone and the cost savings of "not managing your own infrastructure" are completely gone.
The snide rebuttal basically writes itself.
Except - wait, you do have to think about, because of course you. So the promise of AWS is gone.
Or when you need to post on Hackernews to get support from your cloud provider as locked out of your account, being ignored and the only way to get access is try to create as much noise as possible it gets spotted.
Or your cloud provider wipes your account and you are a $135B pension fund [1]
Or your cloud portfolio is so big you need a "platform" team of multiple devops/developer staff to build wrappers around/package up your cloud provider for you and your platform team is now the bottleneck.
Cloud is useful but it's not as pain free as everyone says when comparing with managing your own, it still costs money and work. Having worked on several cloud transformations they've all cost more and taken more effort than expected. A large proportion have also been canned/postponed/re-evaluated due to cost/size/time/complexity.
Unless you are a big spender with dedicated technical account manager, your support is likely to be as bad as a no name budget VPS provider.
Both cloud and traditional hosting have their merits and place.
[1] https://arstechnica.com/gadgets/2024/05/google-cloud-acciden...
Or when you need to post on Hackernews to get support from your cloud provider as locked out of your account, being ignored and the only way to get access is try to create as much noise as possible it gets spotted.
https://www.reddit.com/r/hetzner/comments/1ha5qgk/hetzner_ca...
It's actually kinda frustrating - as an industry we're accepting worse outcomes due to misperceptions. That's how the free market goes sometimes.
As opposed to be with the small provider round the corner who is currently having a beer and will look at that tomorrow morning.
Now - I am in the phase where I ap seriously considering to move my email from Google to a small player in Europe (still not sure who) so this is what may ultimately be my fate :)
Customers call and complain about downtime, I can just vaguely point at everything being on fire from Facebook to Instagram to online banking sites.
They get it.
When the self-hosted server fries itself, I'm on the hook for fixing it ASAP.
If my own dedicated server goes down, I'm going to need to call my admin at 3am 10 times just to wake him up.
in my experience you always need a "Devops team" to operate all that cloud stuff; so to paraphrase - suddenly you're spending $400k on three devops to operate $500k cloud
I think The Promise behind the cloud was you just pay for the service and not worry about it, but in practice you need some team to maintain it
One included a whole OVH building burning down with our server in it, and recovery was faster than the recent AWS and Cloudflare outages. We felt less impotent and we could do more to mitigate the situation.
If you want to, these providers also offer VMs, object storage and other virtualized services for way cheaper with similar guarantees, they are not stuck in the last century.
And I don’t know how people are using cloud, but most config issues happen above the VM/Docker/Kubernetes level, which is the same wether you are on cloud or not. Even fully managed database deployments or serverless backends are not really that much simpler or less error-prone than deploying the containers yourself. Actually the complexity of Cloud is often a worse minefield of footguns, with their myriad artificial quirks and limitations. Often dealing with the true complexities of the underlying open-source technologies they are reselling ends up being easier and more predictable.
This fearmongering is really weakening us as an industry. Just try it, it is not as complex or dangerous as they claim.
Higher-level services like PaaS (Heroku and above) genuinely do abstract a number of details. But EC2 is just renting pseudo-bare computers—they save no complexity, and they add more by being diskless and requiring networked storage (EBS). The main thing they give you is the ability to spin up arbitrarily many more identical instances at a moment’s notice (usually, at least theoretically, though the amount of the time that you actually hit unavailability or shadow quotas is surprisingly high).
But I'd like to sleep at night and the cost of AWS is not a significant issue to the business.
And yes of course such costs are nothing if you are thinking of $300K just on a couple sysadmins. But this is just a bizarre bubble in a handful of small areas in the US and I am not sure how it can stay like that for much longer in this era of remote work.
We built a whole business with $100K in seed and a few government grants. I have worked with quite a few world-class senior engineers happily making 40K-70K.
The truth is that there's still a lot of things you have to handle, including cloud bugs and problems. And other problems you don't have to think about anymore, especially with fully managed, high-level PaaS- like services.
I ran a cloud backend service for a startup with users, using manged services, and we still had an on-call team. The cloud is not magic.
The initial impression that we don't need to hire many people because AWS takes care of everthing, fades away pretty quick.
You still need to hire the same people, they just do the same things in a different way.
Single human point of failure for something tied to the bottom line is NOT a technical problem, it's management.
I'll have you know I am a cantaloupe, you insensitive clod!
What is your point?
Every team with dedicated hardware in a data center it was generally 1-2 people who would have fixed stuff quickly, no matter the size of the company (small ones, of course - so 10-50 devs). And that's with available replacement hardware.
I'm not even one of the "cloud is so great" people - but it you're generally doing software it's actually a lot less friction.
And while the ratio of cost difference may sound bad, it's generally not. Unless we're talkign huge scale, you can buy a lot of AWS crap for the yearly salary of a single person.
AWS isn't going to help you setup your security, you have to do it yourself. Previously a sysadmin would do this, now it's the devs. They aren't going to monitor your database performance. Previously a sysadmin would do this, now it's the devs. They aren't going to setup your networking. Previously a sysadmin would do this, ...
Managing hardware and updating hosts is maybe 10% of the work of a sysadmin. You can't buy much on 1/10th of a sysadmins salary, and even the things you can, the quality and response time are generally going to be shit compared to someone who cares about your company (been there).
is that because they were using AWS so hired people who knew AWS?
I would personally have far more confidence in my ability to troubleshoot or redeploy a dedicated server than the AWS services to replace it.
> Every team with dedicated hardware in a data center it was generally 1-2 people who would have fixed stuff quickly, no matter the size of the company (small ones, of course - so 10-50 devs). And that's with available replacement hardware.
There are lots of options for renting dedicated hardware, that the service provider will maintain,. Its still far cheaper than AWS. Even if you have redundancy for everything its still a lot cheaper.