I've seen the billing costs of the public cloud absolutely demolish an IT org's yearly budget in a month because of unexpected cost upticks, and I've seen a reduction in total cost of ownership by reducing needed licensing/staff/building costs. I get both sides on what the public cloud can do.
I've also seen what you can do on-premises; I've worked with clients who manage 7000+ machines (mostly virtual + some physical) with a team of 4 using pretty reasonably priced on-site hardware. (pro-tip, I guess Hitachi boxes are absurdly great servers with fantastic uptime, pockmarked only by an absolutely horrendous UI to manage)
My experience from the many clients I work with is that it is less about the specific stack you settle on and more your comfort level in getting the most efficiency out of it. The deeper and more intimate you are with all levels of your infrastructure, the better you know how to eke out the most from every single $.01 you spend on it.
You need to be able to do both options before having an opinion on which is appropriate in which case. I am suprised to have to state this. But in my experience people argue one option a lot without being to deliver the other.
People who know bare metal are rare these days from the total of available infrastructure engineers (call them sysadmins, devops, etc). I guess this justifies companies looking at cloud a little bit. But if you really search you can find engineers sub 100k per year being able to deliver 100k per month savings compared to AWS.
There are also engineers who stayed away from cloud and can't deliver that option. A lot more rare though. The same level of wrong if they argue against cloud from ignorance.
The right choice for serious infrastructures is always both these days. Have the bulk on premise for steady loads and 95% of features, expand to public clouds for dynamic scaling and features you don't want do do yourself, at least yet. This combination offers good costs, flexibility, covers possible future needs, etc
Few companies actually need that many instances. The math for the less than 10-20 instances the vast majority of companies actually need is quite brutal. A day of your time basically pays for months/years of hosting. The thing to optimize is devops time. Not hosting cost. It's by far the most expensive thing and also the most likely thing to fail on you (by leaving, by being incompetent, negligent, lazy, sick, etc.) and also the hardest thing to source when you need more of it. Good devops people are scarce.
I've dealt with plenty of companies that had no more than two or three idling t2 instances paying for multiple devops people to babysit that "infrastructure". It's stupid and wasteful. A decent devops person costs about 0.5-1 instance year (i.e. a full year of hosting 24x7) per hour for such small instances. And scaling an instance group from 2 to 500 instances is a 1 minute job if you ever need to. Unless the savings are enormous, the time they spend on minimizing the number of instances or automating their deployment will never be worth the money. It's money down the drain. You need to think in terms of a few hours for getting stuff done to make it worth the cost. Anything more is probably too expensive.
If you need that kind of availability, you need to have people on call anyway to babysit your app. A good infrastructure (unless built to the minimal price point) will handle nearly all cases of hardware failure automatically, without someone having to wake up, so it's not likely to put additional load on those people.
I'm not necessarily disagreeing with your overall point, but if you need five nines, you're talking about an entirely different league of infrastructure compared to people who need two or three VMs that could also be handled by a NUC somewhere in the office (which will amortize itself against AWS in a few months).
Having said that, your requirement is pretty absurd. Billions of people choose to own and maintain houses and cars and cook their own food because it's cheaper than the alternatives. Nobody expects them to be professional mechanics or cooks.
To be fair, there’s also nobody that would listen to them over a professional mechanic or cook.
The real cost of cloud is that nearly anyone think it's impossible to setup a infra for yourself....loosing systemadmin as a role in companys probably the biggest loss.
So why do you limit a project that's supposed to save $1M+ to a cost of $50k or so?
In 2013 I was running a school with 500 students and 70 teachers on a 8GB RAM HP Server that was built in 2005 and had no problems other than disk speeds for network transfers.
The same setup in the cloud would have been much more expensive but then again I had/have access to unlimited Microsoft product licenses because of the MS-ACH agreement so take that with a grain of salt. They even give every public school in the country their own unlimited KMS host key.
Hiring your own janitor is cheaper than paying an outsourcing firm.
Building your own office is cheaper than renting one.
Doing your taxes with pen and paper is cheaper than paying turbo tax.
Making your own food is cheaper than eating out. Hiring a cook directly is cheaper than hiring sudexo.
I could keep going. But sometimes it’s not just cost. The biggest two values you get with AWS is 1) reducing time spent outside your business’s core competencies and 2) a vast ecosystem - 3rd party offerings, readily available devs, consulting services, and compliance services.
I’d add that for those having compliance needs. It’s not always as simple as rack and stacking infra. You have to use services that meet the compliance auditors needs.
A. Does your workload fully utilize 100% of the capacity of the resource? If not, then cloud would be cheaper. Just like if you only need office space for a few people, it’s not cost effective to buy an entire office building. If you only need server with a few gigs of RAM, it’s not cost effective to buy (own) an entire physical server.
B. If you are going to fully utilize a resource and don’t want to purchase/own it - then a service provider needs to provide that asset to you around cost and make margin from the efficiency from scale they have. Example, it’s actually more expensive for me to buy all of the ingredients to make a hamburger than to simply buy a fully prepared hamburger for McDonalds. McDonald’s is able to provide this due to their scale.
What I’ve seen is that when you’re in Group B, many people are finding that AWS/etc is way more expensive. Essentially, their scale in efficiency is not being passed down to the customer in cost savings. And the sizable cost premium is not worth the value received in return.
I’ll give a good example of where this does make sense, and that’s Hetzner or OVH. Their scale allows them to procure & host dedicated servers at a price I’d be difficult to match doing it myself. Or even if I could beat their price, it would be minimally. But folks are finding that with AWS/etc, that premium is extreme and that’s where the equation is unbalanced for folks.
At that point, the question becomes how much is your time worth, or in this analogy, are you ready to hire a professional chef to get better quality food?
The technical considerations are a distant second or third place.
In my previous role, my manager argued for my work as a developer to be charged as Capex to the project instead of Opex.
Why would accountants want the opposite for AWS?
(1) opportunity cost: doing your own systems administration instead of growing your business (features, marketing, etc).
(2) switching cost: once you have a working system that outgrows the free / cheap AWS tiers, and might be cheaper to run outside cloud, switching away from the cloud becomes expensive, and does not look like a good investment to many, see (1).
I have a small GKE cluster and a few databases and I’m well above that, I nearly hit that in a month!
Maybe you don’t have read replicas of your databases? Do you take any traffic?
Most of the issues I see from either on-premises or cloud generally come from not actually understanding the business/use cases/environment. Cloud becomes the 'solution' to a problem of people and process, rather than a value proposition that augments existing reality. You can't make good decisions (e.g. Should I outsource my taxes) if you don't understand what doing those thiggs acrually involves (which most people don't bother to even try).
And usually worked into this is a lot of outsourcing of expertise to the point the business relies on third parties to tell them what to do, which is never a cheaper outcome.
I do my own taxes, but maybe that's just easier in Europe and is definitely easier for individuals. That said, there's no reason why a LibreOffice spreadsheet would be an insufficient solution for handling taxes and other things like that.
I also haven't eaten out in years, the closest to that was ordering some Wolt when hanging out with my friends pre-COVID, because they wanted to try some. Apart from that, it's all just home cooked meals for me and that's pretty great. It also seems to be working out great for the folks over at https://www.reddit.com/r/mealprep/top/?t=month
At work, the company that i work for have their own building and have their own support staff as well, which seems to be working out great for them.
Furthermore, there are plenty of on prem resources that are used and despite the disadvantage of lacking self-service in many cases, there's very little difference in configuring and running software for deployments, with something like Ansible and containers. Even moreso when you have to support clients that have their own particular data centers and on prem deployments, which might differ noticeably from public cloud offerings. That's even not thinking about things like compliance in regards to what data can be stored where.
Personally, i also have a homelab with some repurposed old computers with 200 GEs and value RAM, a few HDDs and WireGuard for working around NAT and exposing my sites to the world through a pretty cheap cloud VPS or two from https://www.time4vps.com/?affid=5294 (affiliate link, to make hosting cheaper if anyone else uses them). Of course, when i need 24/7 uptime, i do use their VPSes in a hybrid cloud setup, especially since my blog getting 30k views could be a bit taxing on a residental 4G LTE modem connection.
The argument about competencies, ecosystems, 3rd party offerings, outsourcing and so on is probably a valid for some, but not for me and not for many companies out there - too often you end up depending on SaaS solutions which vendor lock you and might cause you to spend unreasonable amounts of money, or will let you remain ignorant about how to actually manage the software that you're using, i think SaaSS (Service as a Software Substitute) is a relevant term here: https://www.gnu.org/philosophy/who-does-that-server-really-s...
That said, what works for me and even the company that employs me, won't work for others. And what works for others, won't work for me. This is all because of how different the circumstances of various people out there are: i cannot afford AWS, i cannot afford Azure, GCP and managed services for my own needs.
I currently pay 320 EUR for 6 cloud VPSes per year (and additional amounts for the occasional replacement HDD for my homelab), whereas others pay similar amounts for their cloud platforms of choice per month. For them, depending on their circumstances, it might be more cost effective to spend their time working and throw money at problems, whereas for me it's almost always more cost effective to learn the tech myself.
Similarly to how in Latvia you could hire a team of developers for what one developer would cost in the US. Companies have other factors to consider, of course, but this is just one example - the alternative (opportunity) costs of individuals.
Edit: Of course, some in the comments are talking about hundreds of VPSes/VMs/nodes and in my eyes, that's just an order of magnitude or two higher than what i'm talking about. I've seen plenty of companies in my country running their own data centers and there have been relatively few issues with those that i'm aware of. Something like Ansible and container clusters can scale pretty far!
The problems were more often caused by either mismanaged environments/deployments by developers/agencies who just didn't care about shipping sustainable software but cared more about getting paid and making their software someone else's problem, or making mistakes early in the development and not considering load testing and scalability of the systems as priorities. I'd argue that you can do bad engineering anywhere, though, be it on prem or in the cloud.
Let's say the year is 2012 and Redshift is introduced, completely changing how organizations can generate insights from their data. Running your own datacenter? good luck waiting for the ops team to install something similar! It might take you a couple of quarters assuming they are already competent at it. On the Cloud? Press a few buttons and you're off to the races.
velocity is a competitive advantage
These new services are nice for startups and for eventually outsourcing... Aaaand of course for upselling to folks who are already in the faith.
Velocity is and advantage. Yes, but if you really see a brutally good deal with some new AWS service there's nothing preventing you from using it, DC or no DC.
These are arguments that wreak of developers who don't see the need to worry about things like 'cost,' 'reliability' and 'efficiency' because of the need to be 'innovative' and breaking stuff faster to win the market.
I'm sure it's fine so long as the developers are really competent.
But what about comparing the whole ecosystem?
Can you provide a self hosted granular access permission to your RAID? How hard is it to configure and maintain?
Will your colo deflect a DDOS attack?
When you run your own services, you have to reinvent so much it doesn’t seem to be worth it.
This is the second-level mistake engineers commonly make:
The right questions isn't "Can you do X". Give engineers enough time and resources and they can usually come up with a solution to do X.
The real question is "How much time and resources need to be invested to accomplish X at a satisfactory level?"
And the third-level mistake is to assume that getting something to work once is the finish line. In practice, getting something to work once is just the beginning. Getting it to a maintainable, well-documented, repeatable state is a lot more work.
Cloud services make all of this effort disappear. Type a few commands and it's good to go. Now you can take all of the engineering hours that would have gone into the DIY version and allocate them to working on the company's product instead of reinventing architecture that you could have simply paid for.
Good engineers are scarce and expensive. Using them to reinvent infrastructure that can be trivially purchased for a nominal amount is a terrible move most of the time. Even when it does make sense, the right move is to build the prototypes on AWS and then consider transitioning to self-hosted later if the numbers work out.
Yes.
> How hard is it to configure and maintain?
Very few things are harder to configure or maintain than they are on a cloud service, because if they were, someone (e.g. you) would get frustrated and make them easier, and then they wouldn't be for anyone else.
> Will your colo deflect a DDOS attack?
Ah yes, S3 can handle serving that many requests and keep everything online. But then don't you get a bill for $72 billion dollars?
https://docs.aws.amazon.com/waf/latest/developerguide/ddos-s...
You might be on the hook for bandwidth costs from a more sophisticated attack though.
And yes, running it has a cost.
But it's also has the advantage that devs can run it locally in docker easily. CI can spin up endless test clusters.
And so on.
Obviously you are right that the right way to compare cloud vs non-cloud is to look at the full picture. And that also means we need the context.
Small/hobby project? Doesn't matter. You can run on your own toaster or on Oracle cloud or on AWS/GCP/Azure. Just do what you want, the costs are negligible.
Operating business with stable well predicted size? Again, do whatever you want. If IT is a big part and costs matter, optimize for cost and run it on a few dedicated boxes. If you are not cost sensitive and you want to be one of the cool kids run it in AWS or whatever. (We have a client that exists for 25+ years, reached its optimal size, does some innovations from time to time, but it is basically a new website or app. The underlying backend is the same, maybe they'll replace it eventually. Probably with a complete SaaS and then they'll only need to host a landing page.)
Large multinational company with more departments than sanity? Again do whatever you want, likely you have bigger problems than the cloud bill or the inability to run one more app in your DC.
"Unicorn" startup? Crunch the numbers, do what makes sense. Everyone knows that "Netflix went full AWS" but maybe not everyone knows that they went full on-prem CDN more with their hundreds/thousand (s) of local caches at ISPs / IXPs.
And so on.
I used to work at a medium sized company, and they saved millions by moving to the cloud, and gained much better availability/performance. It wasn't even close, because that medium-sized company didn't have the expertise to operate the service efficiently. They just bought off the shelf stuff from VMWare etc.. Plus, DR meant paying double.
disclaimer, I now work for AWS.
It's also not just cost/availability, but flexibility and scalability. Most high-growth startups would have not been able to scale quickly enough pre-cloud. Facebook is literally the unicorn.
I've been in companies that owned their datacenters and it was much, much cheaper than using any cloud service.
Poorly managed datacenters exist but that's an organization problem. Remove the datacenter and you'll have poorly managed cloud instances and services costing millions.
However, as I’ve eluded to in other threads of this kind; people don’t like to invest in their own tech, an ideal budget would have been around 15%-20% or cloud spend, we could have solved nearly everyone’s pain with that amount of money.
DR does mean paying more, but just so we’re clear: you have to do DR in cloud too, meaning read replicas of database instances and off-cloud backups which auto restore. If your platform costs more money then this is not a good thing and isn’t just baked into the normal cost.
The word competent here is doing a lot of weight lifting.
I know companies in which you have to wait _months_ for a small server to be allocated to your team. AWS does it in seconds
If those companies ever migrated to AWS it would probably still take months to get a small EC2 instance allocated to them. Likely the problem is bureaucracy, not competence.
When I worked in Sony games I had the choice between AWS and the IT department. The IT department was fine, just it was a 6 month lead time for the hardware and they had a tendency to optimize servers they didn't understand.
The problem was that they wanted 6 months and a capacity plan. 2 months to get me a test server. I knew server capacity about 2 weeks before launch. I also needed 5 server class machines with which to test against (which was the end cluster number) and the kicker which was 250 load generators to prove it.
The real kicker is that for many games load peak is the Friday after launch, we really only needed 1 machine a week later, and .25 going forward thereafter.
With the IT department we would have bought $160k worth of servers. And we still couldn't have actually tested things without AWS. AWS Cost $30k the first year and then we actually moved to new instance classes which cost $12k, $6k, and $1k a year thereafter. It was enough cost dropping that it wasn't worth dev time to even downgrade the cluster to a single instance.
None of that includes what IT would have cost to run the hardware. I'd make the same choice every time.
- use the cloud until you're making so much money that you can afford to hire the sufficient talent to replicate cheaper.
fin
But between all the advanced proprietary software solutions and exorbitant egress fees, moving off cloud entirely is rather difficult.
I can see why it's compelling, but for me running my stuff on other people's servers in this day and age is concerning. Like many computing things, it really depends on the situation.
Disclaimer, I barely passed my aws associates cert and have 0 qualifications to weigh in on this subject with any authority. It's just a point I've seen glossed over before. Yeah your data center is cheaper to run, but is it cheaper to replace?
As said in the blog post, with the cloud you will get the same thing that you can pay your inhouse IT for 3 times the price.
For example, lots of people are thinking that by just using 'the cloud', your data is safe / replicated to multiple geographically separated areas. But no, it is not automatic, it needs to be configured like that with the associated costs. If you lose an ec2 instance, it will be lost in the same way as a node in your personal data center.
A bad example: If you are with provide X and they change pricing structure and it impacts you 3 times the cost, you can move it without too much hassle.
Once you embrace all the proprietary and fancy features that your cloud provider has, then you are stuck and moving will be a nightmare.
compute/s3 are easy to move. k8s should be easy to move but I have not tried it myself. Database hosting is easy to change as long as you are using an independent product, not a custom database your cloud provider has.
It is much more difficult when you have AWS/Azure pipelines, AWS/Azure Geo location/manipulation AWS/Azure proprietary scaling etc etc
Terraform is supposed to help here but in my experience with TF consultants it is not at all straightforward not compatible with a lot of AWS/Azure offerings.
Then you are stuck and moving will be difficult and expensive. Which of course is the business plan for the cloud providers.
Personally, I miss the days of monolith web applications that were relatively easy to host on a leased server. I continue to be a big fan of Hetzner and their hosted servers as well as their VPSs are very reasonably priced. Another thing that I like about Hetzner, OVH, etc. is that their bandwidth costs are also very reasonable so moving databases and monolith web applications to a similar service does not have to be a big deal.
I think that each company’s needs have to be assessed separately.
But here's is the point that's completely missing from the article. Many choices in organisations are leaded by two wrong drivers :
- Career risk awareness : like we said in the good old days, nobody was fired for buying Sun Microsystems. Same thing applies for Azure and AWS. You're not paying the bill so why looking for a cheaper alternative that might cost you personally a lot. - CV driven decision : on your CV it's better and more valuable to have 3 years of Azure/AWS than having 3 years on OVH/Hetzner/etc.
Finally, as a leader in an organisation, it's always easier to follow the trend rather than trying to convince your coworkers to follow you to a more "exotic" solution.
He's right. Evaluate your needs and use it if it makes sense. Not a very controversial opinion, I think!
The problem is, how do you evaluate your needs if you aren't an expert in either self-hosting or using managed services (AWS)? I think you should treat it like going to a doctor: get two opinions from two different senior professionals in two different specializations. And definitely make an assessment based on real numbers. Try to get ballpark figures from similar-sized businesses about their costs (capex/opex, infrastructure, staff) and requirements (expertise, time-to-market, FRs/NFRs, regulations, etc). Building a business is a huge thing, and how you use technology can either be a hindrance or an accelerator, but it has to fit your use case.
Datbases and virtual machines tend to be expensive in cloud providers. Steady state workloads.
Object storge tend to be fairly cheap.
There is also cost of vendor lock in if you use propritary cloud technologies such as databases.
You can build open source infrastructure with hybrid solution. Probably less risk of downtime.
Choosing a large cloud provider is often much greener than doing your own or using a smaller provider.
Carbon is a large cost.
GCP has been carbon neutral for over a decade, AWS and Azure have made big public commitments.
Sourcing clean energy for 1000s of servers is not always easy.
Your in-house IT department knows not to take a risk the week before your big product launch. Amazon and Microsoft simply do not care about your product launch at all, and probably aren't even aware of it.
You're going to pay for infrastructure and IT staff either way, so why not pay for IT staff and infrastructure that prioritizes your business needs and not their own business needs?
the same with being able to give your customers flexibility in terms of when maintenance happens. this is a big deal in b2b space, especially with long running client relationships because it gives customers some form of control they never are going to get back in the public cloud unless they are insanely large customers.
If you look at prices from Oracle Cloud ( ignoring whatever feeling you have against it ) than all of a sudden it is extremely attractive.
Could someone elaborate? Is this saying something free like OpenStack would be better than something paid like VMware? Is that really common knowledge?
The "cloud" can just be a blank Debian box on Digitalocean where you have root access _or_ it could be some obscure managed AWS service where all the technical days are abstracted away behind a REST API.
