We reduced our server costs by moving away from AWS (opens in new tab)

> have to hire

But [to state the obvious but sometimes overlooked] you don't just point an AWS account at the company git repo and walk away.

There's a lot of work and expertise needed to keep AWS setup up and running, so you already have to hire people.

At a modest size startup we already have close to ten people DevOps team to manage AWS. That same size team could easily keep bare metal servers running. At our scale it's still a bit cheaper to be on AWS, but not too far in the growth curve it'll start to become cheaper to be on bare metal.

AWS knowledge and engineering doesn't come for free either. People have built whole careers and businesses around it.

> If they saved $800k per year, and they have to hire four additional ops engineers to run it at a cost of $400k per year,

I guess you've never done it yourself?

I'm not sure what those 4 engineers would be doing. You purchase a few servers with 5 year on site warranty and remote management, you take a couple of days to install them in some racks, and you never visit the site for 5 years. If they break, the manufacturer sends someone to fix them on site. The rest of the time you administer them remotely - just like you would AWS. If you want VM's install proxmox.

Whether it's cheaper than renting bare metal in a data centre is an debatable - colo costs a small fortune where I live.

But they aren't comparing it to that. They are comparing it to locking themselves into a closed source system that need specialised expertise to run, versus administering an open source Linux system that even the people running EC2 should be very familiar with. Most of the stuff AWS provides has open source counterparts - hell a whole pile of it is just open source they wrap a proprietary API around and charge you for.

10 times sounds like a lot, I'm guessing it's really less. Even so I come form the camp that shakes his head in disbelief at what people will pay AWS for basic VM and storage services, dressed up in a fancy API and marketing.

Exactly. In the "old IT world" we call this TCO = Total Cost of Ownership.

huge salaries + ten "ops engineers", lol.

I know *data centers* that run on a few naive 20 yos admins/technicans + 1-2 "engineers" and all of them combined do receive salary of $5-10k/month in east eu

You're not including reliability and availability projections, and the intangible cost of transferable skills wrt infrastructure. (ability to hire sufficiently skilled people to run it)

Hiring a person to do 'docker compose up' for you is orders of magnitude cheaper than whatever AWS-specific knowldege is needed to not have AWS crap its bed.

That also just sounds like Uber had hired crap talent....

and they had trouble moving workloads to the cloud because bringing up new capacity was a giant set of circular microservice dependencies

> The whole thing would work just fine on a couple of FreeBSD servers

And you need dev servers. And a way to keep them in sync with production because you can’t just create a branch and provision a bare metal server on the fly. And you need a deployment process to get code from dev through test, UAT and production, and this process is lengthy and fraught because you can’t just fire up a new instance and switch the elastic IP, you actually have to deploy code on the physical machine and make sure there aren’t any configuration differences between your test and prod environments that’ll cause ‘it worked before’ problems. Developer productivity tanks, people are afraid to make changes, any pretence of agile/devops goes out the window and eventually everyone gets sick of the crusty old server and decides a complete rewrite is needed.

> I'd argue that 90% of the SaaS doesn't have SO's scale.

While Stack Overflow is popular in the dev world, not so much outside of it. The show they’re only serving 300 req/s. 19 servers to power the entire stack as well. I wouldn’t call this efficient.

Yes - but the FOMO industry is too strong.

> At what point do you have the time/money/confidence to invest goodness knows how much in a data centre with space to grow, to purchase an enormous amount of capital to have it all installed etc. the building alone could eat that first years saving easily.

Co-locating has no capital investment other than hardware, and is pretty cheap.

A 40U rack of compute charged as equivalent ec2 instances has a retail price easily of hundreds of thousands, if not a million+ USD per year.

Suppose each U has a $10k capital cost to make the numbers round, that is $400k in capital.

All this to say is that I don’t think capital is as big a factor as you might think.

> fix some power issue

> large air-con system

You wouldn't usually jump from AWS to buying up real estate to build your own physical data center.

A sensible first step is to rent a rack at a colocation facility. They handle power, cooling, redundancy, physical access for you.

3 people at 0.10% of their time is already 24K, assuming an 80K salary for each of you. You don't mention patching systems, or the time spend replacing the hardware racks every x years. It is very easy to underestimate the cost of maintenance.

> As to "cloud agnostic", don't believe this bullshit.

> It is the same story as with SQL. Trying to use frameworks to keep your app DBMS-agnostic but then nobody ever migrates their apps to another DBMS.

I agree, I can't tell you how many hours I've wasted trying to keeps something (theoretically) cloud or DBMS agnostic, how many problems it's caused, and at the end of the day we could never "easily" move to a different cloud or DBMS without large rewrites.

I built on top of the Serverless Framework and I am constantly kicking myself for doing so (eventually I'll move off it for my main project). It's the worst of all worlds and in the Serverless Framework docs they even have sections for "GCF" vs "Lambda", if I have to write my functions/declarations differently then why am I using your layer on top? I know you get a few things for "free" with the Serverless Framework but once your project grows you run into all sorts of issues that are a PITA to work around. The truth is I'm not leaving AWS Lamda for GCF or whatever Azure has, I quite enjoy my "walled garden" and using an abstraction layer only means I get the lowest common denominator or a headache when trying to do something that is normally easy if you've fully bought in.

> It is the same story as with SQL. Trying to use frameworks to keep your app DBMS-agnostic but then nobody ever migrates their apps to another DBMS.

A small nit, I think this is partly because we've been burned before. Same idea as backups, if you don't restore from backups, you don't know if you have good backups. Similarly, if you don't use certain code paths, you can't tell if those code paths are sufficiently bug free.

I am thinking of Drupal. Pretty much everyone who uses Drupal uses MySQL/MariaDB as far as I know. I think Drupal supports Postgresql but nobody I know uses it because nobody they know uses Drupal with Postgresql. I don't know anyone who uses drupal with sqlite on production either.

Cloud agnostic has its place if you have enterprise contract with cloud provider.

If you have 100s of projects cloud agnostic infra is good. If for example Amazon buys your competitor and now you get "new contract coming up".

That said I agree "most projects fare much better without" - a lot of people don't understand they are not in position where they would benefit from that.

The SQL example is a great one. I worked on a project where we used MySQL but then used SQLite for unit tests and integration tests. We technically wanted to be able to swap out the RDBMS to something like Postgres but in practice it was never going to happen. Over time it started getting complicated as some stuff we used in MySQL wasn't supported in SQLite. Eventually I just ditched SQLite and changed our testing to start a real MySQL server and destroy it after testing is complete.

> As to "cloud agnostic", don't believe this bullshit.

A notable exception to this is of you have clients whose IT departments require you to work on specific cloud providers. Although, there's tools that help abstract deploying infrastructure to different cloud providers (eg terraform, pulumi), but they still require some familiarity with the providers. With all that said, overall I agree with your sentiment

>It is the same story as with SQL. Trying to use frameworks to keep your app DBMS-agnostic but then nobody ever migrates their apps to another DBMS. And for various good reasons: you hired your staff for their expertise with X, they are used to it, so now going for Y will usually be far higher cost than the benefits.

Yeah, I never got this. If actual, for really real SQL is necessary for your application, then you need an actual, for really real DBA. If it's just a really nifty flat-file interface that "just werks," go with whatever DB engine your framework is most tested/used on.

I get that sometimes you're interfacing two different things, one of which is on one DB engine, and you'd like to use the same software. That's fine. But DB agnosticism has a cost like anything else.

If you can be truly cloud-agnostic, then there is a high probability you can be cloud-free.

Not being cloud agnostic forces you to use one single cloud provider, that is not necessarily the best in everything.

> It is not like there is a good chance Amazon (or Microsoft or Google for that matter) will suddenly pull a significant price hike or another move out of step with other vendors.

I feel like a variety of other circumstances can come to pass, which would negatively affect business continuity, here's what a lazy search turned up:

  "My Google Cloud was suspended too" https://news.ycombinator.com/item?id=32571055
  "Google suspended our domain out of the blue" https://news.ycombinator.com/item?id=32798368
  "Tell HN: Google Cloud suspended our production projects at 1am on Saturday" https://news.ycombinator.com/item?id=32547912
  "AWS account was permanently closed because it was suspended for 90 days" https://news.ycombinator.com/item?id=31571538
  (probably happens with Azure and other smaller platforms as well, e.g. Hetzner, DigitalOcean, Vultr, Scaleway and so on)

That said, most people won't care to put in the work for a multi-cloud/cloud-agnostic setup, since most projects just aren't as important to warrant that much effort. And the ones that are can probably also just talk with the cloud providers through some representative anyways due to spending $$$.

I'd argue that it's good to build on common standards, like OCI containers and the wire protocol of some common DBMS like PostgreSQL or MySQL/MariaDB: so that you can replicate certain parts of what a managed service does in a container locally, for development and testing. In most cases it won't matter that the managed cloud offering has some clever engineering underneath it and scales better, as long as you can check whether your SQL executes as it should and none of your seeded/anonymized test data breaks.

I actually had this project with Oracle DB that ran horribly when the instance was remote and you really couldn't do migrations well without breaking things for anyone using it, because the apps using it were written in a way where hundreds if not thousands of SQL queries were done just to load some data to display a page. Which is passable (if you don't care) when the DB is running in the same data centre as the apps, but absolutely horrible when these many smaller SQL queries have the full network round trip between them, especially when the app initializes data sequentially (N+1 problem). Ergo, the only way to work with something like that is to setup a local database (e.g. Oracle XE) and work with it, after importing a baseline migration/doing automated migrations.

The same largely applies to any other DB with sub par application architectures, as well as many other services (e.g. MinIO/Zenko instead of S3, so you don't need to actually upload 100 MB files to test whether attachment logic in your app works, if you can run them locally).

As for software that should be compatible with multiple DBMSes: sometimes it makes sense (e.g. something that could be used by your customers in a self-hosted setup across numerous different setups, like Zabbix or Nextcloud), but most of the time it would negatively impact how easy it is to develop code for your app (e.g. having to rely on ORM and their abstractions like JPQL) and testing everything would usually take more effort.

There's a lot of other businesses that blindly do cloud for no good reason at all. I calculated the forecast of aws vs own infrastructure (aws is what was being used) for an ML company doing major compute backfills. I did that at the height of the GPU price craze and the whole move would have still amortized itself after just 6 months. Why people stick to aws for that sort of stuff is beyond me. If you ever need to spin up VMs on aws rather than on your own metal, you can always do it then. It boggles the mind that people want to throw away millions to get worse performance and higher error rates.

You think AWS doesn't have enough lock-in now? And yet maximum profits come from keeping customers, not driving them away.

There's no such thing as a managed service so proprietary that it can't be essentially cloned by another provider. I remember people making the same claims about S3 long ago, and yet there are multiple providers of S3-compatible file storage services now.

Microsoft's Azure provides migration tools for people wanting to leave Amazon's AWS, and they are not alone.

Amazon tends to be (in my opinion) a few years ahead of everybody else, and different competitors focus on different areas of competition, but migrating away from AWS isn't really as hard as the "proprietary" label might make it seem.

This is often mentioned scenario but highly unlikely as impractical it is.

First, AWS is already highly profitable, they aren't dumping where this step becomes a necessity.

Also, any such move would dry up future growth from adding new customers, which is suicidal.

Existing customers will flee as quickly as they can, and new customers will look elsewhere. In reality, they'll never charge more than their public rate card, and that rate card has to be competitive with the competition.

That's definitely not true for “all of those managed services” — you can't replicate CloudFront on EC2, for example, and it'd be extremely unlikely that you could build your own S3 replacement on top of EC2 storage without spending considerably more on operations than you save. On-premise storage is cheaper but you're still going to struggle to see a cost savings unless you either don't care about reliability or availability and/or buy your data by the petabyte.

For the other services, it's still going to depend heavily on your usage, staff time, and operational efficiencies. For example, if you replace SQS with RabbitMQ you need to manage multiple EC2 servers for reliability but those servers might be significantly underutilized based on your traffic levels. Whether or not you save money depends on how much you pay your operators and how many messages you use: if you use less than the free tier's million messages per month, there's no way to beat it. Each million requests costs $0.40 or less, so if you pay your ops person $50k annually (haha) you'd need to be somewhat over 120 million messages per month to pay for them to spend a single hour on O&M even before factoring in your EC2 usage.

Or any kind of demand spikes.

If you are using lambda and suddenly need to run a million lambda you just can. If you want to have the ec2 support to run a million lambda at some time you are going to pay for a lot of sleeping computers.

Maybe if you ignore the costs associated with operating EC2 or if you have a steady, predictable workload. Or perhaps if you have enormous (on the order of FAANG) scale to absorb those operational costs.

Are you including the wages of the extra people to support building and managing such services?

What I don't understand is why anyone would spend, precious and expensive engineering resources on coding to a proprietary API owned by one of the largest companies on the planet to whom you are nothing.

Ok so you save money on the monthly bill but what happens when Amazon decides they want to enter your market? What happens if they decide your service is too controversial and they kick you off?

If you were deploying your own services to EC2 instead of using AWS's own services you could at least setup shop elsewhere with just a bit of work.

To me it is antithetical to building a sustainable product. People just hoping their startup gets bought and then it is someone else's problem.

So what should you use for let say, a backend API service? Don't tell me beanstalk/fargate/etc, because they're actually more expensive.

I think that statement about EC2 is a little too strong. You definitely shouldn't be managing individual EC2 instances in most cases, and you probably shouldn't be deploying directly to EC2. But if you're running EKS (Kubernetes) or ECS (non-k8s containers), then you're probably going to get more bang for your buck with EC2 nodes than Fargate nodes, especially if you have a large cluster.

Using Karpenter for AWS has been a game changer. Set your instance family designations, set up a compute savings plan, and let it ride.

In most businesses I've worked in:

developer costs >> infrastructure costs

An AWS large server is around $500/year, which is about 1-2 developer hours (with taxes, overhead, etc) at the cost scales last time I priced this out. That's crazy expensive in the absolute, but if it saves a couple of hours, it makes sense.

PaaS providers cost even more. I've gone with those in the past, since it basically eliminated dev-ops. The major downside wasn't cost, so much as flexibility.

Dedicated servers start to make sense for:

- The very low end (e.g. personal use, or hosting something long-running for a small business)

- The very high end (e.g. once cloud costs start to hit hundred of thousands of dollars per year)

On the very high end, cloud providers will often cut a deal, though.

My problem with AWS, recently, has been reliability. Servers crash or have performance degradation a bit too often. That leads to developer costs, and might be what pushes me back to dedicated.

Lots of web host also have S3 compatible equivalent offerings. They typically market it as "object storage".

That same logic works up and down the line. If what you need is disk storage, then you are limited to dedicated hosts that provide something akin to S3. There are some! But that's also true when you want a database that isn't sqlite. Now you need a dedicated host that provides something akin to DynamoDB, and will manage it for you. Then you decide you need queueing, and you can either install and manage that yourself or look for a dedicate host that provides something akin to SNS/SQS. And so on...

Right -- that's the one implementation detail in the OP that was interesting. It sounds like they ultimately used MinIO to replace S3. I've seen people use Ceph, but it's apparently a nightmare to operate a Ceph cluster. If you're on k8s I think the "cloud native" way might be Rook, haven't looked into that. Anyway, running an object store is painful.

Their notes here are a bit vague:

> When the migration reached mid-June, we had 300 servers running very smoothly with a total 200 million cached pages. We used Apache Cassandra nodes on each of the servers that were compatible with AWS S3.

> We broke the online migration into four steps, each a week or two apart. After testing whether Prerender pages could be cached in both S3 and minio, we slowly diverted traffic away from AWS S3 and towards minio. When the writes to S3 had been stopped completely, Prerender saved $200 a day on S3 API costs and signaled we were ready to start deleting data already cached in our Cassandra cluster.

> However, the big reveal came at the end of this phase around June 24th. In the last four weeks, we moved most of the cache workload from AWS S3 to our own Cassandra cluster. The daily cost of AWS was reduced to $1.1K per day, projecting to 35K per month, and the new servers’ monthly recurring cost was estimated to be around 14K.

It says (briefly, in passing) that they used Cassandra to implement the S3 API for their nodes, but maybe just to replicate the S3 API that they were previously using? That's an interesting choice I'd not heard of before. Perhaps all of their individual files are quite small?

Then they moved to MinIO, which would be the S3 equivalent that you are looking for.

Might not be so obvious when summing all the cost (apex, opex). Not mentioning the difficulty to find experienced people to maintain it. Unless you are really big of course

Having your own data center is almost never cheaper nowadays unless you have very specific hardware needs. And it is far, far, far slower to build your own data center than to just start using a cloud provider.

There are also affordable dedicated server providers with DDoS protection, like OVHcloud.

My hosting provider does and successfully too. Maybe it's time to move providers, I would.

Having an issue pasting at the moment! Just search Trillion Dollar Paradox Casado.

You’re going against the party line.

Our solution targets the physical network in off-cloud, edge and distributed cloud use cases.

The physical network is most likely the absolute root of trust and attestation. Physical devices gain their personalities from what network they are connected to.

Physical networking is still the biggest impediment to operating on-prem infrastructure. You need to hire very expensive engineers, deal with a bizarre purchasing paradigm dictated by a few large companies (Cisco, Arista, Juniper, etc) and the management of these devices is pretty close to being from the Stone Age.

So we’re looking to fix all of that with a pure open source play, using components that everyone is familiar with.

The end goal is that a Devops person with basic network skills should be able to build their own physical network for modern cloud native applications, with very low levels of friction but similar cost efficiencies as cloud operators like AWS, Azure, GCP.

Absolutely! We’re hiring for a number of spots including dev/systest/kernel-dev/k8s.

The whole idea that since they have economies of scale, they must pass those savings on to you is a misunderstanding of how companies work.

Realistically, a company will charge slightly less than whatever your alternative is, and they will provide products that restrict the number of alternatives you have.

AWS, GCP, and Azure provide several differentiated products that lock you in to higher prices on everything else. Effectively, these three have an oligopoly on "highly differentiated cloud services." They are only competing with each other on those services, and they are competing with "servers plus ingress/egress costs to our differentiated services" on their commodity products. That is the real reason why AWS egress costs are so high. It prevents you from picking and choosing where to buy each part of your cloud footprint, and locks you into buying AWS. Bandwidth costs are what keep you inside the AWS/GCP/Azure walled garden.

Lower-end providers like Linode, DigitalOcean, Vultr, and Cloudflare have parts of the differentiated offering, but not all of it. These people will have lower prices than AWS/GCP/Azure since their offering is less differentiated, but they will still charge you more than you would pay by renting a server, since they offer more products.

Finally, hardware rental providers like Hetzner and other operators are competing directly with you buying the hardware and paying for datacenter space and bandwidth. Datacenters typically charge a premium for power, even when they are in areas with low-cost power.

Notice that none of these companies are competing with large-scale server buyers who have the same hardware/bandwidth/power costs that they do. As such, they do not need to pass the savings they get onto you. That is where they get their profit.

Bandwith is obviously extremely expensive on AWS. But even EC2 is easily 5x as expensive as renting a dedicated server. Of course this is a deeply unfair comparison as dedicated servers don't have the flexibility of an EC2 instance and the integration into other services. So if those matter to you then the cloud is certainly worth it. But you are paying a lot more for CPU, RAM and storage compared to other options. You are also getting a lot more with AWS, but you're only saving money if you can fully exploit the flexibility of the cloud and scale the resources very well. And even then if you need a lot of egress you're screwed on the costs in any case.

AWS's bandwidth charges are highway robbery.

It's my impression that AWS competes with the EC2 instance cost (as it that's what new customers look at), and the bandwidth/storage only becomes apparent when you are locked in.

Really what AWS should be is one or two phases of service maturity: dev infrastructure and experimentation is phase one, phase two would be the "its a couple servers" production, BUT: with a scale plan for phase 3 being not-AWS.

Having a mature/battle tested phase 2 --> phase 3 should be a market advantage in the modern business landscape, but that's also a post-acquisition/exit phase, so none of the presumed target HN crowd care about it.

There are extensive articles and public knowledgebases on various technologies and architectures, but there is effectively nothing on AWS independence out there, even though almost every mature organization will need to face it.

That is true of any datacenter, but any offering from AWS/GCS/Azure/Digital Ocean/Vultr is an order of magnitude more expensive than Hetzner/OVH/Scaleway clouds, and orders of magnitude more expensive than if I just get a handful of dedicated servers and run my own Minio servers (to replace S3) and manage my own databases and build my own OpenStack /k8s/Nomad/Swarm cluster.

That is why VPS providers are cheaper than running your own hardware, at least assuming you're not always fully utilizing it. Cloud providers on the other hand, they don't really sell the resources - they mainly sell convenience, and that seems to be worth a lot.

Aws bandwidth costs charged to consumers are a notorious ripoff.

> which certainly makes it less expensive to use cloud

AWS would pass that difference to Amazon shareholders than passing them to you.

The furniture example is great. If I wanted to build an Ikea table from scratch, I could not do it for less than the $20 they charge, given the cost of materials available to me, and the fact that I can’t buy a cup of paint for $2.

imo this is the right takeaway.

You see a lot of posters commenting that it isn't obvious that it's more expensive, but that's a large part of why I said I'm flabbergasted at how many people don't find it obvious.

It is obvious, someone may still choose to pay it because there's a natural tradeoff, but it's absolutely obvious.

I suspect most developers don't find it obvious because they've never dealt with actual servers themselves.

It's akin to arguing changing the oil on your vehicle is more expensive than paying someone to do it. No, it's more convenient, and that may make it worth the price.

It would be shocking if, in the long term, there were not cloud providers with lower prices than any but the biggest companies could achieve by themselves. AWS will have to work hard to keep giving customers a reason to pay the premium.

Sure you can. Ask one of your devs to spend X amount of time on something. Might not be an exact 1 for 1 or 0.1% but it still beats the many many hours you'd waste architecting around cloud-specific issues and nuances. Or the cost of dealing with the friction that comes when you want to do anything with cloud in a manner that isn't cloud-supported.

And, importantly, you can turn off your commitment to spending in a second.

assuming the money isn't an existential risk, that's absolutely true.

I was commenting on the expense rather than why you would (or wouldn't) want to use cloud.

I’m from Hungary, and I have to tell you, you are WAY off with those salaries.

As a senior software engineer, you can make maybe $35k a year, before taxes, if you are good. You can make it $50k if you are very good.

2 years ago I was making $23k (yearly, before taxes), before I moved to Canada and started working at Amazon for $170k usd.

Europe, especially the eastern parts of europe has an extremely cheap workforce.

Fair enough, I assumed typical San Francisco/Silicon valley model.

Not really. You're paying extra to not have to think about how many static resources to allocate in advance.

For a mature business, that tradeoff probably isn't worth it, but for a startup, the opportunity cost is too high. Spending a lot on cost optimization doesn't make sense; you are better off spending on growing income.

Just as a basic sanity check, they claim to store 560 million pages of pure HTML. Internet Archive estimates the median HTML weight of a page to be ~20kb, which I fudged to 30kb in order to make a liberal estimate. Multiplied by the number of pages that would amount to approximately 16.8TB of data stored in s3. According to the AWS pricing calculator that would cost between $300-400 per month, and that's in s3 standard tier pricing with no intelligent tiering set up for less often retrieved data. I didn't bother calculating the compute cost for their rendering because I don't really understand how they do it, and also because the storage estimate was so far off of their claim that it didn't seem necessary to bother in order to refute the claim. I appreciate your challenge for additional evidence, and this is what I have to offer in response. I won't pretend like I did an exhaustive scientific study, this is more back of the napkin type stuff FWIW, but here it is.

well the problem with the article is basically that they left out a lot of important detail, like which bare metal servers, how many, where do they host now, did they use cloudfront what about cloudflare did they use an edge cache? what about reducing costs by killing stopping unneeded resources? a lot of their workload looks dynamic. it's also fishy what they wrote here:

> After testing whether Prerender pages could be cached in both S3 and minio, we slowly diverted traffic away from AWS S3 and towards minio.

if they served directly from s3 that would be, stupid?

> In the last four weeks, we moved most of the cache workload from AWS S3 to our own Cassandra cluster.

is also strange. it misses a lot of detail but it does not look like they just migrated away from s3...

(looks like their new hoster is hetzner, from service.prerender.io )

If you want rocksolid dedicated servers of utmost HA and reliability, look no further than Equinix Metal: https://metal.equinix.com/product/servers/

They run datacenters for other datacenter companies. Most people have never heard of them because they're a $6B/year revenue beast that runs behind the scenes; but they also offer direct metal servers.

> The cheapest provider there is probably Hetzner, but then your servers are in Germany

They have a US DC now. Though it only gives cloud servers. However even those are still way cheaper than any alternative.

In my experience with modest specifications, you can source servers from tier 1 vendors (Dell, HPE, etc) with 5 years of support for less than a 1 year commitment for the roughly equivalent AWS or GCE instances (without prepayment).

The monthly opex to house and power and cool those servers isn't _negligible_, but if you're doing back of napkin math comparing MRCs to Cloud you can just deduct the costs from their bandwidth charges that have been marked up 10,000%

If you want recurring monthly costs for capital devices there are companies setup to do that for you.

They lease you the equipment. Yay.

by servers i think they meant microservice apps/binaries/vms/containers, etc not physical machines. I also want to know what physical hardware they purchased, where they racked them, how much colocation costs, etc, etc and they go into none of that.

May be it was not large. Say for one project DB server on Hetzner with 160GB RAM, 32cores/64theads,2x3.84TB NVMe + 2x512GB SSD costs ~ 240$, hosting mid-sized ~ 1.6TB MySQL DB. On managed RDS it was around ~ $2200/month when I checked.

Even if you have 2 such boxes for master/replica, it's close to 5x savings.

We haven't run into the number of servers issue. For S3, we've switched to Wasabi which very nicely uses the identical API.

You can rent servers if you don't want to bother with this. The choice is not only between doing everything yourself and the cloud, there are a lot of options in between.