Active-active, five nines, fault tolerance. Hard stuff. But managing on-prem is no harder.
This is what we're paid for.
I am paid to relax on my holidays because I know my team and I don't have to drive to a colo to swap out a failing line card since I realized time is worth money and people quit jobs that take up too much of their time. I can A/B test (something on-prem guys NEVER get the luxury to do) so outages just don't happen at all (fingers crossed).
I have rarely met someone happy with their on-prem DC deployments, but after I moved to the AWS world it's just crazy how backwards it is to be anywhere but the cloud.
Honestly just sounds like the environment you describe has greater organizational issues not related to on prem vs cloud.
It's almost as if you feel the cloud is something magically different, it's actually just servers on racks owned by someone else.
You can own the same thing if you want and do everything exactly the same.
(See e.g. Oxide)
Compare something like rocketry or chemical engineering with running an on-prem DC. I don't see what the complaining is about. It's still a luxury compared to what other professions have to deal with.
Most people are not paid to manage infra, they are paid to talk to customers, ship features, fix bugs, and other "core business" items; just like most businesses don't build roads, they pay taxes and utilize them because the cost of doing it themselves for their preferred traffic patterns would be much more than they could justify (for now.)
But when your local IT goon says its going to be 8 months to procure the next set of hard drives for your next order of magnitude, it's a real problem and you have real money to invest in solving it, just not owning a data center money.
I worked for a company and would do it again that did the colo route, and it gave immense cost savings compared to public cloud, taking on risks that you can't do elsewhere. Before they started investing in having folks take care of the infra as a raw startup, it was just some servers and some desktop unmanaged switches. But that gave the company breathing room to survive as the business model probably didn't work without it. But also had a reputation for unreliable service.
I've also built the five nines infra at telcos, and yes you can do it with average engineers, but it's going to be time consuming, slow, and expensive in costs and labor. To allow 26 seconds of unplanned outage a month, you're going to be testing every firmware update for every piece of equipment on an ongoing basis, and practicing every operation and change as best as possible. And you need the scale that you get that 26s by having most outages only impact a subset of your customer base, otherwise you're going to blow that outage budget fast.
