If you don't care about that, then the balances changes. If you're OK with 1 on prem server, you can just buy a AMD or Intel workstation, and tweak the hardware config (e.g. RAM, kernel, software, etc.), BIOS to your liking.
All cloud providers have pretty poor uptime records. Unless you set up multiple geo zones (extra complexity) your app will go down when aws-east does.
This isn't hard to achieve with reasonable quality hardware and reliable power (which likely includes some amount of UPSes).
The scaling is nice, but as another poster said, 2x swing isn't enough; it would be more effective to buy for your peak and just let them idle on non-peak. If the swing is 10x, then cloud might be more economical. But you also need to build your system differently for frequent cluster changes, vs a stable cluster. Having a short cycle for hardware aquisition is nice, of course, if your load changes quickly.
As consumers, we really need more independent benchmarks.
Reading bullshit like "AWS FOOBAR MAKES RUNNING MACHINE LEARNING IOT FINANCIAL MEDICAL APPLICATIONS 20% FASTER" doesn't help me to architect systems.
I was looking for side project ideas, thanks for providing one.
