But for most companies it never "comes time to scale". Especially if you have 100% dedicated hardware and you can start out with a 64 core db server with 120GB of memory and a matching web server pretty much for free.

You need to "scale" early in a cloud environment because you are allocated a fraction of the hardware and someone else on that same hardware is using all the resources. AWS/DO etc boost their margins by packing as many vps's onto one physical box as they can. That's why on a VPS your 95th percentile response time is 1000x your 50% response time. All you can do is "scale" and pray enough of your connections are hitting the boxes that nobody else is using right now.