In my experience doing detailed projections of exactly something like this - with racks full of GPU compute power - the infrastructure has paid itself after one quarter, maybe two, depending on what volume you're at. There very rarely is a reason to use GPU compute on the cloud - and advantages start as quickly as with just one single GPU.

I feel that a lot of posts like this might be under-representing the true costs of running your own hardware. I was only tangentially associated with a large-ish operation and I can tell you that there are loads of things that take a lot of time but are often over looked. Things like detecting and replacing bad hardware. HDs don't last forever and when they go bad it's not fun; especially if you have to source some specific model for your 7 year old server platform.

Understanding your licensing and warranties is another huge cost that people don't take into account. We used to spend hours and hours figuring out if we could replace systems and what it would cost us.

Finally, you have to dispose of all that hardware when it gets too old or out of warranty. If you've never had to do that you probably have no idea how hard it is to do it correctly and so that it satisfies your SOC2 auditor.

All of these things (plus more) really add up. It's not just purchasing the components and installing them in racks. The management is probably even more expensive than the hardware.

And all of these problems go away with a cloud provider.