You can rent GPUs by the hour and there are (limited) serverless options too. So this question makes a gigantic difference in cost.

>of course if I'm launching a public product it should be available 24/7.

Generally yes, but not a given. In some context like data processing you can accumulate tasks and run them in a batch.

e.g. OpenAI just launched batches...50% discount but you don't get instant results, you get them somewhere in next 24h