Because when all you have is a hammer, everything looks like a nail.

We have become so DevOps and cloud dependent that everyone has forgotten how to just run big systems cheaply and efficiently.

Because that high-compute job needs to be distributed on many, many machines, and if you're using cheap preemptible instances you have to handle machines dropping off and joining in while you're running that single job.

It's definitely not something that you can launch manually - perhaps Kubernetes is not the best solution, but you definitely need some automation.

because its not a one time operation.

also, how else do you sensibly deploy and manage a multi-stage programme on >500 nodes?

I mean we use AWS batch, which is far superior for this sort of thing. SLURM might work for real steel, as would tractor from pixar.