Other than that, yes, billionaires have some advantages that us ordinary mortals do not have. But to stick with Elon Musk, he wasn't born a billionaire (but he did go to a private school, but that's not unheard of in SA).
Once you're past $1M net worth or so the game gets easier, sometimes a lot easier. But $1M vs $1B doesn't make much of a difference if the business tanks and you've invested all of your capital in your own business.
And that's the main reason you don't see billionaires usually spending all their money on their own projects: they spread the risk around to avoid possibly losing it all.