The Earth is unique and precious, but the vast out there - we cannot do our worst even if we tried.
So does every other living thing, in fact. It's not like every animal and plant is a nature conservationist and humans are the mindless exploiters - it's actually the other way around. We talk about restraining ourselves because, unlike the rest of the nature, we can choose to be selfless, or at least thing longer-term and at ecosystem scale.
Also, you're absolutely correct. Earth is a gem. Everything else in space we know of is just rocks and deserts and clumps of gases. And there's so much of it that we aren't going to make a dent even if we rode the exponential growth for a while longer.
Meanwhile asteroids are completely uninhabited and dead, and there's millions of them. They have all the same minerals the Earth has, and often much closer to the surface. We could even just mine a few and easily replace all the destructive mining on Earth.