In terms of "starting a business" I just had to register for an ABN ("Australian Business Number"). That may not even be a legal requirement but IIRC it had tax advantages. It's a couple years since I did it but as I recall I did it all online in a couple of hours and it didn't cost much. I then just include that number on my invoice. Very civilized.

At some point I need to register to charge GST but I'm sure I can do that online too. At no point have I had to visit a government office or get a bureaucrat to sign anything to start a business in Australia.