Is there anything to prevent a person/group from setting up a Mastadon instance with a charge to cover hosting, admin, & support costs (something like businesses charging for service on Open Source software support)? This could both make it more stable and sustainable and be a barrier to bots/trolls.
Nope - Mastodon supports invite-only which can aid in this sort of set-up; I'm sure other platforms do as well. And if one runs a close-knit community (which takes more than expertise and infrastructure), donations or something like a Patreon scheme can work.