"If you don't like it, you can leave," is an argument with limited effectiveness. First, unless users spend a lot of time talking about servers, they won't know where to leave
to. Remember the great migration, where we were all saying it didn't matter which server you picked because they all federated? It turns out it does, and they don't - and all the new users who were anxious about joining the "right community" were right!
Furthermore, if users did spend a lot of time talking about their admin's moderation, I guarantee it would come in the form of a lot of complaining, maybe even entitled-sounding complaining, without much deference to the admin's right to do what they want on the server they are, admittedly, paying for. There is no way for the community to figure out who's servers are good to be on without continually hashing out who is doing a good job, which means, in some cases, disapproving.
As for contacting your admin to say you think they're being petty... That is not likely to work! You'd probably just get yourself banned before having a chance to migrate your followers!