This “my tax dollars” argument is so facile. Does this mean then that your employer gets to control your actions because it’s their dollars funding your actions? The money changed hands - it’s the governments.
The underlying principle is the rule of law and the Constitution codifies the powers of the government with legislation codifying more details. That’s why the government is accountable to you, not because of your tax dollars. If you are a citizen who doesn’t need to pay any taxes, the government should be as equally accountable to you as to the very wealthy because of the rule of law and everyone being equal to it.