But you can't avoid trading with a ERC20 token, because they are designed to be tradable. Any exchange can add the token to the supported list, and the users can immediately send the tokens to the exchange and buy and sell them.
Unless these are not really tokens and the "user/owner" can send the tokens without a centralized permission.