No you literally aren't. You are you. The conversation is the conversation. Those are two distinct things. Nobody can ask you to be mindful of the quality of other people. It's trivial to just avoid interminable discussions about voting.

The most telling thing about this is that nobody ever demands explanations for upvotes so it's obviously not because there's some real belief these explanations would make the conversation better. It's just that being downvoted feels bad. But really, at worst, you'd eat -4 points here or there. Best is to just put on your wizard hat and Epictetain stoic robe and move on. And this isn't merely a good idea - it's the law.