I'm not talking about the quantity, I'm talking about a list of who upvoted and downvoted a specific comment/post.
Also what you've described happens anyway. The fact that a number isn't displayed doesn't really matter when comments are ultimately ordered by score. When someone writes a comment that doesn't adhere to the current groupthink, they get downvoted. When people are downvoted too much they delete their comments, stop commenting, or through pavlovian reinforcement change their opinions until they get positive points.