It’s not about being unable to view the issue except from that one perspective. It’s about having an aversion to mass suffering, and recognizing that this group will be subject to it.
You’re basically saying, why are you so worried about all of these people who will have their lives destroyed when there are a bunch of other people who will be totally fine? I hope that when it’s put that way, you can see how ridiculous it is.