I agree with you more than you'd think. I'm aware that my open-borders belief is a radical one, and I accept the dramatic implications and knock-on effects.
But the vast majority of people accept the legitimacy of restricted immigration. IMO the line of moral revulsion they draw between your view and their own is largely illusory. Once you've accepted that you're selecting immigrants, it seems nonsensical to me that your selection criteria should consist of a) rigid and horrifyingly Kafkaesque processes for 99% of countries and b) an inconsistent drip of unselected migrants from an adjacent country (through a process that puts them at substantial physical risk).
Frankly, I'm a little contemptuous of people with this perspective. They don't seem to have thought through their belief that we should restrict the border, but do a shitty job of it in random ways.