"Natural rights" are mostly a thing insofar as a mob of people that sees person A taking away what they consider a "natural right" of person B, will attack person A.

Thus, if the populist sentiment of the people of some land-mass is that you don't have a right to be there with them, then you don't, no matter what abstract arguments you might give to the contrary. Your rights only extend as far as others' willingness to enforce them for you.