"modern passport not existing in past" ≠ "free human movement"
borders have largely had guards, who had no obligation to let people through, or even to treat them fairly and not rob them. frontiers had bandits who existed solely to prey on travelers.
even if you could move to a different settlement, you did not have the same legal rights as citizens of that city.
the "modern passport" has done more for free human movement than anything that came before.