In the Roman Empire if you found a child that would otherwise die you could adopt it into your household as a slave since that was certainly better. Your logic is absolutely sound if you dispute the concept of human rights entirely.
The problem is you create a permanent underclass in the country since you take literally almost all rights from them. Including by the way the right to return where they came from.
