In the US, immigrants are much poorer and come in at the bottom of the labor and housing markets.
But yes, anything besides raising taxes is irresponsible.