That's a serious problem, if only there was some way to raise their wages.

>farmers have to pay 5x as much to attract labor

Yep, there it is. Some of those poor people will transition to these new higher wages in farming. This will reduce labor supply in other sectors which will push wages up across the board. Rinse and repeat for other industries full of illegal immigrants: construction, maid services, etc.