That’s actually sort of a myth. If you work on a farm, there are a few very intense parts of the year, but most days are otherwise prep work and maintenance and you can actually wrap up fairly early and manage your own schedule. What sucks about farming is thin margins and how a bad crop can wipe you out.
Possibly different on a factory farm that maximises production every day but that’s not really what you’re referring to.
This comes from personal experience working on productive small farms.