How long did they live before becoming remains?
Ag gives more calories per square km, so more people. But also more vulnerability to disease, periodic famine, and in some cases malnutrition from over reliance on cereal crops (e.g. Pellagra). Hunter gatherers are/were constrained by wild food source densities but also tended to have smaller families, breast feed longer (so fewer pregnancies), and could sometimes migrate when local wild animal populations were exhausted. If you have a bad crop year you can't migrate to the next valley and find a new field of wheat waiting for you.
However there were likely many millennia of societies that mixed agriculture with hunting/gathering. Eastern woodland native Americans were like that pre-contact; maize agriculture + hunting game + fish etc. Seems like a good overall strategy.
Infant mortality going up is the same thing as life expectancy going down. Neither one tells you much about whether people die before or after you'd expect to start seeing problems in their teeth. After ignoring infant mortality, it's mostly afterward, but "remains" are not a representative selection.
> Hunter gatherers are/were constrained by wild food source densities but also tended to have smaller families
This conflicts with the idea that they had lower infant mortality.
> This conflicts with the idea that they had lower infant mortality.
Not necessarily. Smaller family size can be a result of other factors outside of infant mortality.
Aren't there parts of rural China still living in that mixed model today? And Africa too iirc...