> I’ve heard that humans had a long period of time after we became sedentary and started relying on agriculture that the average height decreased significantly, and it was only in recent centuries that it has gotten back to normal due to more varied nutrition. So even if you can technically survive on a very limited diet, it can still have negative effects.
It's not the variety of the diet but the quality of the food itself. Bread is good for energy, but if all you're eating is bread, you're not getting complete proteins, omega 3s, and other nutrients. It's fine, however, to eat nothing but meat and many societies did this for hundreds of thousands of years.
Farming is anything but sedentary, especially in the 19th century and prior. People's height was stunted because food was not abundant enough. Agricultural societies tended to grow faster than farming productivity could keep up with. A lot of people were simply malnourished and therefore never reached their natural height capacity.
But as you say, the hunter / gatherers such as the native Americans were taller on average than the first European settlers to arrive in America. This is not due to a particularly diverse diet. Most cultures subsisted on meat from hunting (largest source of nutrition) and a select few vegetables. The difference is that hunter / gatherer societies tended to self regulate their population according to available resources.