It's not the variety of the diet but the quality of the food itself. Bread is good for energy, but if all you're eating is bread, you're not getting complete proteins, omega 3s, and other nutrients. It's fine, however, to eat nothing but meat and many societies did this for hundreds of thousands of years.
Farming is anything but sedentary, especially in the 19th century and prior. People's height was stunted because food was not abundant enough. Agricultural societies tended to grow faster than farming productivity could keep up with. A lot of people were simply malnourished and therefore never reached their natural height capacity.
But as you say, the hunter / gatherers such as the native Americans were taller on average than the first European settlers to arrive in America. This is not due to a particularly diverse diet. Most cultures subsisted on meat from hunting (largest source of nutrition) and a select few vegetables. The difference is that hunter / gatherer societies tended to self regulate their population according to available resources.
