The worst is when population goes past carrying capacity and then crashes. That's when you get mass die-offs.

Population can settle down at carrying capacity simply through reduced birth rates matching up with death rates. That's the situation we see in much of the western world, with better availability of birth control etc.