Because the coefficients of the larger powers are more sensitive, and most attempts to fit those models don't account for that, penalising high weights for those values more. You could have a regularisation regime that accounts for that. It's also why extrapolation fails on them so often.
x^2 grows fast, x^3 even faster. They outpace the other parts of the model. Any changes to those coefficients also means you'll need to change the coefficients of the smaller powers to adjust to the change.
A small amount of error in the largest power means two things - one the error will be exaggerated, because the power has an oversized effect, two the values of the smaller powers will also be wrongly adjusted to account for that error, and will contribute that poor fit to the model as well.
