The standard set by SCOTUS was, apart from what you've already said, that a time extension was permissible as long as the "traditional contours of copyright" were not altered. One of them was explicitly called out, fair use.

The Court also explicitly mentioned that for a regulation to survive constitutional scrutiny in regards to copyright, it must not supersede, nullify, or even "disturb" the exercise of fair use.

This set the precedent that fair use is constitutionally required, not only a generous grant by Congress.