It could be left at the discretion of the holder, with fair use being an escape valve to prevent abuse.
e.g., Disney won't stich together all their movies in a year-long pseudo unit of work, because, if it did so, a two-hour clip from that work would fall under fair use.