1) Video provider pays for bytes sent to the user that are not used. If the cache policy is "cache the whole thing", you would find yourself paying (a guestimate) 2x for bandwidth. This is because most people do not watch even nearly the entirety of every video that they start loading.
2) Using server bandwidth for bytes that are not used. This increases contention for bandwidth at the origin. Perhaps less of an issue if your site is small and you're using a giant CDN. This does become an issue if you are youtube or you are serving the bytes yourself.
3) Using client bandwidth for bytes that are not used. This slows down any other download that might be happening on the client computer, even when the video is paused. This could be, a different video in a different tab that the user is trying to watch, other assets on the page, other pages, or even other programs on the computer.
Of course adding an option for the user to cache the whole thing wouldn't be so bad, as opposed to having this behavior by default for everyone.
