optimizing for cache hits and only loading the code for the view you need are not mutually exclusive. Your build can package optimized assets at a page or component granularity. But parsing a little bit too much javascript is not so slow as extra requests, doubly so considering that the biggest unoptimized assets are the dependencies that every page needs, like jquery, angular or whatever. I have worked on several gmail-sized apps and, to date, the profiler told me not to bother.
let's be clear: i'm not advocating lazy-loading assets like jquery or your main app codebase. I'm talking about your ancillary files that may not get used on every visit, especially if your site is a single-page app.