We aren't talking about CSS, JS, Images but purely about the HTML page. If you have a 20K page and 15K of that belongs to header, footer, sidebar then in a server side rendered site that 15K has to be downloaded for every page on the site. On a SPA site it only has to be downloaded once as you can dynamically switch out the other parts.

If there was a mechanism in the HTML spec to tag parts of a page for partial caching then we wouldn't need SPA in most cases. But browsers only cache the entire HTML.