All quite true, server renders pages are the quickest route to first render, in-line your CSS for that first page and it's even quicker. All best practices for sites that fit those categories, content and e-commerce for example.
But not all sites and apps can do that, and almost certainly not for all functionality and pages. Bloat isn't just about tooling, it's organisational too. Lots of teams all working within one product.
Modern ES modules can actually be worse if not bundled on on the server or by a build system. The browser doesn't know what to request until it starts parsing the previous response, that dependency tree can be large. That literally is accumulating latency. However with progressive improvement it's not much of an issue, but again not everyone can do that.
On top of that anyone still on HTTP 1.2 will only be doing ~5 requests to the same domain in parallel.
Point is, latency does accumulate, but it doesn't have to with well designed code structure, "modern" (rediscovered 90s) techniques, and modern browser features like preload and preconnect.
