Absolutely. This is the prime hallmark of shitty, slow web. If you fight your battles here everything else will fall into place.
I've started building our B2B web apps where the server returns 1 final, static HTML document in response to every resource. There are no separate js/css resource URLs at all in our products anymore. Anything we need is interpolated into the final HTML document script or style tag by the server.
If you back up for a few minutes and really think about what kind of power you get with server-side rendering relative to # of requests... It's a gigantic deal. Composing complex client views shouldn't happen on the client. It should happen on the server where all of the answers already live. Why make the client go read a bunch of information piles to re-assemble Humpty Dumpty every goddamn time they want to view the homepage?
In terms of latency, the game theory for me is simple: Either do it in exactly one request, or it doesn't matter how many requests it takes. That said... caching effects are really important for many applications, and separating sources out can save you a lot of $$$ at scale. So, you will need to balance the UX and economic effects of some of these choices.