Typesetting is a rendering a static copy a priori. You can expend a lot more time and resources rendering and reflowing the document. Rendering HTML+CSS, which can be dynamic, is a radically different problem.
I wouldn't be surprised if these posed very different constraints on the possibilities of the layout model
TeX's box model is in no way less optimizable than the HTML+CSS (if anything, it's the opposite due to it being much simpler). It has had a stable algorithm and implementation since 1983, and that implementation was never optimized for reflow like HTML + CSS has. TeX itself is also Turing-complete, which makes processing more expensive by definition, and that's mostly the reason for the slow performance, rather than the box model — and it is from 1983, after all. For some things, the reflow model is even used with TeX today (eg. table of contents generation usually requires at least two passes to "reflow" the output).
>6.6.4 Inspiration from other formatting models In the early design phases of the CSS visual formatting model, other formatting models were frequently consulted for inspiration. In particular, TeX [Knuth 1984] was often brought up in white-board discussions.
As one of its foundations, TeX has a well-defined box model wherein all objects, including individual glyphs, are contained in boxes. The spacing between the boxes can be controlled through TeX commands. In addition to optimal spacing between boxes, TeX also allows maximum and minimum spacing to be expressed. This is referred to as glue (although Knuth suggests that springs is a better term [Knuth&Plass 1981]).
The visual formatting model in CSS is based on a box model, and all elements, both block-level and inline, are turned into boxes. Thus, CSS goes further than most other style sheet languages in creating boxes. For example, DSSSL and P94 do not use boxes for inline elements. However, CSS did not adopt TeX's glue. Although the issue was discussed, it was decided against in order to keep the VFM simple. Glue is very useful when breaking paragraphs into lines, but CSS leaves this problem to implementations. CSS allows, but does not demand, inter-paragraph line-breaking optimizations. Each box in CSS is, however, potentially richer than the boxes found in TeX since it can contain a padding, border and margin bands. CSS also borrows other features from TeX, including the em and ex units.
https://www.wiumlie.no/2006/phd/#ch-css
Håkon also goes into great detail on the historical context and technical requirements surrounding the development of CSS.
I think the reason the CSS box model differs from the TeX box model is because of the different problems they attempt to solve and various contextual difference the technologies developed under. Crucially, it's not because the authors were unaware of the TeX box model or because they thought they could do something better
That also supports my claim that reflowing in TeX's box model would be simpler ("CSS is richer").
I simply highlighted a few things where TeX's box model does a great job for what are commonly hard problems with CSS even today (how do you align columns on decimal points?): I imagine most people, even LaTeX users today, are unaware of the internals, and I hope it's valuable for people to learn of them.
