If you build sites that can support arbitrary viewports, extrapolating to arbitrary scaling factors on a given concrete viewport does not create an explosion of scenarios to manually test -- all you need to know is that your scaling works, and if you always build your functionality responsively, the scaling and responsive layout together just work. (I'm talking about building the whole layout with rems, not just using them for font sizes. For the latter, I'd agree with you that it's brittle and falls apart quickly.)

It's fair to point out that the browser feature is not widely used, but when it is used, it's likely by users who benefit most from the small amount of effort it takes to support it. (It's really a few lines of CSS, plus simply using rems as your websites unit whenever you'd otherwise use px. It's even easier in terms of avoiding weird 16px-based rem calculations if you make your rems 10px for a default 16px browser preference by applying a straightforward conversion on the root element.)