You're absolutely right. I didn't mean to imply that it was. But purpose built hardware has design specifications that can be validated. Good hardware is designed with failure modes in mind. Which means you have some hope of a decent risk management strategy.
And, exactly as you said, using and relying on a hardware RNG isn't a panacea--it still comes with difficult problems--which only emphasizes how hopeless, futile, and misleading entropy estimation is.
I'm not arguing that trying to indirectly capture entropy is a bad idea, even if you also have a proper hardware RNG. My point is just that entropy estimation is a poor idea, particularly of hardware that hasn't been specifically analyzed. By pretending like we can reliably quantify the entropy, we're giving users a false sense of security. That's not justifiable.
Collecting entropy indirectly is best effort. You can't know, except in pathological cases, whether it's working or not. You can only know when it's failing spectacularly, which would be rare when collecting from many different sources.
All those magic, per-device constants could likely be replaced with a single heuristic--collect everything for N seconds, then move on. If it worked, it worked; if not, well then you're not any worse off than you were when the entropy estimators gave a false sense of confidence. You can still add hacks to try to collect more entropy, and to do it faster, to improve effective security; just don't pretend like the kernel is accurately quantifying it.