In the context of a regular end-user desktop machine, this seems overly paranoid to me. The odds of encountering a real, JS-based spectre attack in the wild are basically zero (has anyone
ever seen a browser-based Spectre attack outside of a research context? even once?), and the odds of it then being able to retrieve actual sensitive data are also basically zero. That's two astonishingly tiny numbers multiplied together. The threat just isn't there.
For regular end-user desktop machines, the mitigations only decrease performance for no real benefit. Spectre is a highly targeted attack, it's not something you can just point at any random machine to retrieve all their bank passwords or whatever.