Provided a malicious host could take measures to serve the auditors a binary that matches the source, and something entirely different for everyone else, I know of no good way to confirm that a remotely served binary corresponds to the published source.
That's why the FSF takes a stand against what it calls "software as a service substitute". Technically, they are correct, for the reason stated above. An Internet that is pure protocol, with clients for remote services being local apps, is not difficult to imagine; in fact, that's how it started and that's how it still works.
What's different is the friction of running an embedded blob from a webpage is negligible compared to installing a package on your OS, therefore the "distribution" phase of the software lifecycle is effectively factored out, making the opportunity for user choice minimal. (Unless you run NoScript or LibreJS by default, which I support.)
Economically, SaaS is the path of least resistance. You deliver your product through the same platform that people use to browse cat pics, and it works invisibly enough that users never question whether it's doing the right thing. This creates ample opportunity for externalities to be dumped onto the end user, as we observe with the nascent privacy crisis.
Perhaps such an externality is of a sufficiently complex or speculative nature that users may not be able to identify it, or willing to care. But that doesn't mean it's ruled out; at least in theory, integrating with the existing legal system by way of distributing your code under a Libre license seems like a good way to nip possible shenanigans in the bud. (As far as your code is concerned, of course - as opposed to everyone else's non-free code out there.)
Of course, everyone is free to do what they think best, but it's good to have the FSF's radical evangelism as a point of reference - it's quite possible to do meaningful computing in this way, although maybe not as part of an organization that ignores the ethical underpinnings of its methods! (And that's what computers do best nowadays: algowashing.)
Sadly, I find the FSF's approach to the Web somewhat misplaced - for software that is neither locally deployed, nor accessed as an opaque service, but remotely delivered on-demand, focus should be on the delivery chain being auditable, and source maps being available, in the case of WASM and minified JS.
Instead, we get LibreJS, which is, as far as I can see, a license-based whitelist - regardless of the fact that non-minified JS is open-source by definition and never compiled to a binary form. The GPL doesn't make a whole lot of sense for embedded scripts, powerful as they are. It would be more relevant for WebAssembly, that's for sure.
