If speed is fine, why would I care if a compiler compiled to a human-friendly language?
From a performance perspective it's also a different target for web engines so has different opportunities for optimization. It's not clear to me we've reached the limit on that at the moment, or how much work browser developers are putting in to optimizing their WASM codepaths. The standard also includes support for things like SIMD which is not natively available in JS land which should provice a boost for suitable workloads and there is a lot coming through the proposal pipeline (https://github.com/WebAssembly/proposals) that will also let WASM diverge more from JS.
From a developer perspective allowing more languages than JS also has some productivity benefits. Both in using languages they are more familiar with and those better suited to certain domains. For example you can write math code in JS, in particular making use of TypedArrays to help things along but its much more straightforward in a language that supports more 'native' types.
Furthermore, JS semantics are limiting in other respects. E.g. there are people looking at tail calls and continuations in WebAssembly, which I can't see ever coming to JS.
Having a proper and well defined ISA is a lot more compelling.
Not only that, but it allows WASM-only compilers and runtimes with none of the JS baggage, and thus use of WASM outside of a browser context and with significantly lower resources requirements.
If speed is fine, why would a user care if the application was compiled to a language with "half-assed semantics" in a text format?