I think it mainly failed because Sun basically abandoned it. Security was not the issue I think (Flash was the same, maybe worse, still widely used) but the UX was quite terrible and never really improved: You had the jvm with its long startup times which was more than noticeable in the 90s and early 2000s. Applets always felt very alien inside a Website and using the applet simply to call into optimized code from JS was, while possible not really what it was used for.

Security maybe more of an issue today but WASM probably is not worse than JS in that regard (basically it's just another turing complete language). As others have said APIs are more problematic, which I suspect will ever get worse while browsers further and further try to transform from document viewers to application runtimes (think of such "brilliant" ideas like WebUSB). WASM is not the key issue here, it's running untrusted code on the client which, even if it's sometimes not outright harmful takes control away from the user in one way or antother. In the end the main difference is that it's now a feature of the Browser rather than a plugin which has a hard time to fit in nicely.

If it were just the bytecode then it would not have failed. JVM bytecode and WASM have plenty of similarities. What cost Java is the stdlib surface area and security model. Also that it is a separate runtime was annoying, like Adobe with Flash, there was one primary vendor.

So many JVM bugs and a huge stdlib. WASM has no real lib, and when it does, it's security model is already restrictive... there's no disk, network, threads (yet), etc.

Along with the other responses, the JVM imposes its own object model on any language implemented on top of it. WebAssembly does not, so it is vastly simpler and more flexible.

For example, you had to rewrite your apps to run as Java applets. You do not have to rewrite things to run on WebAssembly- all the existing C, C++, etc. code is compatible, up to the platform API.

A big reason Java failed was that it was painfully slow. Starting the VM and importing everything took tens of seconds on old machines, and still takes a couple of seconds on new ones.

Then people started worrying about security and all of the Java applets broke because nobody wants to go through that code signing process.

I have to admit I don't know much about the security model of WebAssembly, but I wonder if it will become as annoying as Java in the future.

That's very much the question indeed. All that's really needed for a native experience on both Android and iOS is a native ARM binary, optionally bundled with webkit, but then that's what native runtime environments for Android and iOS have already built-in.