That's a lot of wasm and beam. I tried to do it the other way -- compile beam straight to wasm and it seems to work in straightforward cases, producing wasm binaries in kilobytes sizes. With wasm fx it's even possible to do processes natively. You can find the link in bio.
Cool thing would be to somehow wire it up to liveview to make it possible to run some parts on the frontend side transparently.
I'm not doing any active development on it right now, but it looked pretty doable when I tried last time.
