1. Go grab https://conveyor.hydraulic.dev/ (disclosure: my company makes that)
2. Run `conveyor generate {javafx,compose} my-test-project` (pick your preferred UI toolkit). You now have a JVM desktop project.
3. `./gradlew jar; conveyor make site`. You now have a directory with self-updating platform native packages for Windows, macOS (ARM/Intel) and Linux that you can upload to a web server of your choice. Or supply upload creds and it'll do that step for you. Provide certificates and it'll also take care of signing.
That's all it takes to get a self-updating JVM desktop app these days. The same tool can do Electron, Flutter and native apps too (i.e. C++/Rust), so you could also use it with Tauri, but the nice thing about the JVM support is that it'll both strip down and bundle the JVM for you whilst cross-building/packaging so you don't need CI to do releases. From the user's perspective it's a normal app, from the developer's perspective you just hack on your local dev laptop like you could for a web app.
Binary sizes depend on how much functionality you use and how much effort you put into minifying. With the default level of effort (i.e. none) a simple hello world JavaFX desktop app will be about 33mb and a simple compose app will be about 52mb for macOS (skia is a very large graphics library). However, the Mac versions are larger than those for other platforms, and these sizes are still quite wasteful. There's a lot of low hanging fruit. You could make it a fair bit smaller by using ProGuard and other more aggressive dead code elimination techniques, as well as more heavily compressing bytecode. There's also plenty of fat to trim on the native code side.
GraalVM native images are interesting because they have much faster startup time and low memory usage - competitive with C++ apps, even. They take more work to create though. You can see a robotics app that's natively compiled here:
Aren't you just replacing one VM for another?