I worked in AWS previously in the container space + with firecracker. I realized the container is an unnecessary layer that slowed things down + firecracker was a technology designed for AWS org structure + usecase.
So I ended up building a hybrid taking the best of containers with the best of firecracker.
Let me know your thoughts, thanks!
My problem with microVMs was that they usually won't run docker / kubernetes, I work on apps that consist of whole kubernetes clusters and want the sandbox to contain all that.
Does your solution support running k3s for example?
Really appreciate the feedback!
That's the one feature of similar systems that always gets left out. I understand why: it's not a priority for "cloud native" workloads. The world, however, has work loads that are not cloud native, because that comes at a high cost, and it always will. So if you'd like a real value-add differentiator for your micro-VM platform (beyond what I believe you already have,) there you go.
Otherwise this looks pretty compelling.
By what I assume is your definition, there are plenty of "non cloud native" workloads running on clouds that need live migration. Azure and GCP use LM behind the scenes to give the illusion of long uptime hosts. Guest VMs are moved around for host maintenance.
Current status is I'm looking for more feedback. In a few weeks when Ubuntu 26.04 comes out I'm going to set up my dev/stg clusters at work with it, at the moment I've only tested it in a test cluster at home.
It works this way: It creates a snapshot of the zvol, and replicates it to the secondary machine. When that's done, it does another snapshot and does a "catch up" replication (the first replication could take hours on large volumes). Pause the VM, do a final snapshot+replication. Replicate the working RAM. Start up the VM on the new host.
Thanks
I have been working on something similar but on top of firecracker, called it bhatti (https://github.com/sahil-shubham/bhatti).
I believe anyone with a spare linux box should be able to carve it into isolated programmable machines, without having to worry about provisioning them or their lifecycle.
The documentation’s still early but I have been using it for orchestrating parallel work (with deploy previews), offloading browser automation for my agents etc. An auction bought heztner server is serving me quite well :)
Not useful for things it hadn't been trained on before. But now I have the core functionality in place - it's been of great help.
To me, it feels like it could fulfill WASM's promise of a "portable execution unit that runs instantly anywhere," but with the added benefit of keeping the existing Linux binary ecosystem intact via microVMs.
I'm curious—is this kind of use case aligned with the direction you're aiming for?
Linux was built in the 90s. Hardware improved more than a 1000x. Linux virtual machine startup times stayed relatively the same.
Turns out we kept adding junk to the linux kernel + bootup operations.
So all I did was cut and remove unnecessary parts until it still worked.
This ended up also getting boot up times to under 1s. The kernel changes are the 10 commits I made, you can verify here: https://github.com/smol-machines/libkrunfw
There's probably more fat to cut to be honest.
It is on the roadmap, but frankly I haven't used Windows in a decade. I would love for a contributor to take that on as part of the free and open source spirit.
Sounds like it could be you? :)
WSL2 runs a linux virtual machine. Need to take some time and care to wire that up, but definitely feasible.
smolvm is a virtual machine with the ergonomics of containers like packaging and distributing, kernel is not shared. You'd run containers inside of smolvm.
Also, smolvm runs ontop of both kvm for linux and apple's hypervisor for macOS. So it's cross platform.
It also has file copy from host to inside the guest: https://github.com/smol-machines/smolvm/blob/70c97930dda5a92...
smolvm is a vm with some of the properties & ergonomics of containers - it's meant as a replacement for containers.
Probably a lot of other neat usecases for this, too
smolvm pack create --image python:3.12-alpine -o ./python312
./python312 run -- python3 --version
# Python 3.12.x — isolated, no pyenv/venv/conda neededElectron ships your web app bundled with a browser.
Smol machines ship your software packaged with a linux vm. No need for dependency management or compatibility issues because it is baked in.
I think this is how Codex or Claude Code should be shipped by default, to avoid any isolation issues tbh
I can't actually create and test a pack right now because of [1], but I love the idea of using this to distribute applications you might otherwise use a Docker image for.
Also libkrun is not secure by default. From their README.md:
> The libkrun security model is primarily defined by the consideration that both the guest and the VMM pertain to the same security context. For many operations, the VMM acts as a proxy for the guest within the host. Host resources that are accessible to the VMM can potentially be accessed by the guest through it.
> While defining the security implementation of your environment, you should think about the guest and the VMM as a single entity. To prevent the guest from accessing host's resources, you need to use the host's OS security features to run the VMM inside an isolated context. On Linux, the primary mechanism to be used for this purpose is namespaces. Single-user systems may have a more relaxed security policy and just ensure the VMM runs with a particular UID/GID.
> While most virtio devices allow the guest to access resources from the host, two of them require special consideration when used: virtio-fs and virtio-vsock+TSI.
> When exposing a directory in a filesystem from the host to the guest through virtio-fs devices configured with krun_set_root and/or krun_add_virtiofs, libkrun does not provide any protection against the guest attempting to access other directories in the same filesystem, or even other filesystems in the host.
Would you be ok with a trampoline that launched the VM as a sibling to the Vagrant VM?
smol machines is a virtual machine that has properties and ergonomics of containers. It's not an ai project, it's designed to run any software inside.
docker sandbox sounds like it's running ai stuff inside of a microvm.
So if you need to use a virtual machine - use smol machines.
If you need a to run coding agents, use smol machines still because agents are just software.
> If you need a to run coding agents, use smol machines still because agents are just software.
I get mixed signals from your argument
Cheers!
Edit: I see this appears to be a contributor to the project as well. It was not obvious to me.
@binsquare is this one: https://github.com/BinSquare
Though my version was only tested on Linux hosts
Got a lot of questions on how I spin up linux VM's so quickly
Explanation is pretty straight forward.
Linux was built in the 90s. Hardware improved more than a 1000x. Linux virtual machine startup times stayed relatively the same.
Turns out we kept adding junk to the linux kernel + bootup operations.
So all I did was cut and remove unnecessary parts until it still worked. This ended up also getting boot up times to under 1s.
Big part of it was systemd btw.
I'm trying to do away the model of cpu and memory tbh.
Virtio- balloon dynamically resizes based on memory consumed.
CPU is oversubscribed by default
Cheers!
I’m currently evaluating smolvm for my project, https://withcave.ai, where I’m using Incus for isolation. The initial integration results look very promising!
I'm building a different virtual machine.
*yes, FreeBSD is specifically developed against Firecracker which is specifically avoided w "Smol machines", but interesting nonetheless
[0] https://github.com/NetBSDfr/smolBSD
[1] https://www.usenix.org/publications/loginonline/freebsd-fire...
microvm space is still underserved.
So, if I use a "16 vCPUs, 32GB RAM, 400GB SSD" machine for a period of intense compute, and then want to scale that down to "2 vCPUs, 4GB RAM", most of my storage disappears?
That rather ruins the potential of the advertised scalability.
will look into it
Nice job! This looks really cool
So directionally yes. But it relies on kvm. I focus on portability i.e. cross platform for local macOS and linux, so that is not top of mind for me right now.
I build a virtual machine that is an alternative to firecracker and containers.
Can you pipe into one? It would be cute if I could wget in machine 1 and send that result to offline machine 2 for processing.
Yes! GPU passthrough is being actively worked on and will land in next major release: https://github.com/smol-machines/smolvm/pull/96
Yea just tried piping, it works:
``` smolvm machine exec --name m1 -- wget -qO- https://example.com/data.csv \ | smolvm machine exec --name m2 -i -- python3 process.py ```
But smol machines are not an implementation of unikernel - it's basically just the linux kernel but slimmed down. So, more compatible with most software.
[1] shameful self plug: https://gigatexal.blog/pages/i-heart-my-macbook/i-heart-my-m...
But should be easy for anyone to build their own integration with existing as well like nomad.
Looks really nice btw. Hot resize mem/cpu would be nice. This could become a nice tech for a one-backend-per-customer infra orchestrator then.
Thank you, great work!
question: why do you report that qemu is 15s<x<30s? for instance with katacontainers, you can run fast microvms, and even faster with unikernels. what was your setup?
thanks a lot