Microsandbox: Virtual Machines that feel and perform like containers (opens in new tab)

The issue, at least with multitenant workloads, isn't "container vulnerabilities" as such; it's that standard containers are premised on sharing a kernel, which makes every kernel LPE a potential container escape --- there's a long history of those bugs, and they're only rarely flagged as "container escapes"; it's just sort of understood that a kernel LPE is going to break containers.

You cannot build a secure container runtime (against malicious containers) because underlying it is the Linux kernel.

The only way to make Linux containers a meaningful sandbox is to drastically restrict the syscall API surface available to the sandboxee, which quickly reduces its value. It's no longer a "generic platform that you can throw any workload onto" but instead a bespoke thing that needs to be tuned and reconfigured for every usecase.

This is why you need virtualization. Until we have a properly hardened and memory safe OS, it's the only way. And if we do build such an OS it's unclear to me whether it will be faster than running MicroVMs on a Linux host.

Importantly I'd like to see the configurations of the machines. There's a lot you can do to docker or systemd spawns that greatly vary the security levels. This would really help show what needs to be done and what configurations lead to what risks.

Basically I'd love to see a giant ablation

In a way, containers already run as honeypots with cash or crypto prizes, it's called production code and plenty of people are looking for holes day and night. While this setup sounds like a nice idea conceptually, the monetary incentives it could offer would surely be miniscule compared to real targets.

You can optimize a lot to start a Linux kernel in under a second, but if you're using a standard kernel, there are all manners of timeouts and poll attempts that make the kernel waste time booting. There's also a non-trivial amount of time the VM spends in the UEFI/CSM system preparing the virtual hardware and initializing the system environment for your bootloader. I'm pretty sure WSL2 uses a special kernel to avoid the unnecessary overhead.

You also need to start OS services, configure filesystems, prepare caches, configure networking, and so on. If you're not booting UKIs or similar tools, you'll also be loading a bootloader, then loading an initramfs into memory, then loading the main OS and starting the services you actually need, with eachsstep requiring certain daemons and hardware probes to work correctly.

There are tools to fix this problem. Amazon's Firecracker can start a Linux VM in a time similar to that of a container (milliseconds) by basically storing the initialized state of the VM and loading that into memory instead of actually performing a real boot. https://firecracker-microvm.github.io/

On Windows, I think it depends on the hypervisor you use. Hyper V has a pretty slow UEFI environment, its hard disk access always seems rather slow to me, and most Linux distro don't seem to package dedicated minimal kernels for it.

I think you need to provide more details on what VM software you’re using. On VirtualBox what you describe is very noticeable, and it didn’t have that delay in older versions. So it could be just an issue with that VM software and not a general “traditional VMs” issue.

The answer is that it doesn’t have to be like that.

In practice virtual machines are trying to emulate a lot of stuff that isn’t really needed but they’re doing it for compatibility.

If one builds a hypervisor which is optimized for startup speed and doesn’t need to support generalized legacy software then you can:

> Unlike traditional VMs that might take several seconds to start, Firecracker VMs can boot up in as little as 125ms.

Creating the VM itself is fast. It depends on what you run in it. Unikernel VMs can start in a few milliseconds. For example, checkout OSv.

In Linux, VM memory allocations can be slow if it tries to allocate GBs of RAM using 4K pages. There are ways to help it allocate 1GB at a time which vastly speeds it up.

Windows probably has an equivalent.

Try Windows Server Core on an SSD. I've seen VMs launch in low single-digit seconds. You can strip it down even further by removing non-64-bit support, Defender, etc...

Sounds like a VirtualBox problem.

I'm using Hyper-V and I can connect through XRDP to a GUI Ubuntu 22 in 10 seconds and I can SSH into a Ubuntu 22 server in 3 seconds after start.

I mean it is basically booting a computer from scratch, kind of makes sense. You have to allocate memory, start virtual CPUs, initialize devices, run BIOS/UEFI checks, perform hardware enumeration, all that jazz while emulating all of it, which tends to be slower than "real" implementations. I guess there is a bunch of processes for security as well, like wiping like zeroing pages and similar things that takes additional time.

If I let a VM use most of my hardware, it takes a few seconds from start to login prompt, which is the same time it takes for my Arch desktop to boot from pressing the button to seeing the login prompt.

I'm trying this out now and it's very promising. One problem I'm running into with the Python library is that I'd like to keep that sandbox running for several minutes while I do things like set variables in one call and then use them for stuff several calls later. I keep seeing this error intermittently:

    Error: Sandbox is not started. Call start() first

Is there a suggested way of keeping a sandbox around for longer?

The documented code pattern is this:

    async def main():
        async with PythonSandbox.create(name="my-sandbox") as sb:
            exec = await sb.run("print('Hello, World!')")
            print(await exec.output())

Due to the way my code works I want to instantiate the sandbox once for a specific class and then have multiple calls to it by class methods, which isn't a clean fit for that "async with" pattern.

Any recommendations?

Looks great! This might be extremely useful for a distributed/decentralized software testing network I'm building (called Valet Network)...

Question: How does networking work? Can I restrict/limit microvms so that they can only access public IP addresses? (or in other words... making sure the microvms can't access any local network IP addresses)

Only did a quick skim of the readme, but a few questions which I would like some elaboration.

How is it so fast? Is it making any trade offs vs a traditional VM? Is there potential the VM isolation is compromised?

Can I run a GUI inside of it?

Do you think of this as a new Vagrant?

How do I get data in/out?

i'm on a mid-level laptop, at times with slow or expensive internet, running ubuntu. i want to be able to run nominally-isolated "copies" of my laptop at near-native speed

1. each one should have it's own network config, eg so i can use wireguard or a vpn

2. gui pass-through to the host, eg wayland, for trusted tools, eg firefox, zoom or citrix

3. needs to be lightweight. eg gnome-boxes is dead simple to setup and run and it works, but the resource usage was noticeably higher than native

4. optional - more security is better (ie, i might run semi-untrusted software in one of them, eg from a github repo or npm), but i'm not expecting miracles and accept that escape is possible

5. optional - sharing disk with the host via COW would be nice, so i'd only need to install the env-specific packages, not the full OS

i'm currently working on a podman solution, and i believe that it will work (but rebuilding seems to hammer the network - i'm hoping i can tweak the layers to reduce this). does microsandbox offer any advantages for this use case ?

Looks neat. If I understand correctly, I can use it to spin up backends on the fly? You have an ambitious list of languages to support: https://github.com/microsandbox/microsandbox/tree/main/sdk

edit: A fleshed out contributors guide to add support for a new language would help. https://github.com/microsandbox/microsandbox/blob/main/CONTR...

Hi appcypher, very cool project! Does the underlying MicroVM feature provide an OCI runtime interface, so that it could be used as a replacement for runc/crun in Docker/Podman?

How does the microvm architecture compare with firecracker?

Can you explain how this compares to Kata Containers? [0] That also supports OCI to run microVMs. You can also choose different hypervisors such as firecracker to run it on.

[0] https://katacontainers.io/

Cool project. Off topic question: Are the images in the "Use Cases" section in the README from a real app? I like the clean UI design.

Can you use Microsandbox for everything you can use Docker for, or are there cases where containers make more sense?

Congratulations on the launch!

This is very neat tech, but I think you might want to wait until you actually have Windows covered before making claims like https://github.com/microsandbox/microsandbox/blob/main/MSB_V...

Think I can build a notebook on top of this ? Jupyter client has been a pain to manage

Are you ready for the deluge of networking questions for all the buck wild configurations?

What's the story for macOS support?

I like the idea. But when you say "bullet proof" security, there are exploits to break out of VMs that exist. Have you looked into those?

Your statement initially went over my head. Sorry lol. You can always download the installer script and audit yourself. I will set up proper distribution later.

You are right. We leverage libkrun. Libkrun uses virtio-mmio transport for block, vsock and virtio-fs to keep overhead minimal so we basically depend on any perf improvement made upstream.

Firecracker is no different btw and E2B uses that for agentic AI workloads. Anyway, I don't have any major plan except fix some issues with the filesystem rn.

On recent Linux, try:

    findmnt --real

It's part of linux-utils, so it is generally available wherever have a shell. The legacy tools you have in mind aren't ever going to be changed as you would wish, for reasons.

Sorry I am lacking the context to understand this post. What does running mount inside a container do that's so egregious? Are host mounts exposed to the container somehow? I thought everything needed to be explicitly passed through to the container (e.g. using a volume)?

The other SDKs are generated hello-worlds at the moment. I will get to them one by one, but I welcome and appreciate any contributions to them.

> can you share some thoughts on how you compare or future direction?

Microsandbox does not offer a cloud solution. It is self-hosted, designed to do what E2B does, to make it easier working with microVM-based sandboxes on your local machine whether that is Linux, macOS or Windows (planned) and to seamlessly transition to prod.

> Do you also use Firecracker under the hood?

It uses libkrun.

I can't tell if it uses firecracker but thats my main question too. I'm curious as to whether microsandbox will be maintained and proper auditing will be done.

I welcome alternatives. It's been tough wrestling with Firecracker and OCI images. Kata container is also tough.

I recommend trying Javy[0]. Javy allows you to build a WASM file that includes Javy's JS interpreter along with your JS source code. Note that Javy is a heavily sandboxed environment so it doesn't have access to the internet, or npm modules, a desirable feature for running user code.

We're building an IoT Cloud Platform, Fostrom[1] where we're using Javy to power our Actions infrastructure. But instead of compiling each Action's JS code to a Javy WASM module, I figured out a simpler way by creating a single WASM module with our wrapper code (which contains some further isolation and helpful functions), and we provide the user code as an input while executing the single pre-compiled WASM module.

[0] https://github.com/bytecodealliance/javy

[1] https://fostrom.io

You might be able to get away with running QuickJS compiled to WebAssembly: https://til.simonwillison.net/npm/self-hosted-quickjs

> Would that be a good use case for this?

That is an ideal use case

> Are there better alternatives?

Created microsandbox because I didn't find any

runsc / gVisor is interesting also as the runsc engine can be run from within Docker/Docker Desktop.

gVisor has performance problems, though. Their data shows 1/3rd the throughput vs. docker runtime for concurrent network calls--if that's an issue for your use-case.

It has a YAML config format to declare all of that so you could just do that once, or template it, generate it on the fly, fetch it from remote, or many other methods.

Windows already has built-in sandbox and its good.

https://learn.microsoft.com/en-us/windows/security/applicati...

Windows support is a work in progress. I haven't tested using VNC yet but it should be possible.

Thank you!

> Can we build our own python sandbox using the sandboxfile spec?

Yes and I plan to make that work with the SDK.

PS: Multi-stage build is WIP.

They can. I need to improve the doc. Working on that right now

Microsandbox is for people that would like to maintain their own infra. I'm not going to stop trying to make it better to self-host.

They are Linux VMs and you can host any executable that can work on that. The python/node environment you see is part of what makes the SDK work. Really, it's very similar to Docker in use.

As a Nix user, I'm actually really excited to try this out.

I want to run sandboxes based on Docker images that have Nix pre-installed. (Once the VM boots, apply the project-specific Flake, and then run Docker Compose for databases and other supporting services.) In theory, an easy-to-use, fully isolated dev environment that matches how I normally develop, except inside of a VM.

Wow. Just seeing this. I've not done proper benchmarking yet but rn we are lagging behind in file I/O for the OverlayFS impl

If we get enough of these sandboxes, maybe we will finally get one that's easy for me to run on my own machines.

Because those have different directions than microsandbox and you've already mentioned one. I want easy secure sandboxes for AI builders. IMHO, microsandbox is easier to get started with.

That said I don't think either KataContainer or Cloud Hypervisor has first-class support for macOS.