Ask HN: What's a good Linux OS and setup to build a dev “network” on my laptop?

I've been doing my work in VMs running on my local for a long time now and it works great. I got started because I wanted my development environment to be a close match for my stage and production.

For a long time I ran VirtualBox guests on a laptop host running Debian or Ubuntu. VB is great especially if you're getting started with virtual machines, it has a GUI and things are easy to understand. I used a host-only network to keep traffic inside my laptop.

If you're going to edit code inside a VM you'll need an IDE that can handle it. Emacs with tramp has served me well for years. IIRC with VSCode you have to install a server on the VM.

I do web development so I've never felt the need to run desktops inside VMs. It's nice to have some separation between work and stuff like HN but Gnome workspaces gives me just enough.

A couple years ago I levelled up and switched to KVM on a headless Debian box. This system uses bridge networking with DHCP from my router, and I use Tailscale to access my VMs from outside my LAN. You can use virt-manager (which also works for VirtualBox) but I mostly use the libvirt CLI tools and cockpit/cockpit-machines if I need something a little more visual. Caveat: I don't know how KVM would handle suspend on a laptop.

It sounds like you are describing QubesOS [1] I've used it with 16 GB of ram just fine provided each VM does not need a lot of ram. Here [2] are some screenshots and show how networks and firewalls are presented and a video walk-through [3]

[1] - https://www.qubes-os.org/intro/

[2] - https://www.qubes-os.org/screenshots/

[3] - https://www.youtube.com/live/hkWWz3xGqS8?feature=share&t=849

For an off-the-shelve solution there is Qubes but from what I've heard it's probably quite a pain to use unless you're comfortable debugging Linux desktop and driver problems in particular. Generally KVM/libvirt is nice because it's fast and can be set up transparently and networking can be customized in any possible way. (It might also be possible to get GPU Passthrough working with KVM but I think this only works if you have 2 GPUs)

It really depends on why you want to have separate environments. What's the thing you're trying to avoid here?

Anything that requires graphics is much more efficiently done in the Bare Metal machine. You can have a separate user for those or some other kind of abstraction that's not a virtual machine.

You can also use Network Namespaces directly, using "ip netns" or even cgroups. That way you can run every single program natively, but have multiple network environments for them.

You can go for the jails-style containers where you also run everything natively but on a different chroot, without having a Kernel running on top of a Kernel. For this I recommend creating a chroot with debootstrap or febootstrap and then launching it with systemd-nspawn.

You can also have specific applications running on their own "jail" with /usr/bin/firejail.

Now if you want even stricter separation you'll have to go with full virtualization (guest Kernel running on top of Host Kernel). This is much worse in terms of efficiency of used resources, but gives you the most separation from one thing to another. For that you can use virt-manager or just Virtualbox.

I think Debian for all of these usecases. Or Ubuntu.

If I understand the question correctly, you’d like to build a networked system entirely on a single machine.

If this is the case, then I’d recommend trying docker-compose. It’s quite an easy little system to get up and running, just need a single file to declare different containers and their relationships. Then just run “docker-compose up/down” to spin up and spin down the network.

The beauty of this is that it can run on extremely barebones Linux distributions, as it’s all container based.

It’s a great starting point anyway, and I’m sure there’s many alternatives now that use a similar api

Have you considered devcontainers?

Its use results in carrying entire development environments with you, and because it is using containerization these will not clutter your host OS.

Using DevPod (https://devpod.sh/) you are not locked into Visual Studio or Visual Studio Code, but you can use whatever tool you want.

IMO this kind of setup will provide a much better DX than running a bunch of VMs eating away the resources of your laptop.

I use Linux Mint for a similar host set up. Its worth it to move away from virtualBox to the open source versions as I did recently, KVM etc.

There are quite a few paper cuts in setting it up the first time. I recall it took about five hours before I finally had my first VM in the right place, with right size of disk space and ram and the version of linux I wanted. But after the learning curve it becomes very easy. Honest :)

I've been using Fedora since 30 and it has been a very good OS. Install the dnf group for virtualization and you got a KVM host that let's you play with VMs and LXC. There is also a dnf group for containers.

I have something similar and have had for several years. Just needs a recent-ish kernel and decent distro!

I try to avoid virtual machines unless there is an absolute need to virtualise the hardware as well - in most cases single kernel, multiple containers is preferred.

Specifically, I use Debian with systemd-nspawn unprivileged (user namespaced) containers (managed by machinectl), defined and built using mkosi as BTRFS subvolumes (a BTRFS mounted on /var/lib/machines/ allows nspawn to create copy-on-write containers from a 'template' container).

The nice thing about nspawn is being able to create simple declarative configurations with private namespaces. With a private network namespace it has the concept of 'zones' which are containers that share the same (private) bridged network. systemd-networkd has built-in configs for both the host and containers to do automatic configuration.

I use this to create per-domain (e.g. example.com) zones with multiple function-specific containers (one for router/firewall, another for DHCP and DNS, others for single-sign-on, database, web, mail, and whatever else is required). I use veth to connect the zone selectively to the host interface public bridge interface providing external access if necessary.

This allows 'development' 'test' and 'production' domains using mkosi to define and build each repeatedly as needed. In the non-public domains the zone can be configured to use the same public IP addresses as production will use without it leaking outside the zone.

I use IPv6 only in the zones with both ULAs and global prefixes and DNS64/NAT64 in the 'router' container if the zone needs to make outbound connections (to IPv4 destinations).

I use netfilter rules applied to each zone's bridge interface to impose firewall restrictions.

Using veth pairs one can connect two containers (or zones) directly rather than using the common zone bridge interface.

I do dev work on the host itself; if you want to partition application usage I'd suggest keeping it simple by using multiple user accounts, one per 'task' so $USER configs and so on are all kept separate.

In the 'development' containers I do things like bind-mounting the base source-code directory, or project-specific directory, into the container. I do all source-code work on the host but build and test in the appropriate container.

For example I have 'linux-builder' container where I do out-of-tree kernel builds for multiple architectures. The container has the required crossbuild tools installed. Recently this has been used to build latest mainline kernel for amd64, arm64, and powerpc64 (for the Xbox 360!).

Windows Subsystem for Linux.

Best 'Linux distro' for Windows.