Anyway, on the topic of scalable UDP services, does anyone have any experience of load balancing a UDP service? Because UDP is connectionless there's no obvious way to make UDP packets "sticky". Are there any established practices that could help scale this k8s Wireguard service to 2 or more containers?
That said, NGINX can do UDP load balancing and WireGuard is stateless, so it should be possible to use this with a Service + NGINX ingress controller at scale: https://kubernetes.github.io/ingress-nginx/user-guide/exposi...
I have not tried it though.
A client must hard code it's IP address currently, which means if it can connect to more than one node, then it is unclear which path a response from a server should take to get back to that client. Each VPN instance could run NAT, but then users would never be able to talk to each other.
Wireguard makes this significantly harder than say ipsec. WG has nothing to indicate when a client connects. And there is no dead peer detection, so you cannot tell one a client disconnects. IE. Scripting something to update a global routing table to say which sever has which client is near impossible.
I use wireguard daily for personal stuff. However I cannot think how I would make it work in an active-active situation besides NAT, which I don't want.
I agree with you, WireGuard makes this significantly harder than it needs to be. Other protocols do better in this respect.
I didn't know Ubuntu 20.04 back ported WG into its 5.4 kernel. I spent a few hours yesterday fixing a node after breaking ZFS because I upgraded to 5.6 for WG support. I feel rather silly now..
edit: rektide mentioned 'kilo' which actually does exactly what I said (https://github.com/squat/kilo).
Example vpn container:
docker run --name foo --cap-add=NET_ADMIN ...
docker run --net=container:foo ...
[1] i.e. https://github.com/bubuntux/nordvpn/blob/master/start_vpn.sh...
Naturally it breaks if replica count changes.
The other option is conntrack but then you have another stateful component that doesn't scale
Cloudflare shared, in some detail, how they load-balance wireguard traffic for roaming-ip and ports: https://news.ycombinator.com/item?id=21070315
Usually, I've seen UDP client-affinity set on (source-ip, destination-ip) tuple to handle port changes, but it doesn't help client with roaming-ips.
apt-get install -y --no-install-recommends wireguard-tools
So no need to use a builder image
As the author notes: "you can run a road-warrior-style Wireguard server in K8s without making changes to the node."
Which makes this guide ideal for me. I run a lightweight K8s flavor (K3s, https://k3s.io/) as "configuration management" on my home server and home automation Raspberry Pi's because I don't want to mess with OS/userland configuration or the associated tools (Puppet, Ansible, hacked together scripts, etc) or want to maintain any OS state manually.
For my setup I just flash K3s to disk or SD card and let it join the cluster. Everything else is configured in Kubernetes and stored nicely as configuration files on my laptop so I have an overview of everthing and can modify/rebuild whenever I want.
So since I'm doing isolation in containers/Docker already it's a small step to a lightweight Kubernetes. What Kubernetes gives me on top of that is that I can consider everything below the application layer as a declarative API.
This is why I'm still using https://zerotier.com -- also no affiliation.
The hosted Tailscale product is meant for GSuite customers who want an peer-to-peer VPN with corporate SSO. Yes, you have to trust them - SSO login is inherently centralized. My company uses it, it works great.
As a test, I did set up a vxlan tunnel through a wireguard tunnel (linux to linux) to prove that it is possible to get that working. However, I can't do that on something like a mobile android client.
I've been trying out Glorytun, it does multi-path VPN with a relatively similar wire format to WireGuard. Being mostly indoors, due to the microbial boogaloo, I've not been trying it with the most interesting applications.
I think it'll need work for connections with varied performance.