It's a gigantic question you're asking and then on top of it you interleave Tailscale and zero-trust into the mix as well... Oooph that's so much ground to cover, I fear the answers you get won't slake your thirst.
If I were developing my own cloud-based stack I'd probably choose Kubernetes just because of the portability it affords me. I can make a cluster on any cloud provider I want that way - or span cloud providers for extra resilience etc. Plus if I wanted to "host it myself" I could go that route too. I've found the cluster access (RBAC i believe) provided by Kubernetes to be ... eh... we'll call it "cumbersome" but it's there. The cloud vendors I've seen don't make that bit as easy as they probably could (well, from what I've seen, but I'm no kubernetes expert). It does have "minikube" and "k3s" which would let you spin services up locally for use. So that seems reasonable to me.
At that point I would need to decide if I wanted to go ham on one provider and make one giga-cluster or have lots of little clusters all over the place and interconnect them... If so, at that point I'll be looking at some CNI like Flanel or Calico but since you mentioned zero-trust - I don't personally think those fit the bill as they are "too open" (imo). I might then layer on wireguard/Tailscale and that'd probably work pretty darn well. I've not used Tailscale's tooling in a very long time but I'm sure it's maturing and offers loads of cool features. I bet it'd work.
I actually am a dev on this OpenZiti thing you saw this other fella mention. It's also totally open source and we actually use the tun provided by wireguard if you're running on windows so you can set it all up and host it yourself for free (like any open-core company you can buy it too but I focus on the open source stuff). In my opinion it's got some perks over the Tailscale/CNI approach but it's an unknown quantity for you since you'll be blazing new (but HAWT) ground. The biggest benefit it has for my money is that it is geared towards the developer not devops per-se. Thinking about security at that time is "too late" in my opinion. That means OpenZiti doesn't stop at the network tier. Layer3/4 are just not "zero trust" enough. We want that trust to be extended all the way into the application itself. That means adopting one of our SDKs and baking it into your app. Once you do that you won't ever need a CNI again because apps will just address other applications over a totally secure, zero trust connection. But, if you want (zero trust *is* a journey) it's still perfectly capable of working on layer 3. There are other interesting benefits it can get you too but I digress... Oh and OpenZiti would play well with Kubernetes including people's minikubes. I enjoy giving my colleagues access to my minikube through OpenZiti, I don't know why it tickles me when I do but it does.
GitOps is definitely "a great idea". It too is the new hotness right now though truthfully I'm not sure if it's revolutionary more than evolutionary. Git has eaten the world and controlling your infra with a source control is just "a good idea". So I agree with the other person about looking into and maybe adopting a gitops approach as well.
I feel like this could go on and on and I know this response is ALSO shallow but the question you asked just seems gigantic... Maybe it's ratified some of your ideas.
