I also build things that are heavy on the network side, and virtualization drops networking performance a great deal (this may be changed these days, not sure).
Don't get me wrong. I don't fear virtualization. I use virtualbox and vmware heavily for development, and yes they do have a purpose, but it's a bad fit for the type of projects I build.
I just find that anything in life that just continues to add features for the sake of adding features, and create more and more "magic" push buttons to solve your problems eventually goes to crap and becomes a fragmented dependency hell that I would rather not deal with. And yes, I do love Golang.