I'm one of the authors of this research. Feel free to ask any questions, I'll be happy to tell you more.
So you don't need a third party networking solution, but Kubernetes is open so a number have been nicely integrated. flannel is the most popular choice, but there are many others (I hear about Calico a lot)
Our original intention in doing this research is to answer the question "how much do we lose in terms of latency and throughput?" So, as you've said, flannel's aws-vpc is equivalent to out of the box k8s/AWS solution, flannel's host-gw is equivalent to calico. Since they use same "backend", in benchmarks we focused on flannel because it's easier to switch in flannel from one configuration to another (basically just put new config via etcdctl and restart the daemons).
However, thanks in pointing it out. In the article we should state our motivation clearer and add references to another solutions, such as calico.
Netstat -rn and you can see where the traffic is going or coming from. The added benefit is you can then bgp peer with other clusters and get routing across them
For small configurations flannel/host-gw is OK, I guess, but if one have several kubernetes clusters, flannel/host-gw becomes harder to maintain. That's a place where calico should be useful.
Other than that, great benchmarking! IPvlan is pretty neat, it's the natural evolution of macvlan and hopefully catches on quickly.
In the post you mentioned one of the drawbacks to adopting ipvlan was the lack of tooling to manage it. To address this issue, and a few others[2], Kubernetes has recently adopted the Container Network Interface(CNI)[3] standard as the solution for managing network plugins in Kubernetes. CNI ships with a few plugins including ipvlan and, maybe more importantly, the ability to allocate and manage IP addresses[4]. Over time CNI should give Kubernetes the flexibility to work well with multiple container runtimes including Docker and rkt, while allowing third party solutions to fill in the gaps around networking.
[1] https://www.kernel.org/doc/Documentation/networking/ipvlan.t...
[2] http://blog.kubernetes.io/2016/01/why-Kubernetes-doesnt-use-...
[3] https://github.com/appc/cni/blob/master/SPEC.md
[4] https://github.com/appc/cni/blob/master/SPEC.md#ip-allocatio...
The results of these benchmarks do not surprise me at all. To me, they all fall in to the category of 'less (overhead) is more (performance)'. With VXLAN encap being the obvious example of greatest overhead.
I think its also worth mentioning that k8s networking is being enhanced in v1.2 to support isolation and multi-tenancy through ThirdParty resources (back end network solutions). The alternatives included in the benchmarks aren't going to be able to support these kinds of network policy as is.
And, unfortunately, things get a more complicated when you want to provide more than simple reachability (which is all that k8s asks for today). The tradeoff is to be able to control the packets with the lowest overhead possible. VXLANs will give you isolation, but at the cost of encapsulation. Stacking bridges and tunnels and distributing VNIDs/routes not only introduces more latency, but becomes another multi-host coordination problem (matching tunnel IDs, etc).
We're working on a new way to build cloud native networks that avoids the encap, but still lets you control all the packets.
You can learn more at http://romana.io if you're interested.
