Our Failure Migrating to Kubernetes (opens in new tab)

(engineering.saltside.se)

55 pointsahawkins9y ago8 comments

8 comments

8 comments · 2 top-level

millisecond9y ago· 3 in thread

First I'd get a tcpdump on one of the kubernetes boxes and see what's really going on at the network level.

One hunch is that ELBs don't like ephemeral ports and Kubernetes lives on them. Might be a mismatch here. ALBs are much more flexible with ports if that's an option.

puzzle9y ago

Kubernetes doesn't support ALBs natively yet, but there's a third party controller for them. The tcpdump approach is painful, but usually works.

kkirsche9y ago

Just for my sake what's ALB and ELB mean. I assume elastic load balanced and … load balancer?

cosmie9y ago

Application load balancer

http://docs.aws.amazon.com/elasticloadbalancing/latest/appli... Gives a good overview of the two types

sandGorgon9y ago· 3 in thread

What tool are you using to setup k8s? I really recommend kops - because it takes care of some cloud specific config clashes.

For example, it users the carrier grade subnet 100.64 for AWS because other up subnet conflict.

I have moved on from k8s to docker swarm because of the ingress complexity in k8s. I'm hoping some of the momentum behind Istio will solve it in the future.

buahahaha9y ago

it's clear the ingress complexity is what's burning them here. It's just a simple exposed port mixup causing that error rate.

ahawkinsOP9y ago

We considered that as well. We have a smoke test for the chart that hits each service with a generic ping RPC. I thought that was good enough upon reevaluating it. The generic ping RPC comes from an inherited service which means _any_ or our Thrift services could respond correctly with a pong. Which means the correct service may run on the same port. I verified this is not the case. I'll update the smoke test to use a different test rpc as regression for this possibility.

ahawkinsOP9y ago

We are using Kops. It's worked well for us so far.

j / k navigate · click thread line to collapse