undefined | Better HN

0 pointsjrockway5y ago0 comments

I think the problem with xDS is that their example go-control-plane repository is completely useless. It's overly complicated with frightening-sounding details that don't matter to someone experimenting ("you MUST MUST MUST CACHE THIS how to do so is an exercise left to the reader").

I ended up reading the specs and found them very clear, and wrote my own xDS implementation: https://github.com/jrockway/ekglue/blob/master/pkg/xds/xds.g... I did this after reading the source code for the most popular xDS implementations and finding myself horrified (you know the popular xDS implementation I'm talking about). Now I have a framework for writing whatever xDS server I desire, and it can be as simple or as complex as I want it. For example, for my use cases, I'm perfectly happy with a static route table. It is very clear what it does, so I have that. What annoyed me was having to configure the backends from Kubernetes for every little service I wanted to expose to the outside world. So I wrote ekglue, which turns Kubernetes services and endpoints into Envoy clusters and Envoy cluster load assignments. This means that I never have to touch the tedious per-cluster configs, and still get features like zone aware load balancing. And I don't have to take on complexity I don't want -- the woefully under-specified Kubernetes Ingress standard, service meshes, etc. (I also plan to use ekglue for service-to-service traffic because xDS is built into gRPC now... just haven't needed it yet. It's great to use the same piece of software for two use cases, without having to maintain and read about features I don't need.)

TL;DR: take a look at the spec. It's really well thought out and easy to implement. Just don't cut-n-paste from Istio because they got it really wrong.

0 comments

2 comments · 1 top-level

chucky_z5y ago· 1 in thread

On that note, the grpc spec specifically calls for load balancing that doesn't actually do the proxying, but instead hands out assignments, with the server passing it's current load back to the load balancer service. it sounds like in this case the grpc client is using some array of xDS, but the server is using xDS along with...?

jrockwayOP5y ago

I feel like xDS is a relatively new addition to gRPC. I think there is another parallel implementation inside gRPC of external load balancing, which may convey server load information back to the gRPC client.

I looked up the current state of the xDS code, and there's a lot more of it than I remember. The EndpointDiscoveryService based gRPC balancer is here: https://github.com/grpc/grpc-go/blob/master/xds/internal/bal.... It appears to balance similarly to Envoy; locality-aware with priorities.

(That doesn't surprise me because I don't remember any field in the ClusterLoadAssignement proto that sends load information back to the client. Health, yes; load, no. But I could easily not remember it being there because it hasn't been something I've tried to implement.)

But yeah, the way to look at endpoint discovery is like DNS. DNS can return multiple hosts, and clients will spread the load by picking one at random (sometimes, if you're lucky). EDS is similar to this, but is a streaming RPC protocol instead of connectionless UDP, so it's theoretically easier to operate and monitor.

The other xDSes do more things -- CDS lets you discover services (that EDS then gives you endpoints for). RDS lets you make routing decisions (send 10% of traffic to the canary backend). SDS distributes TLS keys (and other secrets). ADS aggregates all of these xDSes into one RPC, so that you can atomically change the various config versions (whereas requesting each type of stream would only be "eventually consistent"; doing it atomically is good where route table changes add new clusters, the client is guaranteed to see the new cluster at the same time the route to that cluster becomes available).

It is all somewhat complicated but very well designed. This reminds me that I want to look more deeply into gRPC's support of xDS and add some integration tests between gRPC and ekglue.