undefined | Better HN

0 pointsvbezhenar2y ago0 comments

Yesterday I was trying to track weird bug. I moved a website to Kubernetes and its performance was absolutely terrible. It was loading for 3 seconds on old infra and now it spends consistently 12 seconds loading.

Google Chrome shows that 6 requests require 2-3 seconds to complete simultaneously. 3 of those requests are tiny static files served by nginx, 3 of those requests are very simple DB queries. Each request completes in few milliseconds using curl, but few seconds in Google Chrome.

Long story short: I wasn't able to track down true source of this obviously wrong behaviour. But I switched ingress-nginx to disable HTTP 2 and with HTTP 1.1 it worked as expected, instantly serving all requests.

I don't know if it's Google Chrome bug or if it's nginx bug. But I learned my lesson: HTTP 1.1 is good enough and higher versions are not stable yet. HTTP 3 is not even supported in ingress-nginx.

0 comments

dilyevsky2y ago

> I moved a website to Kubernetes and its performance was absolutely terrible. It was loading for 3 seconds on old infra and now it spends consistently 12 seconds loading.

My guess is it has more to do with resources you probably allocated to your app (especially cpu limit) than any networking overhead which should be negligible in such a trivial setup if done correctly

vbezhenarOP2y ago

No, I don't use cpu limits and memory limits are plentiful. Besides, the only difference between working and non-working setups were literally single switch `use-http2` in the ingress controller and it was repeatable experience.

dilyevsky2y ago

That is indeed interesting. I’ve used nginx with grpc (which uses http2) years ago and definitely did not see any such latency issues even at 50k qps

mgaunard2y ago

Kubernetes makes everything slow and complicated.

Why do you even need to have proxies or load balancers in between? Another invention of the web cloud people.

vbezhenarOP2y ago

That's not my experience. My experience is that kubernetes does not introduce any noticeable performance degradation. My message was about nginx, not kubernetes.

Front TCP load balancer allows to use a single IP address for multiple backend servers. It allows for server maintenance without service interruptions. Or even server crash.

Ingress controller is nginx which does reverse proxying. It handles HTTPS termination and it handles routing to the application which is supposed to serve the request. It's either another nginx instance configured to serve static resources or some kind of HTTP server which implements an actual server logic.

I agree that having two nginxes for static resources seems weird, but that's not a big deal, because nginx does not consume much resources.

mgaunard2y ago

In real life you cannot achieve that level of failure transparency without specifically building your application for it.

If request N goes to server A, then request N+1 from the same session goes to server B, most likely that request will fail because the session does not exist within that server's context.

xyzzy_plugh2y ago

nginx is notoriously bad at not-HTTP 1.1. I wouldn't even bother trying.

Envoy is significantly better in this department.

doublepg232y ago

Huh TIL. I had always considered nginx the “fast” http server.

dilyevsky2y ago

Nginx is fast enough for most applications. I wouldn’t bother switching if you dont need the power of “software defined” proxy

1 more reply

apitman2y ago

What were your reasons for moving the site to kubernetes?

vbezhenarOP2y ago

Because kubernetes is much easier to operate than dozens of scripts and docker composes thrown around servers. We have like 5 sites, each site operated by like 5-10 services thrown around 4 servers without any redundancy, autoscaling and even consistency. It's truly is a mess.

j / k navigate · click thread line to collapse

0 pointsvbezhenar2y ago0 comments

I don't know if it's Google Chrome bug or if it's nginx bug. But I learned my lesson: HTTP 1.1 is good enough and higher versions are not stable yet. HTTP 3 is not even supported in ingress-nginx.

0 comments

dilyevsky2y ago

> I moved a website to Kubernetes and its performance was absolutely terrible. It was loading for 3 seconds on old infra and now it spends consistently 12 seconds loading.

vbezhenarOP2y ago

dilyevsky2y ago

That is indeed interesting. I’ve used nginx with grpc (which uses http2) years ago and definitely did not see any such latency issues even at 50k qps

mgaunard2y ago

Kubernetes makes everything slow and complicated.

Why do you even need to have proxies or load balancers in between? Another invention of the web cloud people.

vbezhenarOP2y ago

That's not my experience. My experience is that kubernetes does not introduce any noticeable performance degradation. My message was about nginx, not kubernetes.

Front TCP load balancer allows to use a single IP address for multiple backend servers. It allows for server maintenance without service interruptions. Or even server crash.

I agree that having two nginxes for static resources seems weird, but that's not a big deal, because nginx does not consume much resources.

mgaunard2y ago

In real life you cannot achieve that level of failure transparency without specifically building your application for it.

If request N goes to server A, then request N+1 from the same session goes to server B, most likely that request will fail because the session does not exist within that server's context.

xyzzy_plugh2y ago

nginx is notoriously bad at not-HTTP 1.1. I wouldn't even bother trying.

Envoy is significantly better in this department.

doublepg232y ago

Huh TIL. I had always considered nginx the “fast” http server.

dilyevsky2y ago

Nginx is fast enough for most applications. I wouldn’t bother switching if you dont need the power of “software defined” proxy

1 more reply

apitman2y ago

What were your reasons for moving the site to kubernetes?

vbezhenarOP2y ago

j / k navigate · click thread line to collapse