undefined | Better HN

0 pointstoast03y ago0 comments

Yeah, TCP splicing would have been neat. The use case wasn't high bandwidth, and it was transitional: client software has IP addresses to try when DNS fails, but we moved to new hosting related to an aquisition, HAProxy at the old host until all clients in the field have updated.

The bottleneck was connection establishment in the kernel, and as traffic decreased it became harder to run meaningful tests without great risk (send all the traffic to one server to see if it can handle it is fun, but unsafe), and we had to keep a fair number of servers to keep the IPs from the hosting provider, so we had way more capacity than needed and optimizing more wasn't a good use of time :(

0 comments

wtarreau3y ago

Indeed, that's fun at the beginning but past one point when you'd want to stop the servers taking 3 connections a day but you can't because these are the 3 most business critical connections, it starts to become much less fun.

Connection establishment in the kernel was an issue before kernel 3.8 I think though not sure. The problem is finding a spare port when you use a lot of concurrent connections and at some point it costs a lot. The "source" directive in haproxy was made to overcome this (you can define a per-server range so that the kernel doesn't have to scan its lists). But nowadays more efficient mechanisms were implemented in the kernel, that we support in haproxy (using IP_BIND_ADDRESS_NO_PORT) and we haven't witnessed these painful moments for a while now.

toast0OP3y ago

I was running on FreeBSD, and using port selection inside HAProxy to avoid the kernel selecting it, but still had contention on locks when inserting connections into the table. I can't remember if that was FreeBSD 11 or 12, but the next version made improvements which looked nice, but I wasn't able to test.

Somewhere my (not very great, but working) patch set is floating around, where I added rss aware port selection to HAProxy, so client and origin connections would both hash to the same NIC RX queue, and HAProxy never had to deal with cross-cpu communication for reading sockets.

wtarreau3y ago

This interests me. I've long been wondering how we could do that without knowing the NIC's hash and/or without having to brute-force the ports using the NIC's hash to reverse-map them. But maybe there are kernel facilities I'm not aware of to do that. If you happen to stumble upon it and share it to the mailing list, that would be great!

1 more reply

j / k navigate · click thread line to collapse

0 pointstoast03y ago0 comments

0 comments

wtarreau3y ago

toast0OP3y ago

wtarreau3y ago

1 more reply

j / k navigate · click thread line to collapse