undefined | Better HN

0 pointsxxpor3y ago0 comments

>Also, I found it best to work with everything avoiding cross CPU communication; making sure everything happens on the CPU core that handles the RX queue for the incoming connection.

Yes, this is absolutely critical. Atomic ops (including taking a lock) when you have a large number of cores just completely kills performance.

>Do they make nics with 128 queues now?

Yep, the Intel E810 100gbit controller supports 256 queue pairs for just one example.

Hope you can process that traffic pretty fast, the amount of memory dedicated to packet buffers with that many queues is making my head spin a bit.

0 comments

toast03y ago

Oh nice, in that case, time for 256 core haproxy box ;)

My use case was just tcp proxy, inspecting the first few bytes to determine destination (http vs https vs proprietary) and add proxy protocol headers to the origin. Setup is somewhat expensive (tcp table manipulation isn't cheap), but data on established connections is simple and easy.

xxporOP3y ago

IMO for super high packet rate cases, DPDK would make more sense. For proxy protocol injection, this would be fine since you don't really care about a lot of the TCP stack, just keeping track of new connections, and doing the sequence number rewriting to account for the injected bytes. All of the congestion control etc can be handled by the endpoints.

Of course if HAProxy performs well enough, then that's still a million times easier.

toast03y ago

Yeah, I had lots of fun ideas, but had a time crunch, HAProxy basically just worked to get started, and took well to optimizing, and pretty soon I had more capacity that would ever be needed, and never had a chance to rebuild it with exotic technology ;)

wtarreau3y ago

For pure TCP you rarely need threads, indeed! You may benefit from TCP splicing however if you're dealing with high bandwidth.

toast03y ago

Yeah, TCP splicing would have been neat. The use case wasn't high bandwidth, and it was transitional: client software has IP addresses to try when DNS fails, but we moved to new hosting related to an aquisition, HAProxy at the old host until all clients in the field have updated.

The bottleneck was connection establishment in the kernel, and as traffic decreased it became harder to run meaningful tests without great risk (send all the traffic to one server to see if it can handle it is fun, but unsafe), and we had to keep a fair number of servers to keep the IPs from the hosting provider, so we had way more capacity than needed and optimizing more wasn't a good use of time :(

1 more reply

j / k navigate · click thread line to collapse

0 pointsxxpor3y ago0 comments

>Also, I found it best to work with everything avoiding cross CPU communication; making sure everything happens on the CPU core that handles the RX queue for the incoming connection.

Yes, this is absolutely critical. Atomic ops (including taking a lock) when you have a large number of cores just completely kills performance.

>Do they make nics with 128 queues now?

Yep, the Intel E810 100gbit controller supports 256 queue pairs for just one example.

Hope you can process that traffic pretty fast, the amount of memory dedicated to packet buffers with that many queues is making my head spin a bit.

0 comments

toast03y ago

Oh nice, in that case, time for 256 core haproxy box ;)

xxporOP3y ago

Of course if HAProxy performs well enough, then that's still a million times easier.

toast03y ago

wtarreau3y ago

For pure TCP you rarely need threads, indeed! You may benefit from TCP splicing however if you're dealing with high bandwidth.

toast03y ago

1 more reply

j / k navigate · click thread line to collapse