Linux Kernel Tuning for C500k (opens in new tab)

My, how things have changed since the C10k problem...

http://www.kegel.com/c10k.html

Just a technical note on the 64K myth section. My understanding is that TCP connections track by the tuple (remote_host, remote_port, local_host, local_port) so a single client can have 64k unique connections to each port on a remote machine.

If that is actually the case, the document gets its myth correction wrong (by a lot) :)

Can anyone clarify this?

Interesting post, thanks for sharing!

About the suggested sysctl.conf settings: I think you'd also need to adjust net.core.rmem_max and net.core.wmem_max in order for the net.ipv4.tcp_rmem and net.ipv4.tcp_wmem settings to be effective.

Furthermore it couldn't hurt to increase net.core.netdev_max_backlog, which is the maximum number of packets queued on the input side, when the interface receives packets faster than kernel can process them.

Some other tricks that were not touched upon in the article, but which may apply depending on the nature of your traffic:

1) If you have lots of short connections and you want to tune the amount of time that the kernel will keep half-closed connections around then you can play around with changing the values of net.ipv4.tcp_fin_timeout, net.ipv4.tcp_tw_reuse, net.ipv4.tcp_tw_recycle, and net.ipv4.tcp_max_tw_buckets.

2) If you have a modern NIC then you probably need to tweak the txqueuelen in your ifconfig options.

3) If you are get hits from a large number of random browsers then sometimes setting net.ipv4.tcp_no_metrics_save and net.ipv4.tcp_moderate_rcvbuf to turn off cacheing of flow metrics helps.

4) Increase net.core.somaxconn to increase your listen queue size.

5) If you have a local firewall like iptables in place make sure you increase net.ipv4.ip_conntrack_max, direct your high-traffic ports to the NOTRACK table, and play around with all of the various net.ipv4.netfilter.ip_conntrack_tcp_timeout_* settings.

I'm wondering what are the major side effects of this. Hmm?

btw, the most common source of high load is (surprise!) disk I/O.

So, moving a /var/log (not just /var) on separate device connected to distinct controller port is a big deal.

If you're running, say, mail server, you should separate /var/spool and /var/log and /var/db/mysql if any.

Partitioning, serious network card (think Broadcom) and big CPU caches are good things to begin with.

I tested one server up to 1 million concurrent connections a couple of years ago, search for sysctl in this article to see the settings I used: http://www.metabrew.com/article/a-million-user-comet-applica...

This isn't relevant to newer kernels as these settings are dynamic based on memory size since 2.6.26ish - The kernel will set this based on usage no need to tweak. The only real issue is making sure you buy a high end network card that will offload as much as possible to avoid x context switches per second (I don't know what it is exactly with netpoll).

Linking this with the IPv6 stuff currently on the front page: note that none of this would be necessary if the clients were running IPv6 (or otherwise un-NAT-ed) - the server could simply send them a UDP packet or even open a TCP connection.

This is interesting stuff. I jumped into node.js programming a while ago and will like to run similar tests on node.js. Can anyone tell me how client side load of 500K long lived connections achieved ? Is there a standard set of programs to achieve this or some custom scripts.

Did somebody solved C1000K (or C1M) problem?

http://news.ycombinator.com/item?id=1755575

Is maximum number of connections that you can reach on largest EC2 instance is the same as on physical server?

LOL! This is called tuning nowadays?

Even Oracle providing much more good advices, let alone some individual pros.

Good starting point: http://www.puschitz.com/InstallingOracle10g.shtml

Update: Oh, yes, I understood. Newfags doesn't know what Oracle is. MySQL = RDBMS, I see. ^_^