Skip to content

Top New Best Ask Show Jobs

One Million Concurrent TCP connections | Better HN

One Million Concurrent TCP connections (opens in new tab)

(blog.whatsapp.com)

192 pointspvodsevhcm14y ago65 comments

65 comments

mrb14y ago

I remember reading around 2002-2004 about a sysadmin managing a very large "supernode" p2p server who was able to fine-tune its Linux kernel, and to recompile an optimized version of the p2p app (to allocate data structures as small as possible for each client) to support up to one million concurrent TCP connections. It wasn't a test system, it was a production server routinely reaching this many connection at its daily peak.

If it was possible in 2002-2004, I am not impressed that it is still possible in 2011.

One of the optimizations was to reduce the per-connection TCP buffers (net.ipv4.tcp_{mem,rmem,wmem}) to only allocate one physical memory page (4kB) per client, so that one million concurrent TCP connections would only need 4GB RAM. His machine had barely more than 4GB RAM (6 or 8? can't remember), which was a lot of RAM at the time.

I cannot find a link to my story though...

gtani14y ago

(Not your link but) some pointers on tuning erlang/OTP servers:

http://www.erlang-consulting.com/thesis/tcp_optimisation/tcp...

http://www.trapexit.org/Building_a_Non-blocking_TCP_server_u...

http://groups.google.com/group/erlang-programming/browse_thr...

(this erl mailing list thread is pretty typical, if you put up code, describe your app, hardware, network, database/external dependencies, etc, you'll get a ton of good advice about killing off bottlenecks. Another example

http://groups.google.com/group/erlang-programming/browse_frm...

mmaunder14y ago

Agree with this being underwhelming. Maybe they'll hire someone who understands that Erlang is using kqueue to make this possible.

Running netstat|grep like this on a high concurrency server takes a long time to run. I've never found a faster way to get real-time stats on our busy servers and would be interested if anyone else has.

thomasknowles14y ago

Actually I would run ss over the netstat for such high concurrency connection as netstat hits /proc/net/tcp and it's quite slow. However, you need to have the tcp_diag module loaded otherwise it falls back to /proc/net/tcp.

derobert14y ago

You might be able to parse /proc/net/tcp yourself faster, especially for just counting by state.

adient14y ago

You may have tried but generally add -n to netstat to speed it up by just displaying IPs instead of trying to resolve them. I've never seen a system with so many concurrent connections hands on so there could be some other bottleneck.

tworats14y ago

The WhatsApp guys are very sharp ex-Yahoo guys who've had tremendous experience with scaling systems. Rick Reed is fairly legendary. Yahoo was a long time FreeBSD shop, so it's not surprising they went with that.

I hope they publish how they did it - in fact let me drop them an email and see if I can convince them to do so.

cperciva14y ago

Yahoo was a long time FreeBSD shop

FWIW, Yahoo still uses FreeBSD extensively.

lenn0x14y ago

It's been done before. Here is an article using Erlang and Linux.

http://www.metabrew.com/article/a-million-user-comet-applica...

Part 3 is my favorite. http://www.metabrew.com/article/a-million-user-comet-applica...

jen_h14y ago

This article series is kind of a Bible to me. It's not going to solve all of your problems, for sure, depending on yer stack, and setting yourself up for a forkbomb isn't the wisest in all situations, but it's got a lot of good advice & is pretty good about providing you a "Okay, tweak these parameters and then try to break it" baseline.

indygreg214y ago

Technical details would be interesting. Until then, here's Urban Airship's post from last year on 500k connections on Linux:

http://urbanairship.com/blog/2010/09/29/linux-kernel-tuning-...

jsr14y ago

OK, you hooked me with the title. But "FreeBSD + Erlang" was kind of a dissatisfying reason for how you achieved it. Would love to hear more details! How far we've come since http://www.kegel.com/c10k.html

getsat14y ago

They could have done it using C, Ruby, Python, or any other language. kqueue is what makes FreeBSD (and OSX) awesome at concurrency.

http://en.wikipedia.org/wiki/Kqueue

silentbicycle14y ago

How does kqueue compare to epoll on Linux? I've written C code using kqueue on OpenBSD and OS X, but have only used epoll via libev (and not at especially high load). I thought the big change came from trading level- for edge-triggered nonblocking IO, but maybe the kqueue implementation is superior for sockets somehow?

The main advantage Erlang has over C/Python/Ruby/etc. is that asynchronous IO is the default throughout all its libraries, and it has a novel technique for handling errors. Its asynchronous design is ultimately about fault tolerance, not raw speed. Also, it can automatically and intelligently handle a lot of asynchronous control flow that node.js makes you manage by hand (which is so 70s!).

You can make event-driven asynchronous systems pretty smoothly in languages with first class coroutines/continuations (like Lua and Scheme), but most libraries aren't written with that use case in mind. Erlang's pervasive immutability also makes actual parallelism easier.

With that many connections, another big issue is space usage -- keeping buffers, object overhead, etc. low per connection. Some languages fare far, far better than others here.

asomiv14y ago

The kqueue interface is awesome, but unfortunately kqueue is bugged on OS X: http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod#OS_X_AN... (as are poll() and select()!)

Rickasaurus14y ago

This may be a dumb question (I'm not a networks guy) but how do you maintain so many connections with just 65535 ports? Can you have more than one connection per port?

caf14y ago

The server generally only ever uses one port, no matter how many clients are connected. It is the tuple of (client IP, client port, server IP, server port) that must be unique for each TCP connection - so the limit of 65535 ports is only relevant for how many connections a single client can make to a single server.

nathanappere14y ago

I believe this is incorrect. The server usually listen on one port, but everytime it does an accept, a different random port is used, and the client start talking to the server on that new port.

richardw14y ago

You can have multiple connections on the same port from the same client. E.g. all connections via HTTP commonly go over the same port and a single browser usually has 2-4 simultaneous connections to the server. They don't use multiple ports to achieve this.

pacala14y ago

There is one unique connection per (address, port) pair.

huhtenberg14y ago

Actually, the quintet of [protocol, src_ip, src_port, dst_ip, dst_port] is what's unique (protocol being TCP or UDP)

Rickasaurus14y ago

Ahh I see, you just give one box a ton of IPs. Thanks :)

chubs14y ago

They beat me to it! I've only gotten to 500k on EC2, however i believe there's some trickery in their firewalls / NAT which is holding me back... If anyone's interested in the gory details, see: http://splinter.com.au/tag/comet

forsaken14y ago

Curious about what you've seen from the EC2 networking gear that's holding you back? Firewall not letting more connections through?

chubs14y ago

Well, at the moment i can't pin it down to EC2, but it's the only thing i can imagine it'd be. The network/cpu/memory usage is all healthy, and there's nothing in the kernel log, so that's what i'm guessing is the cause. Although i may be wrong.

huhtenberg14y ago

In absolute numbers - wow, that's impressive.

It was several years ago, but I've done my share of high-concurrency stuff under Linux and the highest I got to was about 200K connections - at which point the single-threaded server bottlenecked at its disk I/O.

The main issue is not the actual connection count, it's what the per-socket OS overhead is (so not exhaust non-swappable kernel memory), how many sockets are concurrently active (have an inbound or outbound data queued) and if the application can handle all the events that epoll/kqueue report. This is not a rocket science by any means, and the kernel is relatively easy to fine-tune even when the actual load is present.

cnlwsu14y ago

I would be curious about the hardware used, it can make it much more or less impressive. I have done a test with 1 million concurrent tcp connections using java (mina) on an Ubuntu system... but it had 64 gb of ram. It kept running for weeks under the load which I felt pretty good about.

getsat14y ago

What is the kernel structure overhead (in bytes) per connection on FreeBSD?

imsy14y ago

Good work. On Imsy (www.imsy.com) we hope to achieve the same with Node.js on EC2. Presently the numbers are smaller (in 100K range).. but looks like it will scale smoothly till 1 million.

henry50114y ago

It's hard to say that something is going to scale smoothly up an order of magnitude until you're there. 900K requests is a lot of room for things to go wrong.

nivertech14y ago

Node has memory limit of about 140K connections. I was able to accept and actually do something usable with 1M on EC2 and 3M on physical server (Erlang and Ubuntu/CentOS)

kitsune_14y ago

How much ram does this single server have, what about its cores?

jen_h14y ago

A. That is AWESOME. Props, guys! B. Just how long did it take for that netstat to return? ;)

spokengent14y ago

FWIW, netstat isn't fun to use. conntrack is better, or cat /proc/net/ip_conntrack

Luyt14y ago

They're on FreeBSD. conntrack is a Linux utility.

KonradKlause14y ago

conntrack is not netstat. I requires netfilter's connection tracking feature.

Using conntrack on a 1MC system will waste even more kernel memory!

gabi3814y ago

How would kqueue compare to Windows's IO completion ports in terms of performance?

flzz14y ago

Just because you can doesn't mean you should.

nivertech14y ago

Not many can, but some fortunate enough to actually need this stuff.

j / k navigate · click thread line to collapse