Heartbleed Update (opens in new tab)

In some sense, yes, the special free list caused it. In another sense, the missed bounds check caused it. In a third sense, the lack of proofs of safety, or informal code review process, "caused" it---that one's harder for me to argue.

My own preferred sense is that mixing network code with soft real time performance requirements with crypto in a single library, single process, all in C---maybe that caused it, and will cause problems for any channel-oriented crypto network system. Imagine trying to mix GnuPG with high performance networking! Boom.

My preferred tools for thinking about what causes accidents like this are Leveson's systems-oriented frameworks, explained in Engineering a Safer World. The text is available free from MIT Press, I believe. If you're responsible for the safety of a planetary computer system, you should read it and it's principal competitors.

And if you do, Akamai Infosec is hiring.

Well OpenSSL didn't implement their own malloc exactly, they used some weird indirection to try and speed it up on some platforms, which may have bypassed protections that some versions of malloc have enabled (guard pages).

We did, internally. We got key extraction in an hour, about the same as Cloudflare---so we save $10k and rotate exposed keys. Since we renew most certs on an annual basis, all our easy-to-rotate certs were flipped months ago. The outliers are weird custom stuff, or two-year EV certs.

You ever try rotating 400 EV certs in a weekend? Neither have most Certificate Authorities. They say security is people, processes, technology. Our tech worked well---but not so well I can't wish it was better. Our people did awesome, stalwart work. But the PKI industry processes are due for some serious reconsideration.

I can't tell you how badly I want TACK or DANE or CT live and working right now.

What would that solve? We already know that it's possible to compromise the private key using this bug. The problem is that no one knows whether attackers have actually done so, since very few organizations will log all the data going in and out of their servers.

Akamai posted its special heap implementation</a> to openssl-dev yesterday: <http://marc.info/?l=openssl-users&m=139723972124003&w=2>. We'll follow that up with help to adapt it to openssl's needs---which are much, much broader than our set of problems. We run 100k machines, sure---but all x86-64, almost all Linux, and all running substantially the same software. We don't use DTLS. We don't use any PAKE systems. We are, in some sense, an easy problem.

The OpenSSL Foundation is trying to help people with those needs and needs varying on every imaginable dimension communicate with secrecy and strong authentication. We should expect them to need several times as many developers full time on that problem as any of the planetary-scale computing companies.

Totally agree.

I'm finalizing a tool to scan and visualize the top 1M alexa site URLs to see which are vulnerable - and ~3% (30 000) still are. In the last few days I've observed about ~5% of those getting patched daily (~1500).