Memory Reordering Caught in the Act (opens in new tab)

I don't know about "disasters". For better or worse, lots of software ships with severe bugs that are evidently rare enough never to be fixed but do happen.

For example, every few months, OSX kernel panics on my MacBook Air, with no obvious trigger. The backtrace always implicates the AHCI driver for the SATA SSD. Still, Apple evidently don't receive enough crash reports for this particular bug to actually fix it, as it's happened since I first bought the Air in November 2010.

I have no idea whether it's a threading bug like the one in the article (I'm not about to run my system in single-core mode for months just to find out) or maybe a race condition with DMA or just a simple logic error that only rarely applies. It might even be specific to the exact SSD model and revision I ended up with in my device. But that's pretty much irrelevant - there certainly hasn't been a widespread outcry about it.

Personally, as a developer, I would definitely want to fix that kind of bug in anything I'd built. But tracking it down might take weeks of costly developer time. So Apple's bug triage probably marked this one "WONTFIX" after deciding it only happened on hardware they no longer sell, so fixing it wasn't going to have a positive ROI.

(FWIW, if it sounds like I have a personal vendetta against the AHCI driver, that's because I do. ;-) It behaves erratically in ways unrelated to this crash, but that weirdness doesn't cause kernel panics, just extra work for developers of other drivers.)

Not really. Memory ordering and cache coherency issues are endemic to race conditions, but race conditions are bad bugs even in the absence of fun tricks by the caches.

You "fix" this the same way you fix any race: you get your synchronization right. The instructions used to implement the mutex (e.g. lock cmpxchg) are "serializing", which means that all memory operations issued before them in the instruction stream will be completed and committed before the instruction issues.

But not every programmer is going to implement their own lock-free algorithms (http://en.wikipedia.org/wiki/Non-blocking_algorithm). Experts write such algorithms, and other programmers will build their applications on top of those libraries. And even more programmers will use abstractions that aren't even libraries, but language-level abstractions.

Most programs don't use anywhere near enough cpu to make parallel algorithms pay off. So you may as well keep it simple, especially if you aren't hitting the limits of your current algorithms. Simplicity means less code, which means less bugs.

Correct. Memory operation re-ordering can occur due to the way the cache-coherence mechanism is implemented. If messages on the bus between cores are not ordered, memory operations can effectively be reordered even if each core never performs any reordering (e.g. due to cache misses).

This is a great explanation: http://irl.cs.ucla.edu/~yingdi/paperreading/whymb.2010.06.07...

The Xbox 360 and PS3 processors are very similar, one main difference is the PS3 (ignoring SPUs) is single-core with two hardware threads while the 360 is triple-core with two hardware threads per core. I would imagine that if you locked the affinity of the two threads to the two hardware threads of one of the 360's processors you would see the same behavior. IIRC, The two hardware threads on those CPUs are essentially just duplicate sets of registers that each have their own instruction stream, but execute the instructions in a single shared pipeline (the benefit being that you have more instructions available to fill the rather long CPU pipeline and are less likely to unused cycles while stalled on memory fetches). As far as the actual executing instructions are concerned there would be a single stream of instructions and thus no opportunity for the memory ordering effects that you might see with two distinct cores/processors.

You can most definitely witness memory reordering on the PS3 when you have code on the PPU and SPUs operating on the same memory.