For my part, I once made a bad assumption about how the Google SAX-style parser handled callbacks for zero-length XHTML start-stop tags. I presumed that <title/> would get a callback with the end-of-open-tag and start-of-close-tag pointers being equal, at the character after the close of the tag. Instead, the parser called the callback with the start-of-close-tag pointer after the start-of-close-tag pointer. (I had misinterpreted the API as passing pointers to the start and end of enclosed content.) I had test cases for un-closed <title> tags and <title></title>... but when my code hit production, the few pages (fewer than 1 in a million) that expressed an empty title as <title/> caused my code to try and construct a string with negative length and crashed that portion of the indexing system. I was right to feel very embarrassed for my oversight.

I remember the savage meme so clearly because it was quite out of the norm, and I felt bad for the guys since they were so quick to fix things even when not publicly shamed. (Only the author and reviewer got notified when I left a comment on their code review.)

There is a point where someone has to put their foot down and demand things be done properly, otherwise the inevitable consequence is a giant mess leading to disaster.

You might be used to small startup teams with responsible, experienced developers.

Out there in larger industry you get people doing absolutely crazy things that break huge, expensive systems.

There’s a difference between “oops I didn’t realise this library doesn’t scale the way I assumed it did” and “rewriting language symbols because I’m too stupid to use more than one syntax forever and ever.”

The standard you walk past is the standard you accept.

Are you saying you would walk past C code with DO…OD instead of {…}?

Would you accept that standard just to be “nice” all the time?

researcher types often get to work on problems that swe types find interesting, so some swes get grumbly. researcher types also tend to write pretty horrific code which adds salt to the wounds.

but there also can be a sort of envy that emanates from the research side. many are fully aware of their shortcomings and are envious of the swes ability to get things done on computers cleanly.

it often seems that there can be yearning to wear each other's hats from the two groups. if i were running a company i think i'd try to break down that wall as it would probably make a lot of people happier.

of course, the right answer here isn't a meme... it's performance regression tests in the ci suite. and maybe a little training on why customizing a programming language with macros is bad.

The original bournegol macros for DO and OD were a bit different:

  #define DO ){
  #define OD ;}

We cared because indexing latency is important. Engineers get paged and woken up in the middle of the night when indexing latency gets high. Doubling CPU usage meant doubling indexing latency. The pre-Percolator indexing system ran roughly 1,000 threads per box on roughly 1,000 Warp18 boxes. Of course, most of those threads were blocked on I/O. (Warp18 was something like 16 cores per box... I think those were the RevE Opterons that had the memory fence bug that caused the glibc semaphore implementation to not actually synchronize operations.) I personally would have written things using more async and/or non-blocking I/O, but the main reasoning in Google at the time was that massively multithreaded blocking I/O is a simpler programming model and closer to what was being taught in school those days.

Also, the indexing system's quota is huge. Someone made a mistake in their benchmark program that they thought got maximum utilization out of the Warp18 hardware. The indexing system actually got higher utilization than the hardware stress-test, resulting in cooling being under-provisioned in the Atlanta datacener (more than a decade ago, I'm sure indexing has moved now). There was a couple day stretch one hot Summer where we needed to stay on the phone with staff inside the Atlanta DC to keep track of the temperature inside the datacenter. When the temperatures started to get above something like 80 F, we needed to dial back on the number of workers in our Borg jobs.

While testing the prototype for the Percolator indexing system, we were luckily sitting right next to an engineer who planned datacenters. We luckily overheard him saying he couldn't figure out why the power usage seemed to be swinging so wildly. It was literally the power usage of a decent sized town suddenly ramping up and down. We asked him the exact times power usage was ramping up and down, and figured out it was when Frank and Daniel were starting and stopping the Percolator prototype.

My best guess is that the indexing system right before Percolator was a bit more energy efficient than Percolator, but maybe not. In any case, doubling CPU usage was a big deal. Real money, and in Atlanta, probably real carbon footprint. (The Dales was hydro, but I don't think much of Atlanta's power was renewable at that time.)

Hmm... I don't remember that, and I conducted over 100 interviews back when I was at Google. A blanket failure for using a regex sounds like it's missing some nuance. But, maybe that was before or after my time.

I wouldn't be surprised if they start out with some interview questions that are easy to implement as regexes, but then gradually make the question harder and more general, where a pushdown automaton is necessary for a general correct solution, and fail people who either don't get a correct answer or end up implementing a pushdown automaton by dynamically generating larger and larger regexes.