Some random blog post about it: http://webchick.net/node/99
I worked on a Drupal project... Once.
My approach has been to step exponentially. I go back 1 commit, 2 commits, 4 commits... etc. I'll be using bisect from now on though.
git bisect is "temporal bisection", i.e. you delete half of a time interval. The manual bisect is kinda-sorta "spatial" in the sense that you delete half a region of code. the two things work by similar principles, and their problem domains overlap -- but they are not the same.
When we decided to look into it when we had some downtime, we found out that it was a date bug that only reared its head on the 31st day of the month, hence only happening every two months.
I'm not sure bisect would've helped but still was really funny when we did find the issue.
Previous HN thread: https://news.ycombinator.com/item?id=8171956
We should be monitoring the rate of test failures -- per test -- in a timeseries and looking for periodicities. Just graphing them and seeing regular spikes will help, but having real numerics looking for approximate periodicities will catch this sort of thing (after a few months!).
But, as a general rule, I would try to brute force or fuzz your "inputs", a.k.a data out of your control. Random numbers, dates, user input, network packets, etc. Users and environments often act in unexpected ways and you want to make sure your bug finding isn't reduced to accidentally be working on the right day! :)
But the bigger issue was that since it would "fix itself" we never prioritized it and just left it alone.
But looking back, the system (automated tests) was functioning as designed. :) We just changed our "DateTime.now" to concrete dates.
I always wonder why we gave up on refactoring instead of switching it along so that cross-program changes would become not just possible, but easy and feasible.
What can be done using alt-shift-r in a distributed large scale java app with a GWT frontend and 50 libraries used on the backend, each of which can run distributed across machines ... is baffling.
It shouldn't be !
And now with microservices, it becomes more and more baffling that this worked at one point. That's just horrible. Most of the complaints about it strike me as being complaints that when a toolchain allows for complexity, people make things complex. And of course, that's fair. If you give a 5 year old a laser CNC machine and he finds how to turn it on, things are going to go south quickly. But damn the things you can do with that !
Suprisingly it was possible to implement this in the course although one had to fiddle so much with Python's internals.
Zeller's chair homepage has a lot of additional infos. Here's a page on how the idea evolved after the initial paper in 1999: https://www.st.cs.uni-saarland.de/dd/
I guess this was before Git. I love git bisect
Also works for other kinds of files of course. Our version is easily seen as a kind of simulated annealing, with the input providing sufficient randomness that we did not need to add any internally.
Fun story: this was my first open source project. I had always wanted to be an open source programmer, but the precipitating cause of me releasing delta was that Microsoft Research asked me to. So back when Microsoft was saying that open source is a "cancer" I became an open source programmer because Microsoft asked me to.
Edited: I figured it out. It's because they essentially do a binary search of the changes to find the issue, I should have read a bit further before commenting. The tables / examples cleared it up pretty quickly.
