Because of this thread, I looked through my old backups and I actually still have the code. Should get it working again sometime
It would be interesting to see how to think through building a crawler (as opposed to downloading Nutch and trying to grok it)