I feel like a professor shouldn't consider teaching "overhead". If you're teaching classes, you should be putting at least as much effort into research as teaching. Prof != post-doc.
If your passion is in research, then a move to industry (or post-doc) sounds like a good choice.
Trying to deal with plagiarism (which even goes as far as assignments posted on RentACoder), using ancient marking systems (WebCT is well and truly evil), handling students who obviously don't care at all but feel entitled and so on.
Still, in my experience (which is nothing special, 4 years undergrad, in second year of masters) there are definitely professors who consider teaching an unfortunate side-requirement of their research. And I don't think it's unique to my school. I think this sucks.
I have been working as a teacher's assistant for a few semesters now, and I hear what you're saying about students who really don't care. I think especially in engineering, you get students who chose the major just because they got good grades in math and science and heard engineering pays well. Then those students complain when you ask them to work hard (or even just work).
Sidenote: WebCT :( haha. My school paid (and will continue to pay) a ridiculous amount for some enterprise course scheduling/billing/etc. software that would have been a great project in software engineering (e.g. large project management). All while we're having major budget issues.
"the amount of overhead and red tape (grant proposals, teaching, committee work, etc.) you have to do apart from the interesting technical work severely limits your ability to actually [have a practical impact via your research.]"
Teaching is part of the overhead if your goal is to do research, or in his case more precisely to build great systems. A lot of professors feel that way.
To Matt's credit, by all accounts he put plenty of effort into his classes and they were excellent, even though it wasn't what he really most wanted to do. But other professors who feel that way and, I suppose, have a weaker sense of duty end up as the bad teachers that students hate, because they just don't care.
However, the best place to do research is within academia (large research universities), where there is a very complete and strong infrastructure to do research. Yes, the system is broken in many ways, but it is still the best.
People who really have a strong passion for teaching and not as much interest in research go teach at universities that specialize in teaching more than in research e.g. Reed College.
Last, at a sufficiently advanced level teaching and research are indistinguishable.
This sucks for the world, because the research in these things that Google does doesn't really escape to advance the state of the art elsewhere.
It's a shame, because universities are essentially free labor, just on much longer timeframes and lower probability of immediate success. I understand why things are the way they are, but I'd love to find some way of reconciling this issue. It's already killed a research project of mine before it started.
1. Paranoia about giving competitors the edge they need.
2. Paranoia about giving spammers the edge they need.
3. Privacy. It's nigh impossible to anonymize user data enough to release and still have it in a useful form. The AOL logs debacle hurt everyone.
One of my friends had a really interesting take on this. His belief essentially was that the industry advances when people switch companies and (legal or not) take their institutional knowledge with them. Unfortunately, academia is hard to transfer into.
Of course things could always be better, but it could also be worse.
The question is whether Google is a career. Places like Google, Microsoft Research (and, perhaps Facebook) allow for the freedom that academics crave, but that's only three places. If research-style engineering goes out of fashion in favor of enterprise middle-management hell, you're going to be left high and dry if you don't have a publishing record to fall back on when you try going back to the Ivory Tower.
Fortunately, those places have enough cachet that you can say "I worked at Google" and no publishing record for those years are forgiven, but if you don't have one (say, you went half way through your PhD), coming back is going to be hard. That's why the decision is so difficult: you're taking a bet on whether the benefits taken for granted in academia are going to continue in industry. A broader shift would have to show more companies than those three, in order to provide (for want of a better word) "safety" that your worklife goals aren't in jeopardy.
I think it is a testament to how interesting the work in our field is.
http://matt-welsh.blogspot.com/2010/10/computing-at-scale-or...
So he can leave google for facebook