A few will recall that neural networks all but died from the US after Minsky's damning book. Hinton gave backpropagation which is one of the foundational pillars of feed-forward neural net algorithms. With his new thrust on whats called "deep belief networks" he is challenging his own early seminal contribution in the field. Not often do you see researchers throw away such huge swathes of their own work and start again to solve the same problem. Unless you are Niklaus Wirth of course.
Some background is necessary to get the inside jokes, but I have tried to minimize the requirement.
Geoff Hinton doesn't need to make hidden units. They hide
by themselves when he approaches.
Geoff Hinton discovered how the brain really works.
Once a year for the last 25 years.
Markov random fields think Geoff Hinton is intractable.
Geoff Hinton can make you regret without bounds.
Geoff Hinton doesn't need support vectors. He can
support high-dimensional hyperplanes with his pinky.
All kernels that ever dared approaching Geoff Hinton woke up convolved.
The only kernel Geoff Hinton has ever used is a kernel of truth.
After an encounter with Geoff Hinton, support vectors become unhinged
Geoff Hinton's generalizations are boundless.
Geoff Hinton goes directly to third Bayes.
http://en.wikipedia.org/wiki/Backpropagation
http://en.wikipedia.org/wiki/Perceptrons_%28book%29
http://en.wikipedia.org/wiki/Niklaus_Wirth
Re: Downvotes. I seem to have touched a nerve. Yes I agree humor is frowned upon here, and I largely agree with that, but when its not humor for humors own sake, I sometimes make an exception. Not everyone would know who Hinton is, but may gauge that he is someone important from these fun anecdotes. But of course you are free to like it or dislike it, no hard feelings either which way.
> With his new thrust on whats called "deep belief networks" he is challenging his own early seminal contribution in the field
I don't know if I agree with that, he still uses backprop. Backprop has always been known to have problems when you scale to millions of connections, and his work on RBMs/DBNs is really quite old. What was novel more recently was showing that the contrastive divergence step need only be performed once, rather than 100 times, while the performance remained similar. The networks are generally still 'fine tuned' with backprop.
Still, the focus on generative networks (not sure if that's the right term still, been a while) and single layer training is fairly recent even if the concepts are quite old.
On the comment that RBMs are new, now I have to come to accept that if one looks hard enough almost all things are old, only the name changes !
I can imagine that this sentiment is probably against the prevailing HN mood, but I've been thinking a lot lately about how certain kinds of thought and investigation are only enabled and supported by certain structures. The point of a university used to be being the highest pinnacle of thought. A place where people could have everything not related to intellectual pursuit taken care of so they could devote themselves to the pursuit of greater knowledge. Now that seems to be replaced by a corporate campus.
It's a cliche, of course, but how many breakthroughs of meaning are we missing out on because the brightest and best of our generation are now no longer seeking after truth, but instead seeking after a way to make more people click on an ad? Ah well, it's late, and as ever I'm post-sober. Best of luck to him and his team - I've enjoyed his lectures and papers. I hope they may continue, but I sadly suspect not.
I don't even think it's a conclusion that's the result of actual thought anymore, just reverb.
Google is a big company. It would not even be worth it or effective to have every single engineer dedicated to this ad click business. There's a team for that, and there are lots of other teams, for lots of other interesting things. Many of them do serious intellectual research.
My impression of Google's leadership, employees, and choice of projects often leads me to believe that they sell ads so they can continue making really cool stuff - not that they try to come up with cool stuff so that they can sell ads.
I also disagree with the whole premise that goal-oriented or profit-driven intellectual study is in some way less pure, less worthy, less boundary-pushing, or more short-sighted than purely academic study. How does the initial incentive for a thought affect whether or not the conclusions from that thinking are new knowledge about our universe? It doesn't.
As long as he's not sacrificing what he does to become a paper pusher, which he isn't doing, I don't see how it's different. And I think ultimately the closer his study is to being manifested in real things that we use, the faster his knowledge will be refined towards the path of truth, and the faster humanity will actually learn and benefit.
Now that's lazy thinking. No one accused Google or Verizon or of not having "many people" that do serious research.
Personally, I see absolutely no problem with what he did, he has a family and his future to worry about. After all these years he deserves to make a lot of money money and not have to worry about money anymore. Those that criticize him for "selling out" will probably sell their start-up to just about anyone for FU money.
What I find not true, especially in the past year or two is this:
"My impression of Google's leadership, employees, and choice of projects often leads me to believe that they sell ads so they can continue making really cool stuff - not that they try to come up with cool stuff so that they can sell ads."
Part of my job requires me to surf without ad-blockers and with Chrome. There's no trick in the book they do not use to make you click on ads or sign for their services. Even Chrome, the "open source" the supposed savior of the web now has ads. It totally changed my perception of them, and it's not like Google was struggling to pay their electricity bill. As a commercial enterprise, Google has the right to do that, however they cannot have their cake and eat it too.
Research professors at schools have been mostly funded by either commercial or military interests for a very long time now. Even in non-direct cases, universities have "tech transfer" offices that are commercializing stuff, and continued funding often depends on that.
If you are going to claim this model is bad and that it produces bad results, you are going to have to offer at least a little evidence that the models that existed before this, worked, and were better for humanity.
I might agree with the second part in some ways, but I dont' think they supported anywhere near the amount or level of research that occurs now.
Research is definitely more goal directed than it used to be, but this is not totally good or totally bad, it's just "different".
(It should also be, incidentally, taught as one of the great examples of unintended consequences: by all accounts it was designed to increase the independence of universities by letting them keep the monies they got from commercialization. Then somewhere along the way policy-makers realized Bayh-Dole was a great excuse to encourage universities to do tech transfer. And now essentially every grant that doesn't have some tech transfer proposal is almost automatically in trouble when compared against those that do.)
I have a different reaction than you. It's good to see brilliant minds stepping outside of the academia and join a commercial entity to actually build something.
Advertising is just a means for providing comfortable livelihoods to these excellent people. There's no better place for these people to be working. We got some spectacular output when some bright folks ended up working together in a phone company, better than any university. Why not Google?
I'd say it's a bit more than that; what Google has done in advertising has benefitted millions of consumers and businesses. I'm not sure why people tend to look down on some of the work they do in advertising as somehow being less worthy than working on GMail or Search!
"I would argue that Google has had the largest impact on information dissemination , since the invention of pen and paper."
Because they are a search engine and is used by most people. Now, however, Google is replacing information with ads and with sub-par information that is beneficial to their bottom line. That hurts creators, readers and society in general.
Google also serves the purpose of amplifying the power of great (and not so great) minds by putting so much information at their fingertips.
The ads part is just what writes the cheques.
Does the research get published or otherwise make it outside of the Googleplex? Serious question.
To bring out the real magic out of techniques like deep learning ( http://en.wikipedia.org/wiki/Deep_learning ), availability of large training sets and the infrastructure required to crunch them are a pre-requisite. Once you have that, it is turning out to be a different ball game all together http://deeplearning.net/2012/12/13/googles-large-scale-deep-.... It turns out that groups like google research are the ones at present which have access to such dataset and infrastructure.
I also predict the reverse shift to happen within few years, once the interesting fundamental research problems has been tackled such people might move back to universities. If that happens, that is indeed a healthy process of academica and industry supplementing each other.
The saddest part of corporations replacing academia won't be realized for decades but it's that the death of knowledge is tied too closely with profitability.
The decision of what to do with one's life rests entirely in him or her. But we should ask as a society, what are we striving for? The same goes to how our society (I can only speak of the US) looks at teaching as a profession.
</zenmode>
But after a second thought, I can understand why he did it and if I were in his shoes I would do the same.
I don't think it has to do anything with money. For a connectionist, it is a dream to have access to the massive amount of CPUs that Google has. Take that and add all the incredible and smart people that Google has in its team and then you would see his decision was a no-brainer. I feel very happy for him.
Academia --------
1) Maximum likelihood from incomplete data via the EM algorithm, 1977. Dempster, Harvard.
2) Communicating sequential processes, 1978. Hoare, Oxford at time of publication.
4) Chord: A scalable peer-to-peer lookup service for internet applications, 2001. Stocia, Berkeley.
5) Distinctive Image Features from Scale-Invariant Keypoints, 2004. Lowe, UBC.
6) Induction of Decision Trees, 1986. Quinlan, New South Wales Institute of Technology.
7) Reinforcement Learning: An introduction, 1998. Sutton, UAlberta.
9) Graph-based Algorithms for Boolean Function Manipulation, 1986. Bryant, CalTech.
10) Scheduling Algorithms for Multiprogramming in a Hard-Real-Time Environment, 1973. Liu, MIT.
11) The anatomy of a large-scale hypertextual web search engine, 1998. Brin, Stanford.
12) A method for obtaining digital signatures and public-key cryptosystems, 1978. Rivest, MIT.
14) A scalable content-addressable network, 2001. Ratnasamy, Berkeley.
15) New directions in cryptography, 1976. Diffie, Stanford.
16) Eigenfaces for recognition, 1991. Turk, MIT.
17) Authoritative sources in a hyperlinked environment, 1999. Kleinberg, Cornell.
18) Indexing by latent semantic analysis, 1990. Deerwester, UChicago.
20) Handbook of Applied Cryptography, 1996. Menezes, UWaterloo.
Industry --------
3) A tutorial on hidden Markov models and selected applications in speech recognition, 1989. LR Rabiner, Bell Labs.
8) Optimization by simulation annealing, 1983. Kirkpatrick, IBM.
13) Snakes — active contour models, 1987. Kass, Schlumberger Palo Alto Research.
19) Fast algorithms for mining association rules, 1994. Agrawal, IBM.
I think the trope of "the best minds working on ad clicks" is perhaps not giving people enough credit.
Hackers and academics always find a way to screw around and work on interesting things. The trick is to find interesting structure in mundane problems.
Sounds good on paper, but I'd question if that was ever actually true in reality anyway.
Now that seems to be replaced by a corporate campus
And? Is research less valuable because it's done by a corporation? Look at how much basic research IBM sponsors and how much Bell Labs and others have sponsored in the past. If anything, I would go the opposite direction and lament the decline in corporate spending on fundamental research. I'd like to see more academics go into the business world.
"Startups are the new graduate school" - can't remember where I heard that quote, but I think there's something to it.
EDIT: Can't have apostrophes in links
Search wikipedia for Student's t-test history
old, broken link: http://en.wikipedia.org/wiki/Students_t-test#History
--
[1] For an overview of deep belief networks, see these videos: http://www.youtube.com/watch?v=AyzOUbkUf3M , http://www.youtube.com/watch?v=VdIURAu1-aU , and http://www.youtube.com/watch?v=DleXA5ADG78
http://thenextweb.com/google/2013/03/12/google-acquires-cana...
Or the world will enter into a dark age of knowledge, ruled by huge corporations, and everybody else having to share the crumbs that fall of the table..
I hope that not all big minds fall of for companies.. or at least that companies start to commit with knowledge shareing to the rest of the world
we should stop black box knowledge companies or we are doomed in the long term.. (like the coke secret recipe case)
You mean like this very large list? http://research.google.com/pubs/papers.html
> and open source software
Or this also very large list? https://code.google.com/hosting/search?q=label:google
ps: while google have a pretty much decent portfolio of open source and papers floating around.. its more about "marginal software" for google...
the core of its knowledge about the business still a secret.. what would be of the world now and the cloud economy without Doug Cutting, that do it all by himself?
In one sentence: is only ONE company rulled by the capital market and profits and thats pretty scary.. thats all :s
http://www.nytimes.com/2012/11/24/science/scientists-see-adv...
One of the most striking aspects of the research led by Dr. Hinton is that it has taken place largely without the patent restrictions and bitter infighting over intellectual property that characterize high-technology fields.
“We decided early on not to make money out of this, but just to sort of spread it to infect everybody,” he said. “These companies are terribly pleased with this.”
http://www.google.com/patents/licensing/index.html
Though Google seem to think you can still make money despite/because of sharing, through other means than patent licencing.
Add to this that it is an enormous pain to solve engineering problems at a university, you usually have to hand this to students, limiting what you can produce and maintain. The best I have ever seen for a group was a single full-time software engineer and this was at arguably the most prestigious university in the world. And don't get me started on the fact that there is a poor incentive to produce good software in academia, even though said software can be essential to make research possible.
I don't know Hinton in person, but I can imagine that at his age both the possibility of something new and the promise of a strong engineering and financial backing for a large group is enormously tempting (also, do they force professors into retirement or start denying them grants in Canada?). Oh, and to those pointing out the potential salary, if my knowledge of the financial situation of professors that have been on tenure is generalisable I would be surprised if personal finances would mean much at this point.
It's not hard to see why a neural network researcher in particular would get excited about having access to tremendous amounts of data and computer clusters.
Not that it's a bad thing. Lots of smart people did interesting things at IBM, just as people at Microsoft Research are doing some great things - albeit usually without anyone seeing the work in progress.
Anyway, if you are unfamiliar with his work, here's Hinton't Deep Learning Saga ;) for your enjoyment: www.youtube.com/watch?v=mlXzufEk-2E
Hot topics now seem to be statistical ML stuff, and not the deterministic mathematical proofs that he was an advocate of.