undefined | Better HN

0 pointsbeagle31y ago0 comments

Juergen Schmidhuber and his students came up with much of the basic NN elements, such as RNN, LSTM and others without behemoth resources (and several years earlier). Ttbomk, the one thing they DIDN’T come up with was the transformer; however it is very likely someone would have within a 5-10 year time frame.

0 comments

3 comments · 1 top-level

ldjkfkdsjnv1y ago· 2 in thread

I well aware of Schmidhuber, still the scaling of the compute was critical. The reason Schmidhuber didnt go all the way is still scaling/capital, which accrues to monopolies who can afford wildly speculative research. Also, LSTM, RNN, etc, while effective for their time, were dead ends.

beagle3OP1y ago

compute was critical, but compute was happening anyway. I fail to say why you are convinced that there would not be transformers.

ldjkfkdsjnv1y ago

I'm not saying that, I'm saying you need a company with huge amounts of cash on hand to spend on R&D. The monopoly provides a safe space for it, like it or not, this system has worked.

j / k navigate · click thread line to collapse