> What may take a cluster to compute one year takes a consumer machine the next.
Is that not partly because the hardware is ever improving? I realize this is a bit of exaggeration, but does not yesterday's cluster end up fitting onto the die of tomorrow's GPU? And then since it's all on a single die, is not the overhead of the interconnect drastically reduced? It takes less time to push information to the next core over when the interconnect is a couple micrometers of silicon instead of the couple meters of silicon, copper, and fiber needed when the next core is in the next rack over.
Certainly improving the model will help; who hasn't marvelled at how better his code ran when he fixed that On^2 hot spot? But I can't help but think improving hardware plays a role too.
Am I off base here?
The only reason deep learning exists is because by now we've finally learned how to build GPUs fast enough to run the algorithm invented in 1983.
And let's be honest - most of the current, state-of-the-art algorithms only work today because they've got access to scaled up massive databases of data. You don't really need to be as smart here any more.
Neural Architecture Search: "32,400-43,200 GPU hours" Just over a year later: "single Nvidia GTX 1080Ti GPU, the search for architectures takes less than 16 hours" (1000x less) (paper)
If I remember correctly it was X=5 and n=2.
Moore’s law was magical
Most of hard problems in computer science are that of highly-branched connectivity (both in terms of the number of connections that are mode from the compute node and in terms of the branching required in the algorithms).
E.G. 1 - branched) if it takes you 100ms to send 1Gb of data over network, but you don't have to wait for it's result, then latency does not matter at all.
E.G. 2 - number of connections to the node) if you have a matrix value that depends on a full row in a matrix, this matrix value somehow needs to be computationally connected to a full row in a matrix. Cluster or GPU, these physical connections need to be made either through Ethernet or by reading values from RAM. Again, the latency does not matter as long as you can pipeline these.
So frankly, no. It's not the micrometers on the silicon that matters, it's just that our hardware had been scaled so much that a 50 year old compute cluster can be literally scaled down to a single die. It's _not_ more efficient, though.
Ok? This is not a rebuttal of the parent comment at all. You could also read and write the data on notecards and compute with a mechanical calculator, but that's totally irrelevant to the parent's point. Hardware matters a lot. Other things certainly matter as well, but saying the structure of the computation is fixed doesn't mean hardware can't change the performance (in absolute perf, perf/W, and perf/$) by many orders of magnitude.
https://www.slideshare.net/FlorianWilhelm2/performance-evalu...
The authors were able to outperform Google ML by a large margin for a vision task that involved recognizing numbers from car registration documents. With just 160 manually collected training samples they were able to train a neural net that could recognize characters with 99.7 % accuracy. GoogleML performed very poorly in comparison, which I found very surprising because it didn't seem to be such a hard recognition task (clean, machine-written characters on a structured, green background).
Another concern is generality: just because it performs well on this dataset does not mean it will perform well on another.
It's also about flexibility: If Google ML doesn't provide you with a way to train their algorithms specifically for your use case it won't help you that they work well for generic text recognition tasks.
And most people don't want to solve many different ML problems but just a single one, so I think the result is quite interesting.
https://www.youtube.com/watch?v=bWMrY49qqDw
In the 11th century, a wooden palisade or an earthen berm fortification could be held for something like a half year. By the end of WWII, it constituted a delaying tactic.
https://en.wikipedia.org/wiki/Rhino_tank
A phase change happened with military tactics in the lead-up to the 1st half of the 20th century, where the power of mobile mechanized armor and air support greatly reduced the value of fortifications.
That said, I don't think moats are dead. It's just that the time-scales have changed.
E.g. Compute wars have only intensified with TPUs and FPGA. sure for training you might be okay with few 1080ti but good luck building any reliable, cheap and low latency service that uses DNNs. Similarly big data for academia is few terabytes but real Big data is Petabytes of street level imagery, Videos/Audio etc.
You can write a lot of papers about Penn Treebank data but I can't imagine anything you do with Penn Treebank will be commercially useful.
I just wonder if it will ever feel this way for reinforcement learning.
