undefined | Better HN

0 pointscs7029mo ago0 comments

Great response. Yes, in some cases concentration may be desirable.

However, I'm not persuaded it was necessary in the specific cases you mention:

* Waymo: EVs were repeatedly killed by corporations highly in concentrated industries that would suffer disruption by EVs: https://en.wikipedia.org/wiki/Who_Killed_the_Electric_Car%3F

* TSMC: Wouldn't we all be better off if the entire world weren't so dependent on a single company, located in a such a geopolitically sensitive territory?

* 10B-param LLMs: Wouldn't it have happened regardless, once everyone realized that increasing the scale of early models like GPT-2 and GPT-3 was key to improving performance? I'd add that the model that launched the deep learning craze (AlexNet) and the model that launched the LLM craze and (the Transformer) were developed by tiny teams on the cheap.

0 comments

5 comments · 3 top-level

chermi9mo ago· 2 in thread

"Wouldn't it have happened regardless, once everyone realized that increasing the scale of early models like GPT-2 and GPT-3 was key to improving performance?" Notice how it was massive private spending that uncovered the power of scale in the first place. Would've been hard to say, get a federal grant for that. Would've probably happened gradually, with some gains from moderate scale justifying slightly larger grants for successively larger scale.

As for TSMC, the counterfactual assumes such technology would've happened regardless. Just because technology seems to happen inevitably, doesn't make it so. We have evidence of one approach (private) giving incredible results. And also some examples of public (in wartime) giving incredible results. I don't know the evidence for peacetime public incredible results. Maybe warpspeed?

cs702OP9mo ago

The cost of developing GPT-2 and GPT-3 was on the order of millions of dollars, well within the budget of most tech organizations. See https://arxiv.org/abs/2005.14165 for the total compute invested in them. OpenAI had raised only a few tens of millions in donations at the time, as a non-profit organization.

The increase in performance of computation has been happening for so many decades now that it's been given names like "Moore's Law." People like Hans Moravec predicted way back in the 1980's that the cost of compute would continue to decline and become cheap enough for AGI by the 2020's or 2030's. That's half a century ago!

chermi9mo ago

Fair point, the first few gens weren't that expensive. And like I said I'm certain the scaling would've been discovered soon with some time lag. But the transformer paper was 2017, right? Just from a benefit -to- society perspective, assuming LLMs are a net positive in terms of productivity, perhaps reducing time to drug approval or improving government efficiency (1).. Isn't getting there just one year earlier worth it, if the gains really are that big? We could be talking lives saved via faster approvals or more efficient spending. My point is that a private company made it happen faster, as evidenced by them doing it first. A good thing sooner is valuable.

(1) I'm convinced at the very least LLMs can feasibly speed up paperwork.

1 more reply

fragmede9mo ago

> Waymo: EVs were repeatedly killed by corporations

Not to detract from your point, but Waymo's happen to use electric iPace Jaguar cars in several cities that they serve, but their original self driving taxi service used gasoline minivans at beginning in Phoenix, AZ. EV vs ICE is orthogonal to Waymo's self-driving car technology. Waymo was a pure R&D self driving project for 15 or so years that Google/Alphabet dumped insane amounts of money into before a car ride was ever sold to the public. The are a few competitors to Waymo, at various stages, so market forces likely still would have resulted in self driving car technology eventually arriving, but as its competitors are also well funded, so it seems like it still takes a large org to turn university prototypes into a real live product.

marcosdumay9mo ago

About TSMC, maybe not.

The reason the world is so dependent on a single company is because it costs country-breaking amounts of money to keep-up with the semiconductor manufacturing technology. You can only have cheap semiconductors if there are very few entites building them.

j / k navigate · click thread line to collapse