Yeah, at the time, GB was still very big on mixture-of-expert models and bidirectional models like T5. (I'm not too enthusiastic about the former, but the latter has been a great model family and even if not GPT-3, still awesome.) DeepMind pivoted faster than GB, based on Gopher's reported training date, and GB followed some time after. But definitely neither had their own GPT-3-scale dense Transformer when GPT-3 was published.

Any “brash generalization” is clearly going to be grossly incorrect in concrete cases, and while I have a little gossip from true insiders, it’s nowhere near enough to make definitive statements about specific progress on teams at companies that I’ve never worked for.

I did a bit of disclaimer on my original post but not enough to withstand detailed scrutiny. This is sort of the trouble with trying to talk about cutting-edge research in what amounts to a tweet: what’s the right amount of oversimplified, emphatic statement to add legitimate insight but not overstep into being just full of shit.

I obviously don’t know that publication schedules at heavy-duty learning shops are deliberate and factor-in other publications. The only one I know anything concretely about is FAIR and even that’s badly dated knowledge.

I was trying to squeeze into a few hundred characters my very strong belief that Brain and DM haven’t let themselves be scooped since ResNet, based on my even stronger belief that no one has the muscle to do it.

To the extent that my oversimplification detracted from the conversation I regret that.