However, while these systems based on GPT-3, and similar techniques, can produce some amazing outputs given the right inputs, you don't have to poke them very hard before they start falling apart.
At the end of the day, this isn't really any more "clever" than a very, very advanced and long markov chain, there isn't any intelligence.
https://en.wikipedia.org/wiki/AI_effect - at least sort of
I get what you're saying, but is this really as fragile as you claim? I've been rather impressed by (GPT) input handling.
If anything, this is "less clever" than older techniques like tree-search, as it is very easy to confuse, and will start just producing nonsense.
Or, seen another way, it just has to be smarter than most peoples' web frameworks, or perhaps their IDEs, which is not a very high bar.
_________________
[1] Going by Sharif Shameem's Twitter profile (or bio, whatsitcalled):
founder at https://debuild.co – a faster way to develop web apps // @ycombinator alum // prev built @VectordashGPU // relentlessly optimistic
I guess you could create some kind of UI prototyping app, but better options for that already exist, which don't require you to write everything out.
1. The way I think about these huge models is no longer as something that makes predictions, but rather as a kind of huge knowledge base (compressed training data as representations) with smart query capabilities, which come from the fact that the model is constrained to create syntactically correct output. This could be the next generation of search. There is a fine line between search and creating novel outputs, since most novelty is just a combination of old things with constraints on correct syntax.
2. The shown results are likely a result of massive cherry-picking in the typical "pitch deck demo" fashion. You'll see a lot more failures when trying to use this and you need to get the query "just right" for it to produce good output, which a skill in itself. Also let's not forget that the model must be fed fine-tuning examples, which also need to be "just right.". That's why I believe these kinds of models aren't that useful for fully automating things - for something to be used in production it should be 100% correct - but they are very useful as a search function and alternative to e.g. StackOverflow in this case. Querying the model will give you good enough results that you can use as a starting point for your own use case.
That said the way he generates a table is in no way idiomatic react, where you take an array of objects and map the data on to the table.
In terms of code generation, is it actually any better than using an emmet expression? `table>tr*10>td` (I've probably mangled that). What's interesting with the table example is probably the combination of "knowledge" and knowing enough about a DSL to generate a coherent output.
I agree re cherry picking, though the initial fine tuning set is extraordinarily small, just 2 items. Generating a much larger dataset shouldn't actually be that tricky, though perhaps getting agreement on how to actually describe a layout in plain English might be more of a challenge. Would a plethora of descriptions for the same layout be a bad thing?
Lastly, I've been using TabNine (https://www.tabnine.com/) for the past few months, and I'm convinced that some kind of language model text generation is going to change how we write code, it's already changed it for me. If I signal my intent pretty clearly up front, I would say most of my coding is now tabbing and occasionally pressing the down arrow. Even writing comments becomes an exercise writing an initial three of four words and tabbing out the rest. Ditto tests.
Check a look https://www.youtube.com/watch?v=_8yVOC4ciXc
The fact that the Transformer attention mechanisms works based on matching queries and keys/values is probably why the model is so good at finding relevant information to your input - that's kind of what it was trained to do.
The GPT-3 model hasn't been published, and training it yourself would probably cost millions of dollars...
i.e.:
You didn't say what colour 'welcome to my news letter' should be, so I picked white. The numbers from 1-5 are totally a possible random permutation in the 1-10 range. I find the crying laughing emoji to be really ugly, so it's the worst one.
I was wrong. It's coming for front-end engineers and it's coming soon.
