If you ask CoPilot to solve something it hasn't seen, it won't be able to solve it.
It's a transformer. Do you understand what that means? It's just matrix multiplication.
It generates maximum likelihood token strings, based on its training data.
It doesn't "understand" what those token string mean.
You are amazed because you're testing the transformer by asking the transformer to generate human-written code THAT IT WAS TRAINED ON. To make CoPilot fail, all you have to do is ask it to generate something unlikely, something it hasn't seen in training.
Maximum likelihood token strings. Period.
