GPT-3 can generate well-formed programs, so yes, it does things well beyond this complexity.

> After 500,000 iterations I'm still getting invalid sequences.

How frequently? If it's only the occasional issue it might be down to the temperature-based sampling that code uses, which means it will, with some small probability, return arbitrarily unlikely outputs.