It often starts hallucinating the input in the exact format advent of code gives, so I'm like 99.9% certain it has been trained on a large corpus of advent of code.

Personally I use codex sometimes for debugging help so I agree it can do cool stuff. I just disagree it is "almost" right at solving problems -- it almost never generates code that even compiles for me when prompted to do anything not super trivial like advent of code brain teasers.