Literally any out-of-distribution project in which I used LLMs lead to catastrophic failure. The models can't "see" stuff outside their training data.
You should already know what to ask to extract the answer OpenAI claims gpt-5.2-pro gave them.
Then you should be lucky to get an answer that makes sense.
Then you should already know how to verify the model's response.
Only after all these steps should you cherry-pick the one-in-a-million successful response to feature on your website.
And finally, you should prove that the answer didn't already exist in the training data. It's highly likely that the problem was solved before and the model picked that up. I have yet to see a genuinely novel discovery these models can produce.
* I'm an LLM researcher, but that doesn't mean I should close my eyes to the unjustified hype around language models.
The human authors have positions at the Institute for Advanced Study (Einstein's old institution), Vanderbilt, Harvard (Strominger) and Cambridge in the UK.
If you have to gauge this by the reputation of the experts involved in it as I do, that seems like a good list to me.
