Early Evals for OpenAI O3 (opens in new tab)

(openai.com)

30 pointsmaurycy1y ago2 comments

2 comments

Wow, the demo where the user asks for untraceable payments shows some pretty sophisticated reasoning. The word "crafty" comes to mind.

New SOTA's on:

SWE-Bench - 71.7

Competition Code - 2727

ARC (Semi Private Eval) - 75.7 on low, 87.5% on high compute

Frontier Math (previous SOTA was 2%) - 25% on high compute

j / k navigate · click thread line to collapse

Wow, the demo where the user asks for untraceable payments shows some pretty sophisticated reasoning. The word "crafty" comes to mind.

New SOTA's on:

SWE-Bench - 71.7

Competition Code - 2727

ARC (Semi Private Eval) - 75.7 on low, 87.5% on high compute

Frontier Math (previous SOTA was 2%) - 25% on high compute

j / k navigate · click thread line to collapse