undefined | Better HN

0 pointszamadatix1y ago0 comments

Sorry, I probably phrased the question poorly. My question is more along the lines of "when you already scored e.g. OpenAI's o3 on ARC AGI 2 how did you guarantee OpenAI can't just look at its server logs to see question set 4"?

0 comments

gkamradt1y ago

Ah yes, two things

1. We had a no-data retention agreement with them. We were assured by the highest level of their company + security division that the box our test was run on would be wiped after testing

2. We only tested o3 against the semi-private set. We didn't test it with the private eval.

QuadmasterXLII1y ago

Are you aware that OpenAI brazenly lied and went back on its word about its corporate structure, board governance, and for-profit status, and of the opinion that your data sharing agreement is different and less likely to be ignored? Or are you at step zero where you aren’t considering malfeasance as a possibility at all?

zamadatixOP1y ago

Makes sense, particularly part 2 until "the final results" are needed. Thanks for taking the time to answer my question!

YeGoblynQueenne1y ago

>> We were assured by the highest level of their company + security division that the box our test was run on would be wiped after testing

Yuri Geller assured us he was bending the spoons with his mind. Somehow it was only when the Amazing Randi was present that Yuri Geller couldn't bend the spoons with his mind.

levocardia1y ago

Ironically "I have a magic AI test but nobody is allowed to use it" is a lot closer to the Yuri Geller situation. Tests are meant to be taken, that should be clear. And...maybe this does not apply in the academic domain, but to some extent if you cheat on an AI test "you're only cheating yourself."

1 more reply

j / k navigate · click thread line to collapse