undefined | Better HN

0 pointscharcircuit2y ago0 comments

SOTA does not require being productionized. eg. GPT-3 was SOTA and it was not publicly accessible.

0 comments

There has to be some way to verify the claim. Trust me bro isn't science.

gpm2y ago

"Trust that I ran these tests with these results" is extremely common in science.

It's not an objective test like you are talking about. These benchmarks are far from accurate and also can be tainted in the training data.

The trust is established by others reproducing the results with the same methodology, it's not just supposed to be taking people's word at face value

j / k navigate · click thread line to collapse

There has to be some way to verify the claim. Trust me bro isn't science.

gpm2y ago

"Trust that I ran these tests with these results" is extremely common in science.

It's not an objective test like you are talking about. These benchmarks are far from accurate and also can be tainted in the training data.

The trust is established by others reproducing the results with the same methodology, it's not just supposed to be taking people's word at face value

j / k navigate · click thread line to collapse