undefined | Better HN

0 pointsXCSme17d ago0 comments

They mentioned in their release page, that the Claude team noticed memorization of the SWE-bench test, so the test is actually in the training data.

Here: https://www.anthropic.com/news/claude-opus-4-7#:~:text=memor...

0 comments

sigmoid1016d ago

Any static benchmark older than 12-18 months is basically worthless, because the content will have spread all over the internet and have found its way into the latest model's training set.

William_BB16d ago

Good luck arguing with SWE benchmark purists

j / k navigate · click thread line to collapse

0 comments

sigmoid1016d ago

Any static benchmark older than 12-18 months is basically worthless, because the content will have spread all over the internet and have found its way into the latest model's training set.

William_BB16d ago

Good luck arguing with SWE benchmark purists

j / k navigate · click thread line to collapse