undefined | Better HN

0 pointsjryio3mo ago0 comments

1 million tokens is great until you notice the long context scores fall off a cliff past 256K and the rest is basically vibes and auto compacting.

0 comments

3 comments · 2 top-level

olliepro3mo ago· 1 in thread

I bet they lack good long context training data and need to start a flywheel of collecting it via their api (from willing customers)

jbergqvist3mo ago

This would be my guess too. It can probably be generated synthetically or via agentic rollouts, but high quality long context examples where outputs meaningfully depend on long-range interactions probably remain scarce

rrr_oh_man3mo ago

It's the same now with Gemini as well. Unfortunately. :(

j / k navigate · click thread line to collapse