undefined | Better HN

0 pointswcoenen3y ago0 comments

> Learning and remembering things outside the 2048 token context window

It is probably possible to do this with fine tuning.

Once the context window is full, it might be possible to convert the content in there into additional training examples and feed those to a fine tuning process, which then retrains the model. (The OpenAI API for fine tuning is here: https://platform.openai.com/docs/guides/fine-tuning)

It would be a bit like sleeping. Whenever the context window fills up, the model would have to go offline for a while to move memories from its context window (short term) to its network weights (long term).

0 comments

dr_dshiv3y ago

So an 18 hour long token window is acceptable as AGI, I like it.

j / k navigate · click thread line to collapse