undefined | Better HN

0 pointsmatusp2y ago0 comments

I wouldn't say it is an unexpected behavior. I remember reading papers about this memorization behavior few years ago (e.g., [1] is from 2019 and I believe it is not the first paper about this). It should be expected from OpenAI to know that LMs can exhibit memorizing behavior even after seeing the sample only once.

[1] https://bair.berkeley.edu/blog/2019/08/13/memorization/

0 comments

1 comments · 1 top-level

furyofantares2y ago

My expectation is that it can't memorize most of its training data. I expect it to memorize some.

j / k navigate · click thread line to collapse