undefined | Better HN

0 pointsSanderNL3y ago0 comments

You have ingested truly astronomical amounts of data. Your body is covered with sensors of various kinds. Millions of datapoints streaming in day and night.

You are not storing it all, but you are finding patterns and correlations in it from the day you were born. This all forms a base where after a decade+ you can learn things fast from books, but that’s comparable to an LLM’s in context learning. It’s fast, but it depends on a deep base of prior knowledge.

0 comments

depr3y ago

My body may have "ingested" it in some way but filtering starts even at the sense ("sensor") level. So there is no way I have learned from it. I didn't even have to learn how to filter, it is part of my genetic makeup.

And even if I did, most of it would not be of the same category as what LLMs are learning; information about how a fabric feels or the sound of an ambulance doesn't teach me anything about programming or other things GPT can do. So when comparing the inputs, it makes no sense to count all the information that our senses are getting, to all the input for an LLM.

>This all forms a base where after a decade+ you can learn things fast from books, but that’s comparable to an LLM’s in context learning

Not really comparable

SanderNLOP3y ago

I think I implied a strong connection between LLMs and how biological intelligence works, but I certainly didn’t intend to.

I think you discount general knowledge acquired non-verbally too easily. The sound of an ambulance and how it moves through space and how it correlates with visual information represents an astronomical amount of information that can be generalized from handsomely. All these data streams have to be connected, they correlate and intertwine in fantastically complex ways.

I think the sound of an ambulance does teach you things that eventually help you “program”. You have witnessed similar (non-)events thousands if not millions of times. Each time it was accompanied with shitloads of sensory data from all modalities, both external and internal.

The base of general patterns you start out from once you are ready for language is staggering.

Again not saying LLMs work like that, because they do not. All I mean to do is put their information requirements in perspective. We ingest a lot more than a bunch of books.

j / k navigate · click thread line to collapse