undefined | Better HN

0 pointsllm_trw1y ago0 comments

That's a distinction without a difference. The majority of data is from a distribution that's already been sampled multiple times.

E.g. how often does a baby go out and experience something novel? The majority of it's time is spent getting the same stimulus over and over again, as anyone listening to childrens television can attest.

Humans learn in fundamentally different ways to our current systems and information poverty is not a problem for us.

0 comments

sigmoid101y ago

And what do you think epochs in machine learning are? Or why more modern training efforts (i.e. for LLMs) are focussing hard on deduplicating scraped data?

llm_trwOP1y ago

Why don't you tell me instead of asking questions that you surely know the answer for?

sigmoid101y ago

It was rhetorical. But in case you actually don't know: what you described (i.e. multi sampling) has been common practice in ML for ages. Only now the latest models are getting so big that people are actually trying hard to move away from this idea because it would take a human lifetime in wall clock time to train a cutting edge LLM on similar datastreams.

j / k navigate · click thread line to collapse

0 pointsllm_trw1y ago0 comments

That's a distinction without a difference. The majority of data is from a distribution that's already been sampled multiple times.

Humans learn in fundamentally different ways to our current systems and information poverty is not a problem for us.

0 comments

sigmoid101y ago

And what do you think epochs in machine learning are? Or why more modern training efforts (i.e. for LLMs) are focussing hard on deduplicating scraped data?

llm_trwOP1y ago

Why don't you tell me instead of asking questions that you surely know the answer for?

sigmoid101y ago

j / k navigate · click thread line to collapse