undefined | Better HN

0 pointsdanielmarkbruce2y ago0 comments

The idea that there has been no progress in functionality is silly.

Your whole brain might just be doing "information compression" by that analogy. An LLM is sort of learning concepts. Even Word2Vec "learned" than king - male + female = queen and that's a small model that's really just one part (not exact, but similar) of a transformer.

0 comments

ActorNightly2y ago

Let me rephrase that.

One level deep information compression is cool, but I want actual AI.

Its true that our brains compress information, but we compress it in a much more complex manner, in the sense that we can not only recall stuff, but also execute a decision tree that often involves physical actions to find the answer we are looking for.

danielmarkbruceOP2y ago

An LLM isn't just recalling stuff. Brand new stuff, which it never saw in it's training, can come out.

The minute you take a token and turn it into an embedding, then start changing the numbers in that embedding based on other embeddings and learned weights, you are playing around with concepts.

As for executing a decision tree, ReAct or Tree of Thought or Graph of Thought is doing that. It might not be doing it as well as a human does, on certain tasks, but it's pretty darn amazing.

ActorNightly2y ago

>Brand new stuff, which it never saw in it's training, can come out.

Sort of. You can get LLMs to produce some new things, but these are statistical averages of existing information. Its kinda like a static "knowledge tree", where it can do some interpolation, but even then, its interpolation based on statistically occurring text.

danielmarkbruceOP2y ago

The interpolation isn't really based on statistically occurring text. It's based on statistically occurring concepts. A single token can have many meanings depending on context and many tokens can represent a concept depending on context. A (good) LLM is capturing that.

1 more reply

j / k navigate · click thread line to collapse

0 comments

ActorNightly2y ago

Let me rephrase that.

One level deep information compression is cool, but I want actual AI.

danielmarkbruceOP2y ago

An LLM isn't just recalling stuff. Brand new stuff, which it never saw in it's training, can come out.

The minute you take a token and turn it into an embedding, then start changing the numbers in that embedding based on other embeddings and learned weights, you are playing around with concepts.

As for executing a decision tree, ReAct or Tree of Thought or Graph of Thought is doing that. It might not be doing it as well as a human does, on certain tasks, but it's pretty darn amazing.

ActorNightly2y ago

>Brand new stuff, which it never saw in it's training, can come out.

danielmarkbruceOP2y ago

1 more reply

j / k navigate · click thread line to collapse