undefined | Better HN

0 pointsmarcus_holmes4mo ago0 comments

Yes, I've seen the same things.

But; they don't learn. You can add stuff to their context, but they never get better at doing things, don't really understand feedback. An LLM given a task a thousand times will produce similar results a thousand times; it won't get better at it, or even quicker at it.

And you can't ask them to explain their thinking. If they are thinking, and I agree they might, they don't have any awareness of that process (like we do).

I think if we crack both of those then we'd be a lot closer to something I can recognise as actually thinking.

0 comments

theptip4mo ago

> But; they don't learn

If we took your brain and perfectly digitized it on read-only hardware, would you expect to still “think”?

Do amnesiacs who are incapable of laying down long-term memories not think?

I personally believe that memory formation and learning are one of the biggest cruces for general intelligence, but I can easily imagine thinking occurring without memory. (Yes, this is potentially ethically very worrying.)

lz4004mo ago

>If we took your brain and perfectly digitized it on read-only hardware, would you expect to still “think”?

it wouldn't work probably, brains constantly alter themselves by forming new connections. Learning is inseparable from our intelligence.

WalterSear4mo ago

Our intelligence, yes. But that doesn't establish it as essential for thought.

1 more reply

zeroonetwothree4mo ago

> If we took your brain and perfectly digitized it on read-only hardware, would you expect to still “think”?

Perhaps this is already known, but I would think there is a high chance that our brains require "write access" to function. That is, the very process of neural activity inherently makes modifications to the underlying structure.

theptip4mo ago

> a high chance that our brains require "write access" to function

There are multiple learning mechanisms that happen on different time-frames, eg neural plasticity, hippocampus are both longer-term processes for memory consolidation. Whereas the content of “RAM” might be better modeled as a set of fast dynamic weights representing ions and neurotransmitter concentrations.

My hunch is that you could model the latter fast weights in volatile memory, but I wouldn’t count these as “modifying the structure”.

Do you have any particular systems in mind where you have reason to believe that permanent modification is required for brains to function?

xwolfi4mo ago

I wonder why we need to sleep so much though

2 more replies

trenchpilgrim4mo ago

> You can add stuff to their context, but they never get better at doing things, don't really understand feedback.

I was using Claude Code today and it was absolutely capable of taking feedback to change behavior?

marcus_holmesOP4mo ago

Within a session, yes, it will add each input to the context.

Start a new session with a blank context and it will not have learned anything.

trenchpilgrim4mo ago

> Start a new session with a blank context

Why would I use a blank context with Claude Code? I have it dump the stuff I want to save in CLAUDE.md and it reloads it in future sessions.

1 more reply

jatora4mo ago

This is just wrong though. They absolutely learn in-context in a single conversation within context limits. And they absolutely can explain their thinking; companies just block them from doing it.

j / k navigate · click thread line to collapse

0 comments

theptip4mo ago

> But; they don't learn

If we took your brain and perfectly digitized it on read-only hardware, would you expect to still “think”?

Do amnesiacs who are incapable of laying down long-term memories not think?

lz4004mo ago

>If we took your brain and perfectly digitized it on read-only hardware, would you expect to still “think”?

it wouldn't work probably, brains constantly alter themselves by forming new connections. Learning is inseparable from our intelligence.

WalterSear4mo ago

Our intelligence, yes. But that doesn't establish it as essential for thought.

1 more reply

zeroonetwothree4mo ago

> If we took your brain and perfectly digitized it on read-only hardware, would you expect to still “think”?

theptip4mo ago

> a high chance that our brains require "write access" to function

My hunch is that you could model the latter fast weights in volatile memory, but I wouldn’t count these as “modifying the structure”.

Do you have any particular systems in mind where you have reason to believe that permanent modification is required for brains to function?

xwolfi4mo ago

I wonder why we need to sleep so much though

2 more replies

trenchpilgrim4mo ago

> You can add stuff to their context, but they never get better at doing things, don't really understand feedback.

I was using Claude Code today and it was absolutely capable of taking feedback to change behavior?

marcus_holmesOP4mo ago

Within a session, yes, it will add each input to the context.

Start a new session with a blank context and it will not have learned anything.

trenchpilgrim4mo ago

> Start a new session with a blank context

Why would I use a blank context with Claude Code? I have it dump the stuff I want to save in CLAUDE.md and it reloads it in future sessions.

1 more reply

jatora4mo ago

This is just wrong though. They absolutely learn in-context in a single conversation within context limits. And they absolutely can explain their thinking; companies just block them from doing it.

j / k navigate · click thread line to collapse