undefined | Better HN

0 pointswwarner1y ago0 comments

well that's exactly the point -- no such result is available for language models.

0 comments

There are multiple papers and efforts that have inspected the internal state of LLMs. One could even see the word2vec analysis along these lines, as evidence that the model is specializing neurons

One such example: The Internal State of an LLM Knows When It's Lying (https://arxiv.org/abs/2304.13734)

Searching phrases like "llm interpretability" and "llm activation analysis" uncover more

https://github.com/JShollaj/awesome-llm-interpretability

wwarnerOP1y ago

Yes, lots of activity in the space. I thought you were saying it was a dumb problem, but I was wrong.

I think this is a great paper.

verdverm1y ago

yup, if you look at drop out, what it does and why, you can see additional interesting results along these lines

(drop-out was found to increase resilience in models because they had to encode information in the weights differently, i.e. could not rely on single neuron (at the limit))

1 more reply

j / k navigate · click thread line to collapse

0 comments

verdverm1y ago

There are multiple papers and efforts that have inspected the internal state of LLMs. One could even see the word2vec analysis along these lines, as evidence that the model is specializing neurons

One such example: The Internal State of an LLM Knows When It's Lying (https://arxiv.org/abs/2304.13734)

Searching phrases like "llm interpretability" and "llm activation analysis" uncover more

https://github.com/JShollaj/awesome-llm-interpretability

wwarnerOP1y ago

Yes, lots of activity in the space. I thought you were saying it was a dumb problem, but I was wrong.

I think this is a great paper.

verdverm1y ago

yup, if you look at drop out, what it does and why, you can see additional interesting results along these lines

(drop-out was found to increase resilience in models because they had to encode information in the weights differently, i.e. could not rely on single neuron (at the limit))

1 more reply

j / k navigate · click thread line to collapse