These emergent abilities are not actually that, but a result of humans' poor understanding of cognition and communication.
What concerns me very much is how the harms that can be caused by LLMs has been so greatly under reported.
I imagine being someone with the right power and access telling ChatGPT "find all people that would vote against this candidate in real time and devise ad content and social media messaging and bit interaction to change their minds or discourage them from voting" heck, any intel org of a major country is probably already working on this. No more whistleblowing or posting anonymously on social media, companies would even share models based on private email and conversations you had so other companies could use LLMs to identify everything you posted elsewhere and to have LLMs designate a score for hoe hireable you are. Police can crack down on crime better but also crack down on dissent or any police reforms.
And we aren't even talking about war time use of LLMs or what happens when you marry something like ChatGPT with Dall-E and make it all real-time.
I am warning anyone who will listen. Smartphones are the most dangerous things out there. Any service or interaction that depends on them is deteimental to peace and liberty of the masses long term. People have not learned a thing from Snowden or 2016 elections.
And why are all the smart journalists asleep on the job on this topic. Where are the unreasonable scaremongerers when you need them!
This doesn't seem to make sense. If anything the opposite is true - if the things that are hallucinated make sense (even if not true) it means there is some "understanding" or a world model.
For example, you tell Dall-E to draw a picutre of a man smoking a pipe but then it draws the pipe coming out of the mans butt and instead of a head the man has a leg on his neck. This is approximation. Now if the pipe looked wrong or it was a trumpet or the guy's head looked weird but still a head maybe it knows what a head is and it just made a mistake and it knows that the pipe goes into the mouth so it will be somewhere near the face.
Undesrstanding is a 3yr old child drawing terribly, approximarion is drawing really well but all wrong.
All this applies to LLMs I used Dall-E because pictures are easier to talk about.
You can see this because of the big deal people made about image gen tools getting hands wrong: it was the most significant error that was systematically occuring.
Alexander Grothendieck, one of the greatest mathematicians of the 20th century, answered "57" when asked in a public lecture to provide an example of a prime number. 57 = 3 * 19 is not a prime number.
According to your argument, this would imply Grothendieck did not understand what a prime number is. Which is laughable.
Also mathematics is not the right field to compare this to because there are rules. In languages and object recognition/synthesis it is all subjective. Understanding here means understanding context and human-subjective interpretation.
When I say "that sunset is beautiful" you understand what I mean, ML models simply approximate based on what they see other humans do or say. I am not calling the sunset beautiful because other people are, it is my own subjective interpretation.
https://en.wikipedia.org/wiki/Braitenberg_vehicle
Samuel Johnson: “Depend upon it, sir, when a man knows he is to be hanged in a fortnight, it concentrates his mind wonderfully.”
human wrong => no problem, mistakes happen
> These emergent abilities are not actually that, but a result of humans' poor understanding of cognition and communication.
My counterargument is that humans hallucinate too, and often. As just one small example, eyewitness testimony is stupifyingily unreliable. Neurological research and even basic behavioral research shows our brains act as bullshit machines fabricating satisfying narratives constantly. Not to even get into the fact that the word hallucination still has a non-AI meaning, and that dreams exist. As I put see it GPT models simply hallucinate more often and, more noticably, in a different manner than humans. The hallucination frequency need not reach zero, only human equivalent or better , and GPT-4 is already much better than GPT-3.
I agree with everything else you said fully. "True" reasoning machines or not, society will be catastrophically destabilized. Amongst the chaos I expect plentiful of "normal" conventional and nuclear war to go on.
Hallucinations are where you hear, see, smell, taste or feel things that appear to be real but only exist in your mind. Get medical help if you or someone else have hallucinations.
Why do we say LLMs hallucinate and why do people keep parroting the same thing "oh humans hallucinate", do we really hallucinate all the time? I can think of only one time a healthy adult hallucinates and it's not sitting around drinking a cup of tea.
It’s not about humans intentionally lying or truly hallucinating like on drugs. It’s about confidently thinking they are right about something which turns out not to be true.
I mean, if I say it like this is like humanity’s core business.
ChatGPT models semantic relationships in the data. That's what your smart buddy means by "understanding". That is a high dimensional model of the data set which infers abstract semantic relationships / concepts. But he would not claim that those semantic relationships are exactly the same as any human interpretation of the data (which you refer to as the "real" response).
Language models also have limited reasoning abilities. They are capable of misunderstanding as much as understanding.
Now, for all the disagreements in this thread I have not seen anyone claim it knows and reasons why things are the way they are, I don't disagree that it knows what to do for different inputs but it is not processing the input and making a reasonable decision more like mapping input with most approximate output. Is thay incorrect of me to say?
Yes it is actually modelling the meaning and context around it. ChatGPT is a ~100 layer deep neural network that was trained specifically to solve "natural language understanding tasks". The term "understanding" is an important term in NLP research along with the concept of "semantic similarity".
I believe it is more powerful and ultimately (eventually) more dangerous than you fear it is.
> Language models also have limited reasoning abilities.
They have no reasoning abilities at all.
Posted by Jason Wei and Denny Zhou, Research Scientists, Google Research, Brain team
https://ai.googleblog.com/2022/05/language-models-perform-re...
The paper's claim is essentially "these metrics which appear to demonstrate emergence can be replaced by other metrics that also represent model behavior, but that do not have scale discontinuities, so emergence isn't a real phenomenon".
But an equally valid interpretation would be "none of these metrics actually capture the properties we are truly interested in". Which, given the complexity of what we are dealing with here, seems entirely reasonable. It's not like we suddenly learned how to accurately quantify performance at language tasks. The whole reason LLMs are so great in the first place is because traditional 'mechanical' language models suck so bad.
That might mean that behaviors we consider emergent are the consequence of a process that scales continuously with model size.
i.e.: there may exist a bijection between, say, a step-function `can_do_arithmetic(size)` and a smooth, continuous function `arithmetic_skill_metric(size)`
If we can use continuous metrics to back out the step-function equivalents, that'll help us predict when and how to get particular abilities to "emerge."
For example: If a change results in a steeper slope on the continuous metric, we can predict it would cause the associated capability to emerge at relatively smaller model sizes.
Or it might mean that the metrics used are worthless for describing high-level model behavior. That's my whole point. Emergent behavior was observed, which is why we want those metrics so we can try to understand what is going on. But just because we have metrics that exhibit discontinuities at scales where humans observe emergence, doesn't mean that those metrics really represent the hard-to-define behavioral changes we have observed.
Using the example of 4 digit multiplication in the source paper: The researcher wants to know if the model has developed the ability to multiply two four-digit integers, so they generate a battery of such problems, e.g. "what is 4363*1285? output only your answer." The metric is what percentage of the problems the LLM answers correctly.
This is pretty much the same way a human observer would identify the same emergent behavior, and also how we assess it in other humans. It's not some contrived metric that's detached from the emergent ability in question.
The takeaway from this paper is actually that this says more about the human observers than the LLMs themselves. Humans observe "emergence" using some loose metrics themselves. Whether we formalise the metrics in a quantitative way or we stay in the realm of humnan intuition is not important; we still use some criteria to analyse things, in the latter case these are more clear and amenable to critique, in the former case they are obscure and can evade critique by moving goalposts and ground easily.
In the end of the day, either we talk about quantification or just qualitative observations, it is the same phenomenon we observe with the same qualities. The problem is that human intuition uses a lot of discontinuous metrics; we judge if an LLM passes or fails what we ask for it, but it is harder to judge the underlying tokenisation process as itself. For this reason, and considering the findings of this paper, the observations and claims of emergence in LLMs carry less value now, imo.
Whatever "insight" and "understanding" actually mean, there is no denying that they are immensely useful, which is why we want AIs that can replicate them.
When trying to understand complex systems that don't yield to quantitative analysis in an obvious way, the starting point should be to assume that the intuitive evaluation is (roughly) correct, until proven otherwise. Trying to cast this intuition into a simple metric and then using that metric (or other simple metrics) to demonstrate that the intuition is wrong is circular reasoning.
It's also the number one most common cognitive bias. Humans are especially prone to reification - the confusion that the construction of a measure equates to an objective reality.
Humans are often launder subjectivity through the creation of metrics without 1) knowing they've done so, and 2) become emotional when accused of having done so.
I think the paper is correct that there are no "emergent abilities", i.e. abilities that might suddenly appear when scale of the model is increased. And though it might not be accurate, but the paper did make some effort to formalize and I think it is a good attempt to kind of prove the point.
However as we recognize, there are still some weird discontinuities in which at one point the model is useless and suddenly it becomes very useful. This "discontinuity" IMHO is probably just perceptional, but the underlying metric is continuous.
The problem, of course, is that people's intuition is particularly awful for this sort of thing. We have a very strong tendency to anthropomorphize everything, and that illusion can be quite overpowering.
If you can't quantify that, then it is just a hunch. What exactly do you think "observers already know"?
If everything that is real could be quantified, we wouldn't need AI. Traditional computing is already absolutely phenomenal at dealing with quantifiable systems. The whole point of wanting "artificial intelligence" is because we don't know how to quantify the high-level properties of speech, thought, intellect, and consciousness. And not for lack of trying.
The quest for more generalized models boils down to studying emergent behavior because we could never prescriptively define all the parameters/behaviors/requirements necessary for such a complex outcome. We don’t even understand how the relatively easily observed interactions between neurons in our own brains result in emergent intelligence.
What’s so impressive about LLMs is they understand the semantics of some concepts so well that they can consistently produce higher quality outputs for tasks like “explain this complicated concept with a nursery song from the perspective of a pirate” than humans could, with approximately no instances of that task in their training data. That is emergent behavior and it’s a pretty big deal.
I am skeptical, although not completely unconvinced, that LLMs like GPT are going to produce truly emergent phenomena, such as true first-principles logical reasoning. The limitations of the underlying transformer architecture itself are, in my opinion, the problem. The first problem is that the embedding space of the transformer needs to grow much, much larger, and it's already huge. This matters because you need to model the order of neurons in the brain. The second problem is that you're never going to train an LLM (as they're designed today) that is going to produce a truly good 'emergent-phenomena' answer without multiple network traversals. This is because the human mind constantly and autonomously refines its thoughts.
Perhaps a good counter-argument is that emergent phenomena are fundamentally a space-time domain concept.
I am aware that things like Conway's Game of Life are a fantastic counterargument to my "the transformer architecture doesn't support it" argument. But I agree that the definition of "emergent behavior" when it comes to machine learning is too easily corrupted to be novel rather than rigorous.
Why? Did we just lose all interest in understanding things? Wasn't that the whole point in the first place?
Somehow, people are throwing up their hands and giving up at understanding the thing; yet at the same time they are acting like the thing will magically evolve into their wildest dreams!
The most fundamental feature of LLMs is that they cannot be literal. They can only infer, never define. Why is it that the people studying LLMs think they have to emulate that trait? It's like they are only allowing themselves to look at it as a mysterious black box: to infer its behavior from its results. Did they forget that they are the ones who wrote the damn thing?
You should consider that despite considerable effort, human brains are themselves black boxes. And you know less about your own knowledge than you think. I do not know where I learned that Timbuktu is both a placeholder name and real place, though I could go find evidence for both. I don’t have to expend any effort to distinguish the sounds of different words, I don’t know why two things can both taste “good”but in completely different ways. Nobody ever taught me that newly met acquaintances tended to not care to discuss current events in the business world, I just figured it out based on a collection of experiences whose individual instances I can’t even remember. Even the best neuroscientist could not tell you why neurons interacting in a certain way makes it so I can both drive a car and sing, or why one person’s brain seems to better at some or generalized tasks than another’s.
And, well, deep learning overturned the paradigm of handcrafting AI systems by automating the process of “have the model produce this output from this input” without requiring humans to define the “how” beyond tuning the shape of the model, which was itself a hugely important innovation in reducing the human time required to build an AI system. But it’s not just faster to make these models, it’s so ridiculously better at making AI models for things like “is there a dog in this picture” that nobody would even consider doing those things without deep learning.
You actually can fiddle with DNNs to get an idea of how they work similar to what we do with brains and CAT scans, you have it do some stuff with commonalities and you figure out which common parts get activated. This is easy to do with convolutional layers as they very commonly learn for themselves how to perform edge detection.
Anyway, long story short, fully explainable AI utterly sucks ass at many tasks that are like a walk on the park for blackbox AI. And we cannot explain our own intelligence and knowledge except in terms of emergent phenomena, nor can we give the full provenance of some factoid or skill we have on demand (just like an LLM cannot tell you where it learned something) in many cases[0], so it seems reasonable that we’d be in the same situation with AI.
[0] The main difference is that we have memory of the various discrete experiences of our lives (which we can associate with some knowledge or skill), and there is no binary separation between “learning mode” and “doing mode” or “active memory” and “long term memory” for us like with AI. We can definitely associate some knowledge with a particular event, but this seems like it could be a false ontological representation of our knowledge because if the knowledge and event were unimportant (like what you had for breakfast on a particular day) we’d forget both of them; it’s actually all the subsequent cases in which the knowledge and memory of the event came in handy that contribute to us being able to explain it.
When we are dealing with a system that is made of logic and reason, we can use logic and reason to construct an understanding of that system. This is the explicit approach to understanding.
When we are dealing with a mysterious black box, we must take the implicit approach: using testing and inference to construct a model, we can construct an explicit understanding of that model. This is effectively the same process, but one step removed: we understand our model, not the system it applies to. That model may be incomplete and/or misaligned.
The human mind is a mysterious black box. We have made a lot of progress modeling that system, but our models are not complete or perfectly aligned.
While our study of the human mind is limited to the implicit approach, the human mind itself is capable of both implicit and explicit understanding.
So far, no software has been able to emulate that feature. Every tech that exists today uses only one of the two approaches.
> Anyway, long story short, fully explainable AI utterly sucks ass at many tasks that are like a walk on the park for blackbox AI.
What you have called "fully explainable AI" is any tech that uses the explicit approach. Because everything is explicitly defined, there is a clear place for logic to exist as part of the system. Because everything is explicitly defined, there is no room for ambiguity in the system.
What you have called "blackbox AI" is any tech that uses the implicit approach. Because nothing is literally defined, ambiguity can exist in the system. Because nothing is explicitly defined, there is no clear place for logic to exist in the system.
But is it really a black box? The program itself is explicitly defined! We should be able to use the explicit approach to understand it, just like we do any other software.
Go tell that to the researchers who are working hard on studying emergent properties.
For example if it was possible to train a Hidden Markov Model with billions of hidden states on a trillion tokens, you could more literally look and see what was going on.
Other than not being able to scale HMMs to this kind of scale, is there any good reason to believe they would not perform equally well but without the magic?
The hidden state in HMM would be (num_tokens ^ context size), so something like 60000^2000.
Anything longer than 4 tokens/words of context and you bump into 30k ^ 4-10 -> cross a billion/trillion state boundary and you lose any chance of using markov chains.
Also - but here I may be wrong - there is no way to "train" markov chains to do generalisations - that is, if a given sentence didn't appear on the internet, it won't be available as a state for the chain. In this aspect they are more similar to a database than anything else.
I and I assumed a lot of people, already thought performance was a function on model size (# of parameters). Is this not what the prevailing thought is for DNN performance?
Agreed with the other posters that this title is misleading.
I guess the disagreement has been in whether this function is "continuous" or not.
I do not think the title is misleading, considering the article answers to quite specific claims in other articles. I agree it sounds misleading if you do not put it in that context.
Like, there's supposed to be nothing academic researchers like more than re-using the same word in a title, or making it into a clever pun or quip -- it's like the Dad Jokiest subfield -- but instead we just get a title that implies one common argument that people make, and actually delivers an unpredictable different argument that seems plausible but not necessarily interesting.
Text contains more data than language. Large Language Models work implicitly: they are not limited to finding language-specific patterns in the text that they model.
Humans look at LLMs through a lens of expectation. Any time we find a feature we did not expect, we categorize it after-the-fact. That's our biggest mistake: LLMs are not made of categories!
Anything measured by "winning" has this property. Small changes near the "winning" threshold result in large changes in wins. This is well known in sports.
Is there more to this issue than this amplification effect?
Two softball players. One hits the ball 230 feet on average. The other hits the ball 210 feet on average. The homerun fence is at 220 feet.
One is considered a GREAT homerun hitter. The other is considered a poor one.
The measure is non-linear.
That takes nothing away from the GREAT homerun hitter.
It's the difference between siri not understanding you 1 word out of 10 (very accurate!), and it basically just understanding you. It's a continuous accuracy function and a discontinuous usefulness function.
Two softball players. One can hit the ball an average of 230 feet, 40% of the at bats. The other can hit the ball an average of 210 feet, 40% of the at bats. The homerun wall is 220 feet.
One is a GREAT homerun hitter. The other has a poor batting average.
The issue is that the success measure is non-linear.
However, many natural phenomena that are fully explainable from first principles show this property, giving rise to sigmoidal "S-curves", as shown in Figure 1.
