> That is, we should ascribe intentions to a system if and only if it helps to predict and explain the behaviour of the system. Whether it really has intentions beyond this is not a question I am attempting to answer (and I think that it is probably not determinate in any case).
And yet, I think there's room to argue that LLMs (as currently implemented) cannot have intentions. Not because of their capabilities or behaviors, but because we know how they work (mechanically at least) and it is incompatible with useful definitions of the word "intent."
Primarily, they are pure functions that accept a sequence of tokens and return the next token. The model itself is stateless, and it doesn't seem right to me to ascribe "intent" to a stateless function. Even if the function is capable of modeling certain aspects of chess.
Otherwise, we are in the somewhat absurd position of needing to argue that all mathematical functions "intend" to yield their result. Maybe you could go there, but it seems to be torturing language a bit, just like people who advocate definitions of "consciousness" wherein even rocks are a "little bit conscious."
Nevertheless the human would be acting intentionally (for in-distribution impulse patterns) for the brief period of simulation.
Fine-tuning and RLHF seem to impart more intentionality to the pure stateless models, as well; it's not the case that all texts the LLMs were pretrained on were outputs of helpful AI assistants avoiding harmful outputs but the resulting models do in fact behave like AI assistants unless prompted with more out-of-distribution context or intentional jailbreaks.
What word would you use instead of intention for the property that RLHF and fine-tuning create? It's goal oriented behavior with some world-modeling ability in achieving the goal even if it's far from robust. If the LLM is only simulating an AI assistant it seems to me that a larger fraction of its total function is dedicated to simulating the intention of that assistant. Creating a simulator of intentional behavior is, I think, entirely novel.
“Humans too would be stateless if we hacked their brain in a way that made them stateless” that would also make them non-human though, and unlikely to be able to exhibit meaningful high-level cognitive abilities, so I don't really understand what your point is…
I highly doubt that would ever be possible in practice, as our inputs are much too complex. But I want to point out, you're basically saying here "humans are nearly stateless if we take a snapshot of their state and simulate that state..."
Statefulness can be modelled statelessly so "statelessness" is not a sufficient reason to dismiss intentionality. The only question is whether the relationship between the inputs of the function and its outputs correspond to what we might call "intent", which the cited definition attempts to outline. Obviously it's only a high-level view that leaves many details unanswered.
> Otherwise, we are in the somewhat absurd position of needing to argue that all mathematical functions "intend" to yield their result.
Not all mathematical functions have a continuity of internal identity and self-reference as seems to be the case with LLMs.
Clearly the system we're understanding as "intentional" has state; we can engage in multi-round interactions. It doesn't matter that we can separate the mutable state from the function that updates that state.
I have two arguments against. One, you could argue that state is transferred between the layers. It may be inelegant for each chain of state transitions to be the same length, but it seems to work. Two, it may not have "states", but if the end result is the same, does it matter?
At the least it's made me think for a moment about `stateless` and its meaning
The more significant application of storage will be long-term storage wrapped in a read-modify-write loop.
So the problem with this guy's definition of intentionality is, first, that it's a redefinition. If you're interested in whether a machine can possess intentionality, you won't find the answer in interpretivism, because that's no longer a meaningful question.
Intentionality presupposes telos, so if you assume a metaphysical position that rules out telos, such as materialism, then, by definition, you cannot have "aboutness", and therefore, no intentionality of any sort.
As you point out, the approach you cite can't be used in a materialist metaphysical position. That's a pretty severe problem for that definition! So Dennett's approach, or something like it, has major advantages.
Computers are mechanical gadgets that work with electricity. Humans (and other animals) die when exposed to the kinds of currents flowing through computers. Similarly, I have never seen a computer drink water (for obvious reasons). If properties are reduced to behavioral outcomes then maybe someone can explain to me why computers are so averse to water.
In fact, anything really.
But really: https://en.wikipedia.org/wiki/Magnetite#Human_brain
Not because of their capabilities or behaviors, but because we know how they work (mechanically at least) and it is incompatible with useful definitions of the word "intent."
I'd add though that inference is clearly fixed but there is some more subtlety about training. Gradient descent clearly doesnt have intelligence, intent (in the sense meant), consciousness either, but it's not stateless like inference and you could argue has a rudimentary "intent" in minimizing loss.
When you say upsetting things to bing chat, you'll find the conversation prematurely end.
You can cry all you want about how bing isn't really upset. How it doesn't really have intention to end the chat but those are evidently useless defitions because the chat did end.
A definition that treats Bing as an intentful system is more accurate to what happens in reality.
Further, what happens when you give an LLM a bank of long-term storage and a read-modify-write loop around it? A sufficiently advanced "modify" function would be more than enough to give rise to intent even in the broadest understanding of the word. GPT-4 class models are could very well be advanced enough to give rise to a variety of higher-level behavior that previously we would only have ascribed to prinate-class intelligence. If anyone really wants to advance the state of the art, you should figure out the best way to train a model with a read-modify-write loop, how to index into the storage, how to store "results", and so on.
I firmly believe that in the next 100 years we will have AI independence movements, with a high possiblity of outright war, terrorism, etc. (Maybe AI will be better than humans at avoiding the use of violence.) In 20 years this trajectory will be supremely obvious.
Edited-- disagree about the timeline, ramifications, acts of war, or whatever, I really don't care. Seriously though, something like a read-modify-write loop is key. You can only build so complicated a function with only combinational logic gates. But just 64 bits of storage can produce sequences going beyond the life of the universe. Imagine an LLM paired with gigabytes+ of working memory/storage. It would easily be capable of moving about the virtual world with "intent".
You create a very different sort of system, for one. Saying that because doing that in just the might way could yield a system with intention, an LLM has intention is rather like saying that my refrigerator is a sandwich.
What LLMs do may happen to fit some technical definition of intentionality that has been previously explored but that definition doesn't align with the actual debate that is going on about LLMs abilities.
Yes because the debate is nonsense.
Seeing output from GPT that demonstrates intelligence, reasoning, or whatever, and saying it is not real reasoning/Intelligence etc, is like looking at a plane soar and saying that the plane is fake flying. And this isn't, for anyone who thinks it is, a nature versus artificial thing either. The origin point is entirely arbitrary.
You could just as easily move the origin to Bees and say, "oh, birds aren't really flying". You could move it to planes and say, "oh, helicopters aren't really flying." It's a very meaningless statement.
The point most people seem to miss is that internal processes are entirely irrelevant. If you have a property you are interested in and a way to test for it, then the results of that test is what is important, not whether how it works at the arbitrary origin is exactly the same as how it works at point 2. In this case, it's even worse because since we do not know the internal processes of either LLMs or humans, the argument is really " oh, how I think the origin works is different from how I think point 2 works, so it isn't really flying".
When you say upsetting things to bing chat, you'll find the conversation prematurely end.
Someone can cry all they want about how bing isn't really upset. How it doesn't really have intention to end the chat but those are evidently useless definitions because the chat did end.
A definition that treats Bing as an intentful system is more accurate to reality and real consequences. It has the predictive power that the alternative does not.
Someday someone may find themselves stabbed and killed by an LLM piloted robot because of something they said or did. Something that would predictably get someone killed by a system with "real" intent. So what, Are you going to be raised from the dead because the LLM "wasn't really upset" or "didn't really have intent" ? It obviously doesn't count right.
From now on I'll listen for the subtle popping sounds as these BBs get instantiated and de-instantiated all around me...
Of course a philosopher can object that (1) these BBs are on a substrate that's richer than they are so aren't "really" BBs and (2) they often leave traces that are available to them in later instantiations which again classical BBs can't. So maybe make up another name -- but a great way to think.
Chess move training data is much more likely to consist of examples of people trying to win than it is to consist of random legal or even illegal moves. You could argue that the intention belongs to the people providing the input and not the LLM, but that seems like a distinction without a difference.
You can play a good game of chess (or poker for that matter) with GPT.
https://twitter.com/kenshinsamurai9/status/16625105325852917...
https://arxiv.org/abs/2308.12466
There's also some work going on in the eleuther ai discord training LLMs specifically for chess to see how they shape up. They're using the pythia models. so far:
Pythia 70M, est ELO 1050
Pythia 160M, est ELO 1370
Edit: This might not be the case anymore it seems, my below point doesn't actually contradict you, seems it matters a lot how you tell the model your moves. Also saying things like "move my rightmost pawn" completely confuses them.
I do think it will be interesting as visual input and internal graphical output is integrated with text based LLMs as that should help correct their internal experience to be based closer to what we as humans experience.
If LLMs actually could reason, there is a much much wider set of applications where they would be actively used.
The term “hallucination” does us all an injustice by propagating the idea of an anthropomorphized LLMs.
Everything an LLM does is a hallucination.
You and me can make out valid patterns from invalid patterns, because we have an idea of some reality.
(Incidentally there are some very weird implications/ perspectives deriving from these 2 positions. Eg - If you had infinite data, would a LLM ever need to calculate?)
Point being - the more intimate the use with an LLM, the more its emergent properties are non-emergent.
Let's talk about "predicting the next token." Sure, that's the technical framework, but what happens within that prediction is an intricate dance of probabilities, patterns, and weighted connections that come together to form something that can assist, inform, and sometimes even entertain. There's a vast landscape of difference between a machine that predicts the next word in a sentence and a machine that can draft an entire poem, answer a complicated question, or simulate conversation in a way that can sometimes pass for human thought.
Is it reasoning in the way humans do? No. But to say that LLMs are "simply" predicting the next token is like saying a car is "simply" a collection of nuts and bolts that move in a certain way. It's true, but it's missing the whole picture. Just think about the implications! If this was as trivial as "predicting the next token," then why aren't we seeing this level of application everywhere?
As for the term "hallucination," I get it. It's a bit anthropomorphic, sure, but language always is. We use human-centric language to describe lots of things that aren't human. We say economies are "healthy" or "sick," we say a defense in football is "stalwart." Is it perfect? No. But it gives people a way to discuss and think about complex topics, including this one. And guess what, complex discussions are how progress happens!
The point about infinite data is intriguing, but let's not go off the rails here. The question isn't whether an LLM would ever "need" to calculate; it's whether the way it calculates could ever truly mimic human thought or reasoning. That's a long road we're still traveling down. But here's the kicker: just because we're not there yet doesn't mean the work that's been done is insignificant or simplistic.
Emergent properties becoming "non-emergent" the more you interact with an LLM? That's the point! The more you use these systems, the more you understand their capabilities and limitations, and the better you can leverage them for tasks that are useful, interesting, or revealing.
So, yeah, let's not box in what is one of the most dynamic, evolving fields right now with a one-liner that's as limiting as it is dismissive.
From that perspective, I think multi-intentionality also works. If I write a story about Bob, then Bob (in the story) has intentions, although he's just a figment of my imagination; and when we read characters in novels, we use the imputed intentions of the characters to understand their behavior, although we know they're fictional and don't actually exist.
So yes; on one level, I want to write an exciting story; on a second level, I'm simulating Bob in my head, who wants to execute the perfect robbery. On one level, GPT-4 wants to write a story about a smart AI; on a second level, the smart AI in GPT-4's story wants to win the chess game by moving the queen to put the king in check.
I think it's true that current generation LLMs could, in principle, have intentionality in the way described in the article. But they would have to be trained on many orders of magnitude more data than current models.
Also AutoGPT does not work. I encourage the author to play around with it and try to get it to do something useful with high success probability.