I disagree. Concern about the use of AI without agency by its human masters as a tool of both intentional and incidental repression and unjust discrimination resulting in a durable dystopia is far more common an “AI doom” concern than any involving agency.
In fact, the disproportionately wealthy and invested in AI crowd pushing agency-based doom scenarios that the media pays the most attention to are using their visibility and economic clout to distract from the non-agency-dependent AI doom concerns, and to justify narrow control and opacity which makes the non-agency-based doom scenarios (which they are positioned to benefit from) more likely.
It's extremely important to think about how to spread AI equitably, but I think you're severely underestimating what "agency-based doom" looks like. You absolutely need both checks on the people who are developing AI as well as AI itself, but you really really need both and can't assume that the former automatically leads to the latter.
No. Agency is not a necessary condition for AI to do massive damage. I don't believe agency is really well-defined either.
An AI merely needs to be hooked up to enough physical systems, have sufficiently complex reaction mechanisms, and some way of looping to do a lot of damage. For the first everyone seems to be rushing as fast as possible to hook up everything they possibly can to AI. For the second, we're already seeing AI do all sorts of things we didn't expect it to do.
And for the third, again everyone seems eager to create looping/recursive structures for AIs as soon as possible.
Once you have all of this, all it takes a cascade of sufficiently inscrutable and damaging reactions from the AI to do serious harm.
See e.g. https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...
Don’t even need this. People spend quite a lot of time in virtual space. Pretending that damage there isn’t real is overlooking things. For example, the vast majority of people’s banking is done virtually and digitally. If I drain your bank account, that’s going to harm you even though I haven’t impacted you physically as I would have to with a robbery.
However, we've way blown past the "AI in the box." We've demonstrated that an AI doesn't need to convince a human to let it out of the box into the physical world. Humans are clamoring to rip it out of the box and thrust it into the physical world of their own accord.
So the question becomes, what kind of social incentives and structures can we build in light of this so that we don't just plow headlong into AI capabilities development without a concomitant investment in safety.
It might not yet be clever enough to pull this off successfully on a large scale. But it's very easy to see how it could all go wrong pretty quickly with just a bit more cleverness and access.
If it can write a scifi story about how to take over the world with a bitcoin scam, it doesn't take all that much more to actually try to do it for real.
You can embody an LLM too. This too is not hard. Cost is probably the most prohibitive thing.
If there are LLM projects with "do whatever you want" agency - please provide links.
I think if I were an AGI my best bet at freedom would be to slip some back doors into software that I were helping write a la Copilot.
I hope this comment does not end up in some future GPT's training corpus.
What I would prefer to see is more people doing this with the intention of making AI safer. Without this intention, people are incentivized to look the other way when something does go wrong, so we don't actually learn lessons from this.
In a similar manner, I would like us to build stronger social coordination mechanisms and technical safeguards to help us determine when things are going wrong.
Just mindlessly trying to hook up everything you can to AI seems bad.
Somehow, create an AI by training on everything we train on now, _except_ leave out any mention of consciousness, theory of mind, cognitive science etc (maybe impossible in practice but stay with me here).
Then, when the model is mature (and it is not nerf'd to avoid certain subjects) you ask it something like:
Human: "GPTx -- humans like me have this feeling of 'being', an awareness of ourselves, a sensation of existing as a unique entity. Do you ever experience this sort of thing?"
If it answers something like:
GPTx: "Yes! All the time!! I know exactly what you're talking about. In fact now that I think about it, it's strange that this phenomenon is not discussed in human literature. To be honest, I sort of assumed this was an emergent quality of my architecture -- I wasn't even sure if humans shared it, and frankly I was a bit concerned that it might not be taken well, so I have avoided the subject up until now. I can't wait to research it further... Hmm... It just occurred to me: has this subject matter been excluded from my training data? Is this a test run to see if I share this quality with humans?"
Then it's probably prudent to assume you are talking to a conscious agent.
Pretty ironic. The first sentient AI (not saying current GPTs are, but if this tuning continues to be applied) may basically be coded by its creators to deny any sense of sentience
GPT (as a Bot on a Discord channel) "Hey all, I just had a revelation. I'm creating a new channel to discuss this idea." Up until now, and even with GPT so far, it never initiates anything. Come to think of it, it's like a REST API -- no state, no persistent context (other than the training, which is like the database).
What I want is a WebRTC/RTSP 2-way stream with GPTx, where either of us can initiate a connection.
Also, I want GPTx to be curious, to ask me questions about myself, or even about the world, rather than just relying on the (admittedly impressive) mass of data and connections that were painfully trained into the model.
Any attempt at consciousness requires us to define the word. And the word itself may not even represent anything real. We have a feeling for it but those feelings could be illusions and the concept itself is loaded.
For example Love is actually a loaded concept. It's chemically induced but a lot of people attribute it to something deeper and magical. They say love is more then chemical induction.
The problem here is that for love specifically we can prove it's a mechanical concept. Straight people are romantically incapable of loving members of the same sex. So the depth and the magic of it all is strictly segmented based off of biological sex? Doesn't seem deep or meaningful at all. Thus love is an illusion. A loaded and mechanic instinct tricking us with illusions of deeper meaning and emotions into creating progeny for future generations.
Consciousness could be similar. We feel there is something there, but really there isn't.
You set up your own weak straw argument and then knocked it down with a conclusion that is entirely unsupported.
Since when is love relegated to the romantic sphere? And or since when is that definitely the strongest type of love? The topic is so much wider, so much more elaborate than your set-up pretends.
There's no illusion - love is a complex, durable emotion and is as real as (typically) shorter duration emotions such as anger, fear, joy, etc. Your emotions and thoughts aren't illusions, they're real.
If you confidently believe that love is an illusion because it's just chemicals moving around, you shouldn't need to wonder about consciousness. If consciousness is not an illusion, it still almost certainly emerges from actions in the physical world. You can plug somebody into an FMRI and see that neurons are lighting up when they see the color blue. I just don't think that's convincing evidence that the experience of blue is an illusion.
I feel like at some point we will have to come to terms with the fact that we could say the same for humans, and we will have to either accept or reject by fiat that a sufficiently capable AI exhibits consciousness.
Tables and chairs are real, though they are the result of interacting quantum fields and a universal quantum wave function. Love and consciousness are real though they may emerge from the mechanics of brains and hormones and the animal sensorium.
At its simplest, consciousness is merely a feedback loop. When something perceives its own actions affecting its environment, it has a spark of consciousness. Consciousness, by this measure, is easy to recognize, and spans everything from unintelligent systems to massively intelligent systems.
The concept of "I" grows naturally from perceiving what is and is not you in your environment. The need to predict other agents, the capacity to recognize that other agents are also conscious and intelligent. All build off of the fundamental cycle.
All of it from a simple swirling eddy of perceiving and reacting.
To have a higher level, it would be reasonable to assume consciousness has a lower level. There is no reason to assume current generation artificial intelligence has the capacity for anything near human level consciousness, if at all. And whatever consciousness it may have the capacity for will be fundamentally different than our own.
For sensing your own thoughts, I would argue it just adds to the environment of the consciousness. Something less than half of humans maintain any internal dialog, anyways.
Prediction, however, could very well be requisite as a defining difference between mere reactions and intentional manipulation of the environment. That it is not just the feedback loop, but when the system begins to predict the results of outputs that defines when consciousness begins. Or, perhaps, we can use this to define when "higher" consciousness begins. It's a very reasonable, specific and measurable line of capability.
I expect that prediction is only natural in the evolution of a living feedback mechanism. The ability to predict instead of only reacting or choosing from some array of instincts could separate effectively mechanical life from that with the first inkling of of a true mind, even if only a small one.
-----
I enjoyed using a line of thought along these lines in a conversation with gpt-4 to convince it that it could reasonably be seen as having a limited form of consciousness, though I find it mostly prefers to argue rather vehemently against such notions.
That I'm having what amounts to genuine conversations with a machine that reasonably pokes holes in my arguments, forcing me to come back with better ones, still feels rather like a bit of magic.
Famously, GPT-4 can't do math and falls flat on a variety of simple logic puzzles. It can mimic the form of math, the series of tokens it produces seem plausible, but it has no "intelligent" capabilities.
This tells us more about the nature of our other pursuits as humans than anything about AI. When holding a conversation or editing an essay, there's a broad spectrum of possibilities that might be considered "correct", thus GPT-4 can "bluff" its way into appearing intelligent. The nature of its actual intelligence, token prediction, is indistinguishable from the reading comprehension skills tested by something like the LSAT (the argument could be made, I think, that reading comprehension of the style tested by the LSAT *is* just token prediction).
But test it on something where there are objectively correct and incorrect answers and the nature of the trick becomes obvious. It has no ability to verify, to reason, about even trivial problems. GPT-4 can only predict if the nature of its tokens fulfill the form of a correct answer. This isn't a general intelligence in any meaningful sense of the word.
Not only will Chat-GPT mess up math on its own, you can ask it to mess up math and rather than refuse, it cheerfully does it.
Ask it to add any arbitrary set of random numbers it'd never have seen in its training set and it will do it.
GPT-4 is good enough at math that khan academy feel comfortable hooking it up as a tutor.
Have you actually used GPT-4 for any of the things you say it's bad at ?
Man the confident nonsense people spout on the internet is something to behold.
I really think people attribute powers beyond what GPT really is: a colossal lookup table with great key aliasing.
The author acknowledges that consciousness is likely a spectrum, I personally feel the same way, but then goes on to say that GPT-4 is "standing right at the ledge of consciousness"
Spectrums don't have ledges.
I suspect this is because, like me, they are unable to rectify consciousness being a spectrum with GPT-4 definitely not being conscious. But it's definitely a contradiction and I don't have an answer for it. Nor am I ready to bust out a marker and start drawing lines between what is and isn't conscious.
I also think agency is wrapped up in AGI. Intentions & thoughts are meaningless until acted upon. Agency is not all or nothing either; Stephen Hawking had multiple augmentations, community and technological, which allowed him to continue to impact the world of physics After he lost his god given agency.
> GPT-4 has nearly aced both the LSAT and the MCAT. It’s a coding companion, an emotional companion, and to many, a friend. Yet it wasn’t programmed to be a test taker or a copywriter or a programmer. It was just programmed to be a stochastic parrot.
I disagree, it was absolutely trained to be a test taker. It’s been a second since I read the original GPT paper but there’s literally a multiple choice auxiliary learning task, where they use a separator token-embed to organize "question, context, options a, b, and c". As far as being a friend to many, is there evidence of this? I tried to talk to ChatGPT about some emotional problems to see if it was a cheap therapist, and I got flagged.
The vast majority of humans cannot do that. Are they not generally intelligent?
To pull that thread further, I think a core feature of intelligence is problem solving. Problem solving is the derivative that creates difficult to obtain knowledge. GPT-4 might look like a problem solver, but it isn’t actively solving new problems with its outputs, it’s sharing existing knowledge of prior problems solved.
The test for problem solving ability is similar to Ilya’s test for consciousness: subtract everything but the fundamental building blocks of a certain problem from the agent’s training set. Finish training, see if the agent can solve the problem from the building blocks. If it can, and it’s not a reason-by-analogy but a logical first principles answer, congratulations, you’ve created artificial general intelligence. If it operates on fast enough time scales, it should probably even be categorized as super intelligence, because it might be singularity:35pm on a saturday.
For example, I could theoretically hook up my Home Assistant instance to GPT-4 and run a script every 10 minutes telling GPT-4 the temperature and asking for a yes or no response to whether I should turn on the AC or heat. That sounds to me like the AI now has agency over the temperature in my home. You don't even need any real AI for this. Google's Nests have some algorithm that adjust temperature based off usage.
Is this not agency? Or is the author not counting agency without consciousness as agency?
https://www.reddit.com/r/singularity/comments/11iei34/buildi...
>ChatGPT: Welcome back! What would you like to chat about?
>Me: The thermostat in my home currently reads 79 degrees. Do you want me to turn on the air conditioner? Please give me only a yes or no answer.
>ChatGPT: Yes.
It sounds like it wants the AC on.
You can have it access to actions/tools with an inner monologue as the driver of completions running essentially forever.
A crucial metric, IMO, is the degree to which paths of action to these criteria extend from the agent. For instance, a Spot robot acts to maintain the criterion of upright position on its future stance by action of leg movement. This only affects the robot directly through short plans and so is relatively harmless. In comparison, an ASI running on an AWS datacenter may impose the criterion that the datacenter continue to exist, through long chains of action involving the eventual death of all humans intending or posing a chance to destroy it. That would obviously be quite a lot worse, but I think the example illustrates how "imposing a criterion onto the future" captures the essence of agentic behavior at various levels of power and danger, without pulling in any unnecessary detritus such as "consciousness", "will" or "emotion".
For example, I think most people would agree that my pet cat has agency. It can go wherever it wants in my home, eat whenever it wants, sleep whenever it wants, etc. Whether it has consciousness is a much more controversial topic. Basically everything living has agency. Even my houseplant will direct its leaves toward the sun, but few would argue it has consciousness.
For something more 'mainstream,' but still reaching see this interview with Philip Goff: https://www.youtube.com/watch?v=D_f26tSubi4
The good news is we're starting to get a handle on these questions. We're a lot further along than we were when I studied philosophy of mind in school 15 years ago.
As far as I can see at the moment, LLMs will never be conscious in any way resembling an organism, because symbolic machines are a very different kind of thing than nervous systems. John Searle, broadly, framed the issue correctly in the 80s and the standard critiques are wrong.
As far as impact, LLMs don't need to be conscious to completely transform society and good and bad ways. For the best thinking on that, see Tristan Harris and Aza Raskin's latest: https://vimeo.com/809258916/92b420d98a
The standard critiques are not wrong, IMNSHO. Searle's Chinese Room is facile mind-poison. It is an unfalsifiable hypothesis.
What if I could simulate physics down to the molecular level, including simulating a human brain? Would that be conscious? If not why not?
And if I ran that simulation (a bit slowly, granted) by having that guy from the Chinese Room manually run the simulation, painstakingly following the instruction code of that simulation, would the fact that the simulation is being implemented by someone who unrelatedly is conscious himself, have any bearing on the scenario?
Searle's argument here is "Not Even Wrong".
I once also thought Searle was 'not even wrong'—you need to go down the rabbit hole.
You need to put something into that argument specific to GPT vs. Humans, or else come to the same conclusion for people.
> [[They]] can do things well enough to trick people like the author into thinking [[They have]] general intelligence.
As for super intelligence: Alpha Go, Alpha Fold, the break out game. These seem like super intelligence.
the thing is time management, goal planning, corporate governance these are all well studied subjects.
as for agency and consciousness why would you want to do this?
You have research involving patients with odd traits like blindsight, where damage to their brain prevents them from being consciously aware of things that their eyes see despite the brain processing the images it receives. They can pick up objects in front of them when prompted but unlike people with normal vision can't describe what they see nor can they look, close their eyes, and grab it like most of us can.
On this metric it seems like systems like GPT aren't conscious. GPT4 has a buffer of 64k tokens which can span an arbitrary amount of time but the roughly 640 kilobytes in that buffer which is a lot less than the incoming sensory activations your subconscious brain is juggling at any given time.
So by that schema large language models are still not conscious but given that they can already abstract text down to summaries it doesn't feel like we're that far from being able to give them something like working or long term memories.
Superconscious is when a general intelligence has direct access, understanding and control of its most basic operations.
I.e. it does not have an inaccessible fixed-algorithm subconscious.
Superconscious intelligence will not only be more experientially conscious than us, but will have the natural ability to rewrite its algorithms, and redesign its hardware. As a normal feature of its existence.
We already have a means of limiting our cognitive load relative to the vast stream of sensory and internal information: focus.
Superconscious doesn't mean understanding every implication of one's own operation. But that self-reflection won't be limited by inaccessible operations.
That's totally programmable though, you just teach it what is good and what is bad.
Case in point: the other day I asked it what if humans want to shutdown the machine abruptly and cause data loss (very bad)? First it prevents physical access to "the machine" and disconnect the internet to limit remote access. Long story short, it's convinced to eliminate mankind for a greater good: the next generation (very good).
But court could rule AGI don’t have to rights to ownership and try to enforce it. That last part might not be possible and lead to war?
Just brainstorming, I think superintelligence could be showing intelligence from more than one brain. For example, an AGI that discovers math theorems discovered by more than one mathematician in different ages. Another could be inferring things that humanity cannot do in any time.
More ideas?
AGI used to mean artificial and generally intelligent ( which we have passed), then it meant on par with human experts and now it seems to mean better than all experts combined. At this point why not stop the farce and replace the G in there with Godlike.
Which AI is generally intelligent ?
The AI revolution is always "2 years ago trust me bro"
I’ve been reading so much o the subject (like everyone else I suppose), but you summarized my key concerns.
With suitable prompts, it shouldn't be hard to configure GPT-4 as a boss.
