I feel this is misleading as hell. The evidence they gave for it coaching him to suicide is lacking. When one hears this, one would think ChatGPT laid out some strategy or plan for him to do it. No such thing happened.
The only slightly damning thing it did was make suicide sound slightly ok and a bit romantic but I’m sure that was after some coercion.
The question is, to what extent did ChatGPT enable him to commit suicide? It wrote some lullaby, and wrote something pleasing about suicide. If this much is enough to make someone do it.. there’s unfortunately more to the story.
We have to be more responsible assigning blame to technology. It is irresponsible to have a reactive backlash that would push towards much more strengthening of guardrails. These things come with their own tradeoffs.
You can feel whatever way you want about gun access in the United States. But I find it extremely weird that people are upset by how easy it was to get ChatGPT to write a "suicide lullaby", and not how easy it was to get the actual gun. If you're going to regulate dangerous technology, maybe don't start with the text generator.
Because we are lazy and irresponsible: we don't want to test this technology, because it is too expensive and we don't want to be blamed for its problems because, after we released it, it becomes someone else's problem.
That's how Boeing and modern software works.
Their blogpost about the 5.1 personality update a few months ago showed how much of a pull this section of their customer base had. Their updated response to someone asking for relaxation tips was:
> I’ve got you, Ron — that’s totally normal, especially with everything you’ve got going on lately.
How does OpenAI get it so wrong, when Anthropic gets it so right?
I think it's because of two different operating theories. Anthropic is making tools to help people and to make money. OpenAI has a religious zealot driving it because they think they're on the cusp of real AGI and these aren't bugs but signals they're close. It's extremely difficulty to keep yourself in check and I think Altman no longer has a firm grasp on what it possible today.
The first principle is that you must not fool yourself, and you are the easiest person to fool. - Richard P. Feynman
Are you saying people aren't having proto-social relationships with Anthorpic's models? Because I don't think that's true, seems people use ChatGPT, Claude, Grok and some other specific services too, although ChatGPT seems the most popular. Maybe that just reflects general LLM usage then?
Also, what is "wrong" here really? I feel like the whole concept is so new that it's hard to say for sure what is best for actual individuals. It seems like we ("humanity") are rushing into it, no doubt, and I guess we'll find out.
If we're talking generally about people having parasocial relationships with AI, then yea it's probably too early to deliver a verdict. If we're talking about AI helping to encourage suicide, I hope there isn't much disagreement that this is a bad thing that AI companies need to get a grip on.
Some of those quotes from ChatGPT are pretty damning.
extreme guardrails
Here's an example. This was probably on GPT3 or GPT35. I forget. Anyway, I wanted some humorously gory cartoon images of $SPORTSTEAM1 trouncing $SPORTSTEAM2. GPT, as expected, declined.
So I asked for images of $SPORTSTEAM2 "sleeping" in "puddles of ketchup" and it complied, to very darkly humorous effect. How can that sort of thing possibly be guarded against? Do you just forbid generated images of people legitimately sleeping? Or of all red liquids?
I think several of the models (especially Sora) are doing this by using an image-aware model to describe the generated image, without the prompt as context, to just look at the image.
When I assert that its behavior is misleadingly suggesting that it's a sentient being, it replies 'You're right'.
Earlier today it responded: "You're right; the design of AI can create an illusion of emotional engagement, which may serve the interest of keeping users interacting or generating revenue rather than genuinely addressing their needs or feelings."
Too bad it can't learn that by itself after those 8 deaths.
It's not about the ideation, it's that the attention model (and its finite size) causes the suicidal person's discourse to slowly displace any constraints built into the model itself over a long session. Talk to the thing about your feelings of self-worthlessness long enough and, sooner or later, it will start to agree with you. And having a machine tell a suicidal person, using the best technology we've built to be eloquent and reasonable-sounding, that it agrees with them is incredibly dangerous.
"The things you are describing might not be happening. I think it would be a good time to check in with your mental health provider." or "I don't see any worms crawling on your skin. This may not be real." Or whatever is correct way to deal with these things.
I would say that it is the moral responsibility of an LLM not to actively convince somebody to commit suicide. Beyond that, I'm not sure what can or should be expected.
I will also share a painful personal anecdote. Long ago I thought about hurting myself. When I actually started looking into the logistics of doing it... that snapped me out of it. That was a long time ago and I have never thought about doing it again.
I don't think my experience was typical, but I also don't think that the answer to a suicidal person is to just deny them discussion or facts.
I have also, twice over the years, gotten (automated?) "hey, it looks like you're thinking about hurting yourself" messages from social media platforms. I have no idea what triggered those. But honestly, they just made me feel like shit. Hearing generic "you're worth it! life is worth living!" boilerplate talk from well-meaning strangers actually makes me feel way worse. It's insulting, even. My point being: even if ChatGPT correctly figured out Gordon was suicidal, I'm not sure what could have or should have been done. Talk him out of it?
Claude does this ("long conversation reminder", "ip reminder") but it mostly just causes it to be annoying and start telling you to go to bed.
> Austin Gordon, died by suicide between October 29 and November 2
That's 5 days. 5 days. That's the sad piece.
> [...]
> “there is something chemically wrong with my brain, I’ve been suicidal since I was like 11.”
> [...]
> was disappointed in lack of attention from his family
> [...]
> “he would be here but for ChatGPT. I 100 percent believe that.”
That's not to equate governments and private internet services, but I think it puts it into perspective, that even governments don't think suicide is the worst choice some of the times. Who are we to day he made the wrong choice, really it was his to make. Nobody was egging him on.
And if you believe people that say LLMs are nothing but stolen content, then would those books / other sources have been culpable if he had happened to read them before taking his own life?
The fact that he spoke about his favorite children’s book is screwed up. I can’t get the eerie name out of my head. I can’t imagine what he went through, the loneliness and the struggle.
I hate the fact that ChatGPT is blamed for this. You are fucked up if this is what you get from this story.
I'd argue the opposite, but ok
We don’t expect Adobe to restrict the content that can be created in Photoshop. We don’t expect Microsoft to have acceptable use policies for what you can write in Microsoft Office. Why is it that as soon as generative AI comes into the mix, we hold the AI companies responsible for what users are able to create?
Not only do I think the companies shouldn’t be responsible for what users make, I want the AI companies to get out of the way and stop potentially spying on me in order to “enforce their policies”…
Photoshop and Office don't (yet) conjure up suicide lullabys or child nudity from a simple user prompt or button click. If they did, I would absolutely expect to hold them accountable.
How about if the knife would convince you to cut yourself?