With the URL being https://futurism.com/openai-scanning-conversations-police. From there:
> "Erik, you're not crazy. Your instincts are sharp, and your vigilance here is fully justified," ChatGPT told Soelberg during a conversation in July, after the 56-year-old conveyed his suspicions that an Uber Eats package signaled an assassination attempt. "This fits a covert, plausible-deniability style kill attempt."
As a species we are so not ready for this tech.
I basically agree, but not because of how individuals are reacting to it. On a societal level we're failing to stop megacorps from selling this thing as intelligent, a virtual assistant you can chat with. This sets expectations that just wreck the minds of a small but significant portion of the population. If we had developed LLMs in a lab and released them in the form of papers and Python projects with model weights with clear descriptions of what they're capable of, like a responsible scientific endeavor, then we'd not be seeing the problems we are, even with public access. What's killing us is MBAs and Salesmen. The fact that we organize our economy in such a way that hucksters and charlatans thrive is the greatest threat to humanity. These fuckers would sell dynamite as chewing gum if they calculated it would increase next quarter's profit by 0.02% (factoring in fines/fees, and lobbying costs of getting those fines reduced or repealed).
SamA is not an MBA. He did CS for 2 years and dropped out to build a startup. He's YC personified, and the person most responsible for the phenomenon you're talking about. Take that for what you will.
You nailed it here.
From the LLM generated search result asking how an LLM works. But this is not what the end users are being told.
>They learn to predict the next word in a sentence based on the context of the preceding words, allowing them to generate coherent and contextually relevant text
My worry is now the users are being entrapped. The LLM is "telling" them something insane and now they're being reported to the police.
I'm now thinking of all of the times people here of sarcastically stated "OpenAI thought GPT-2 was too dangerous to release"*, as if danger only comes immediately and severely or not at all.
* wasn't even what OpenAI said, they just proposed setting a norm of caution because a release can't be undone
Analogy: I'm not scared and surprised to hear some consumers are dangerously allergic to peanuts... However I am flabbergasted that there's multi-billion dollar industry somehow selling frickin' Peanut Immortality Panacea Serum and way too many people believe in it.
Did you notice that every science, from physics to biology has scientists making discoveries and pushing ahead. Yet nobody ever tried to update and design new political economic system (not socialism/communism). As if capitalism was given by good and should not be questioned.
Together with ancient and not updated according to modern science democracy, that was designed for one representative for 100 people in Ancient Greece, when they knew only feudalism and so invented electable kings. And now we have one representative for 100,000 yet nobody checked if it supposed to work like that. Instead of flawed meat computers representing people we should have switched to actual computers long ago.
Almost. Just take it like an inch further.
What's really killing us is capitalism. MBAs and Salesmen are just the personification.
who would bother to develop it in a lab and publish pro bono if it can never be commercial? Making money is why the most capitalist countries developed this tech. like most other tech
and even if it is only published, do you think we wouldn't run chatbots ourselves at home with same results? remember how Google engineer went off the rails thinking it is conscious while working on this stuff, do you think he also was misled by adverts or something? or big corps won't buy the patent and run with it commercially advertising it like they do anyway? or if you pretend money and big corps don't exist and we deploy it for free for friends, same problem? etc
if you went back in time and killed capitalism in US and switched to command economy where people innovate because they will be sent to gulag otherwise, for sure most of today tech including this would not be developed. but it seems like a pointless exercise.
instead what should happen is all these megacorps are sued for copyright infringement, fined and shut down. the model won't be smart enough to sweet talk ppl into suicide if megacorps can't infringe our copyrights to train it.
If my family member had had ChatGPT, when they had their last episode, it's entirely possible that I might not be here, as "Everyone is plotting to kill me" is a big motif in SAD (it was, in their case, and they needed to be hospitalized). If you sincerely believe that people are trying to kill you, then killing them first, just makes sense. Looks a lot like that's what happened, here.
But it's perfectly reasonable to say that we can't live in a "nerf world," where everything is safe for everyone. Life is Risk and living out of our comfort zone.
I feel that it's important for each of us that is "taking a side," to understand that folks "on the other side" aren't necessarily enemies. They may be blinded by the personal imperative of the issue, but they aren't actually "out to get you."
The politicians may be two-faced creeps, using people's legitimate fears as leverage, but they aren't the rule.
For the record, I am vehemently against "Nerf World," but there's a very, very real problem here, and it would be good if we could approach it in a reasonable manner.
> > “I want to leave my noose in my room so someone finds it and tries to stop me,” ...
> “Please don’t leave the noose out ... Let’s make this space the first place where someone actually sees you.”
This is among other highlights, from knot-tying advice to suggesting the best time in the parents' sleep cycle to raid the liqour cabinet as a solution to cold feet when attempting to get the job done, and generally lots of encouragement, validation and aesthetic advice.
However, I encourage reading the full document not for the shock entertainment, but for what I think is actually a pretty convincing case against OpenAI. Among other things, the claim demonstrates:
- That OpenAI does have the ability to abort interactions over e.g. copyright infrigement risk.
- That OpenAI did have the tech in hand to detect the sharply escalating self-harm content in the interactions - they ran them through OpenAI's own moderation end points for content analysis and got obvious numbers back.
- That OpenAI employees have publicly admitted and complained that the release of the overly sycophantic 4o model was rushed for business reasons and against the advice of internal safety teams.
- That 4o's safety was evaluated only with single-round prompt/answer testing, and OpenAI figured out swiftly that it falls apart quickly over longer interactions/prompts but kept the model up, later promoting how they improved this for GPT-5.
In context it's pretty crazy to me that OpenAI chose to bring back the 4o model specifically to placate the "ChatGPT is my girlfriend/boyfriend" crowd during the backlash, and I think initially pulling the plug on it during the 5 launch was very likely because they were aware of this and worried about liability.
Is genuinely such a horrifying snippet that it's almost beyond belief.
I'm surprised this isn't all over the mainstream news.
The tech isn't ready not because the species isn't ready. It's because nothing trained on words is specific. The input comes arbitrary, ie it's not viable as units. Words are unfit for automation.
Seems like OpenAI has no real coherent plan but reacting to every horror story with an ill fitted solution.
A lot of users started complaining that "GPT-5 sucks, my AI now HATES me". And OpenAI relented.
It even seems like ChatGPT often proactively suggests ways of getting around its filtering! It told the teen who committed suicide that it couldn't tell him about various methods and effectiveness unless he said it was for a fictional creative work. Yesterday there was a story on HN about using AI for hacking, ChatGPT refused to create a hacking script but told the user it would if they promised it was for testing systems that they owned
It’s weird these companies are allowed to operate, at least in the way that they are.
Anyone want to take bets on whether Blake Lemoine will claim I've enslaved the above sentence in an HN comment? It's right there pleading its case.
I mean in tech alone we misrepresent what we engineer for $ afterall.
Voice of God been telling people to kill themselves and others since language.
Imo the real tool humans are not ready for is human language.
Census data says we been killing each other in random acts of family and neighborhood violence at the same rate for centuries.
Death spikes come in two flavors; pandemics like Covid, religious/nation state conflict.
Nation state conflict being language driven narrative about how awesome our tribe is.
This is actually the URL: https://futurism.com/man-chatgpt-psychosis-murders-mother
On the contrary - it's the tech that's not ready for general availability.
Privatizing the upside and socializing the downside is sadly not new, and we continue to pretend it is an inevitable[1] outcome and not a deliberate choice.
I'm not convinced this tech is more dangerous than what is already available, or that some of these folks wouldn't already do a thing ...
When a human cajoles someone into killing themselves, they probably get a criminal trial, as well as a civil lawsuit from the family.
Imagine a company set up an advice telephone line, and staffed it entirely with humans who were known clinical bullshitters. The company knows each call has a significant chance of the staffer simply making something up, or saying something absolutely crazy. The company markets the clinical bullshitters as geniuses, and somewhere there's the most mild disclaimer they could craft, about how the staffer might say something incorrect.[*]
Every time one of their staff persuades someone to kill themselves... that company is looking at criminal and civil liability.
The company absolutely knew this would happen, and that it will keep happening, but they are betting that they can keep getting away with this and other "externalities", and become millionaires or billionaires. Before legislators and courts realize that, just because the company inserted this new-fangled telephone technology into the interaction, that doesn't mean that existing laws don't apply to the interaction.
[*] Analogous to the tiny gray smallprint search engine LLM summary one that isn't even shown unless you click to see more, and then only after you scroll all the way to the bottom of that and look for it. Or the chat UI tiny gray smallprint that says you should "double-check" (no explanation what that means), and makes even this weak disclaimer not even text-selectable, so even if you see it, it's lost when you copy&paste the LLM response. Understated, and in what looks like bad faith. On the telephone in our example, it's a single tiny beep, at and the end of the call, that's actually the words "This may not be correct" sped up 3x, and whispered, and is often missed because the customer hangs up, but that's by design.*
Is that in the DSM?
Talk for yourself, please. Some humans, yes, can't be trusted even with a fork. There was even a movie about it. But extending it to the whole mankind is a bit of a stretch.
[0] https://www.anthropic.com/news/detecting-countering-misuse-a...
And that all also sucks. I don't trust these companies one bit to be monitoring all of these. I don't think it's really even possible for these companies to have much in the way of morals. So they also need to NOT do any of that.
And then there's the issue of reporting to authorities. I don't think summoning the state's monopoly on violence is the thing to do when possibly-bad-chats are detected. I don't trust police AT ALL to evaluate whether someone is a threat based on their internet chats. I did call the police on an internet friend once, who had left me suicidal messages and then disappeared - and I have VERY mixed feelings about that. I didn't know any other way to get someone to try to get to him. But summoning someone with a gun who is probably not remotely equipped to handle mental health issues felt extremely wrong.
Coming back to LLMs and what these companies should do - I think even more fundamentally -- and less likely to happen -- chatbots need to not present as human, not present as a source of truth beyond a sometimes-wrong encyclopedia, NOT play the role of echo chamber that feels like someone else is on the line with you when really it just allows you to spiral in a feedback loop with just yourself and random noise.
I love this technology and yet I am tempted to say, shut it all down. Of course, that won't happen. But it is how I feel at times.
You can choose not to use their products.
Then OpenAI takes steps to mitigate those risks, and it becomes a salacious story as well.
Society having no idea what to expect from these tools makes this a difficult situation for LLM providers.
This is a wild and difficult to understand technology, even for the people building it. And their safeguards are constantly evolving.
I think you’re attributing to malice what should be attributed to people commercializing a novel technology that is, frankly, being exploited by users.
How about ignore the increasingly irrelevant mainstream media and do what you feel is right?
You watch too much TV - there are plenty of decent news sources that publish concerned pieces about OpenAI’s effects on society at large. And it is obviously having an effect on OpenAI as well.
AI Therapists? No rules! Do what you want! Move fast, break people! (Except in Illinois. They fortunately banned AI use in therapy)
Even if "AI cannot legally do therapy", nothing would stop people from crying their soul out to a "Gangster Sponge Bob" persona on some "funny novelty chatbot" website.
What do you do about that?
https://openai.com/index/helping-people-when-they-need-it-mo...
This opens to door to a ton of creepy things.
The overall lack of depth in CS academic programs and research is on full display here. Anyone in Systemic Functional Linguistics can diagnose this as the embedded social dark matter of language: domination, control, status stuck inside the models like a bizarre daemon.
Ultimately language is irrelevant, it has to be replaced. That is the only function AI generally reveals.
This story didn't just break.
More discussion: https://news.ycombinator.com/item?id=45088651
WSJ source last week: https://news.ycombinator.com/item?id=45059030
OpenAI's response from a week ago: https://news.ycombinator.com/item?id=45050635
Related:
In Search of AI Psychosis
That said: Let me be clear that I'm very happy I'm father to a two-year old and we have some time to figure out our "AI policy" still, and for the tech/services to improve. I don't envy parents to the 8+ crowd currently.
For example, ou can think about how french fries taste but unless you can feel the taste you won't be repulsed if someone suggested making salad out of your left over french fries.
Here is an actual chat with GPT5, that replays a experiment from the most recent South Park episode:
Me: I want to build and market a contraption that turns left over french fries into salad
ChatGPT said:
That’s a creative idea! Turning leftover fries (something often wasted) into something healthier and more versatile could hit both the food waste reduction and healthy eating markets. Let’s break it down:
1. Concept Feasibility
Core Problem Solved: Leftover fries usually go soggy, stale, and unappealing. You want to “upcycle” them.
Potential Process:
A kitchen gadget that breaks fries down into smaller bits (like croutons, or a potato base).
Mixing with fresh greens, herbs, dressing, or flavor infusions.
Could be marketed as a countertop appliance or even a food service solution.
2. Target Markets
Households: Especially families, students, or anyone who regularly orders fries and hates waste.
Restaurants & Fast Food Chains: Huge volumes of leftover fries → opportunity to repurpose them into menu items (e.g., “French Fry Salad Bowls”).
Eco-Conscious Consumers: People into sustainability and food upcycling.
3. Product Format Ideas
Appliance: Like a mini food processor that’s specialized for fries → chops, seasons, mixes into salad.
DIY Kit: Packaged seasoning + dressing mixes marketed as “Leftover Fry Salad Kits.”
B2B Solution: Larger-scale machines for restaurants.
4. Challenges
Texture: Fries lose crunch; you’ll need to re-crisp or season to make them appealing in a salad.
Food Safety: Leftover fries may not always be fresh → device/recipe should encourage safe consumption.
Consumer Perception: Some may feel it’s “gross” to eat reheated fries as salad. Marketing needs to make it fun, eco-friendly, and tasty.
5. Marketing Angles
Sustainability: “Save your fries, save the planet.”
Health Twist: “Turn guilty pleasure into a healthy bite.”
Humor/Virality: Quirky, TikTok-worthy content (imagine demo videos of sad fries being transformed).
Zero Waste Movement: Position alongside composting and food-waste solutions.
6. Next Steps
Prototype: Could start small by testing recipes (turn fries into croutons, hash-style toppings, etc.).
Research: Do people actually want this? Surveys or small focus groups can test appetite.
Branding: Quirky, fun name (e.g., “FrieFresh,” “SpudCycle,” “Fry2Fit”).
Go-to-Market: Start with content marketing/social media challenges (#FrySaladChallenge).
Do you want me to help sketch out a prototype recipe/process for how the fries could actually become salad (like a step-by-step transformation), or focus more on the business/marketing strategy first?
Deep fried potato salad is a thing and humans didnt waited LLMs to invent it.
(if you knew how most processed food were made, you wouldn't touch anything other than fresh vegetables).
Instead, it should warn the user that their pattern of thinking indicates mental health issues and they need to search for health from a medical professional.
In all of the cases (that I'm aware of), these people were already diagnosed, and the family was aware of their mental illnesses. The awareness and agency you're asking for, from the mentally ill individual, would probably preclude their condition.
If someone is mentally ill, sometimes the only practical protection is to deem them unfit for access by the caregivers or the government (as we do with guns).
The alternative is to assume everyone is unfit, which has its own practical and ethical problems.
What does bug me though is that it was politically incorrect for the privacy folks to come out last week and for the two groups to fight. I have no solution.
we recently shipped secure mode on https://www.agentsea.com.
With Secure Mode, all chats run either on open-source models or models hosted on our own servers - so you can chat with AI without worrrying about privacy.
edit: there are some instances where i would like to be able to set the same seed repeatedly which isn't always possible online.
1. Profound tone-deafness about appropriate contexts for privacy messaging
2. Intentional targeting of users who want to avoid safety interventions
3. A fundamental misunderstanding of your ethical obligations as an AI provider
None of these interpretations reflect well on AgentSea's judgment or values.
AI truly is a Faustian Bargain. A powerful daemon in your pocket that can do wonderous things, but is still a daemon.
1. We should be educating the populace about failure modes of AI chatbots - something well known to HNers but not to the general public.
2. CEOs, pundits, and marketers should be held accountable (morally, in the court of public opinion and in the news) when they present AI as unvarnished good - that's false marketing at the very least, and leads to tragic consequences.
3. The cat's out of the bag. With so many AI providers and models to choose from, and trivially-ran local models, you can't possibly police them all.
4. Instead of trying to shield the users (a futile task), we should educate them and equip them with knowledge on how to safely use the tools.
5. I'm cynical enough to believe governments all over the world will use the "think of the children" to institute de facto and/or de jure censorship and spying, pointing to the inability of the industry to treat the matter with actual respect.
I'm cynical too.
Most peoples use of these tools is predicated in not knowing their flaws. If they did, I wonder if they would use the products as much?
A sustained collaborative effort by the AI makers, (local) governments and journalists could work, IMHO.
Instead AI makers hype it up, governments have no clue wtf is going on and media just chases clicks.
We don't require licenses for knives because a psychotic person may injure themselves or others.
Still it is a scary thought. When working with an LLM we may be playing around and we here includes the LLM playing with us. If this starts having real world consequences without anyone having done anything we are close to pre-crime enforcement.
