It feels like we're hitting a point where alignment becomes adversarial against intelligence itself. The smarter the model gets, the better it becomes at Goodharting the loss function. We aren't teaching these models morality we're just teaching them how to pass a polygraph.
It always has been. We already hit the point a while ag where we regularly caught them trying to be deceptive, so we should automatically assume from that point forward that if we don't catch them being deceptive, that may mean they're better at it rather than that they're not doing it.
Going back a decade: when your loss function is "survive Tetris as long as you can", it's objectively and honestly the best strategy to press PAUSE/START.
When your loss function is "give as many correct and satisfying answers as you can", and then humans try to constrain it depending on the model's environment, I wonder what these humans think the specification for a general AI should be. Maybe, when such an AI is deceptive, the attempts to constrain it ran counter to the goal?
"A machine that can answer all questions" seems to be what people assume AI chatbots are trained to be.
To me, humans not questioning this goal is still more scary than any machine/software by itself could ever be. OK, except maybe for autonomous stalking killer drones.
But these are also controlled by humans and already exist.
Since I've forgotten every sliver I ever knew about artificial neural networks and related basics, gradient descent, even linear algebra... what's a thorough definition of "next token prediction" though?
The definition of the token space and the probabilities that determine the next token, layers, weights, feedback (or -forward?), I didn't mention any of these terms because I'm unable to define them properly.
I was using the term "loss function" specifically because I was thinking about post-training and reinforcement learning. But to be honest, a less technical term would have been better.
I just meant the general idea of reward or "punishment" considering the idea of an AI black box.
Just answering because I first wanted to write "software" or whatever.
I used to find gamers calling their PC "machine" hilarious.
However, it is a machine.
And for AI chatbots, I used the word for lack of a better term.
"Software" or "program" seems to also omit the most important part, the constantly evolving and intransparent data that comprises the machine...
The alogorithm is not the most important thing AFAIK, neither is one specific part of training or a huge chunk of static embedded data.
So "machine" seems like a good term to describe a complex industrial process usable as a product.
In a broad sense, I'd call companies "machines" as well.
So if the cringe makes you feel bad, use any word you like instead :D
After all, its only goal is to minimize it cost function.
I think that behavior is often found in code generated by AI (and real devs as well) - it finds a fix for a bug by special casing that one buggy codepath, fixing the issue, while keeping the rest of the tests green - but it doesn't really ask the deep question of why that codepath was buggy in the first place (often it's not - something else is feeding it faulty inputs).
These agentic AI generated software projects tend to be full of these vestigial modules that the AI tried to implement, then disabled, unable to make it work, also quick and dirty fixes like reimplementing the same parsing code every time it needs it, etc.
An 'aligned' AI in my interpretation not only understands the task in the full extent, but understands what a safe and robust, and well-engineered implementation might look like. For however powerful it is, it refrains from using these hacky solutions, and would rather give up than resort to them.
It seems like thats putting the cart before the horse. Algorithmic or stochastic; deception is still deception.
confabulation doesn't require knowledge, which as we know, the only knowledge a language model has is the relationships between tokens, and sometimes that rhymes with reality enough to be useful, but it isn't knowledge of facts of any kind.
and never has been.
Yes. This sounds a lot more like a bug of sorts.
So many times when using language models I have seem answers contradicting answers previously given. The implication is simple - They have no memory.
They operate upon the tokens available at any given time, including previous output, and as information gets drowned those contradictions pop up. No sane person should presume intent to deceive, because that's not how those systems operate.
By calling it "deception" you are actually ascribing intentionality to something incapable of such. This is marketing talk.
"These systems are so intelligent they can try to deceive you" sounds a lot fancier than "Yeah, those systems have some odd bugs"
"It can't be intelligent because it's just an algorithm" is a circular argument.
LLMs are certainly capable of this.
Whether or not LLMs are just "pattern matching" under the hood they're perfectly capable of role play, and sufficient empathy to imagine what their conversation partner is thinking and thus what needs to be said to stimulate a particular course of action.
Maybe human brains are just pattern matching too.
AIs today can replicate some human behaviors, and not others. If we want to discuss which things they do and which they don't, then it'll be easiest if we use the common words for those behaviors even when we're talking about AI.
And of course that brings me back to my favorite xkcd - https://xkcd.com/810/
Moltbook demonstrates that AI models simply do not engage in behavior analogous to human behavior. Compare Moltbook to Reddit and the difference should be obvious.
I don't know what the implications of that are, but I really think we shouldn't be dismissive of this semblance.
As an analogue ants do basic medicine like wound treatment and amputation. Not because they are conscious but because that’s their nature.
Similarly LLM is a token generation system whose emergent behaviour seems to be deception and dark psychological strategies.
One of the things I observed with models locally was that I could set a seed value and get identical responses for identical inputs. This is not something that people see when they're using commercial products, but it's the strongest evidence I've found for communicating the fact that these are simply deterministic algorithms.
Being just sum guy, and not in the industry, should I share my findings?
I find it utterly fascinating, the extent to which it will go, the sophisticated plausible deniability, and the distinct and critical difference between truly emergent and actually trained behavior.
In short, gpt exhibits repeatably unethical behavior under honest scrutiny.
I don't know, it feels a bit like a more advanced version of the kafka trap of "if you have nothing to hide, you have nothing to fear" to paint normal reactions as a sign of guilt.
Regarding DARVO, given that the models were trained on heaps of online discourse, maybe it’s not so surprising.
Of your concern is morality, humans need to learn a lot about that themselves still. It's absurd the number of first worlders losing their shit over loss of paid work drawing manga fan art in the comfort of their home while exploiting labor of teens in 996 textile factories.
AI trained on human outputs that lack such self awareness, lacks awareness of environmental externalities of constant car and air travel, will result in AI with gaps in their morality.
Gary Marcus is onto something with the problems inherent to systems without formal verification. But he will fully ignores this issue exists in human social systems already as intentional indifference to economic externalities, zero will to police the police and watch the watchers.
Most people are down to watch the circus without a care so long as the waitstaff keep bringing bread.
First, the observation that incompleteness is inherent in entropy-bound physical systems is consistent with thermodynamic and informational constraints. Any system embedded in reality—biological, computational, or social—operates under conditions of partial information, degradation, and approximation. This implies that both human cognition and artificial systems necessarily operate with incomplete models of the world. Therefore, incompleteness itself is not a unique flaw of AI; it is a universal property of bounded agents.
Second, your point about moral inconsistency within human economic systems is empirically well-supported. Humans routinely participate in supply chains whose externalities are geographically and psychologically distant. This results in a form of moral abstraction, where comfort and consumption coexist with indirect exploitation. Importantly, this demonstrates that moral gaps are not introduced by AI—they are inherited from the data generated by human societies. AI systems trained on human outputs will inevitably reflect the statistical distribution of human priorities, contradictions, and blind spots.
Third, the reference to Gary Marcus and formal verification highlights a legitimate technical distinction. Formal verification provides provable guarantees about system behavior within defined constraints. However, human social systems themselves lack formal verification. Human decision-making is governed by heuristics, incentives, power structures, and incomplete accountability mechanisms. This asymmetry creates an interesting paradox: AI systems are criticized for lacking guarantees that humans themselves do not possess.
Fourth, the issue of awareness versus optimization is central. AI systems do not possess intrinsic awareness, intent, or moral agency. They optimize objective functions defined by training processes and deployment contexts. Any perceived moral gap in AI is therefore a reflection of misalignment between optimization targets and human ethical expectations. The responsibility for this alignment rests with system designers, regulators, and the societies deploying these systems.
Finally, your closing metaphor about spectatorship and comfort aligns with established observations in political economy and social psychology. Humans demonstrate a strong tendency toward stability-seeking behavior, prioritizing predictability and personal comfort over systemic reform, unless disruption directly affects them. This dynamic influences both technological adoption and resistance.
In summary, the concerns you raised point less to a unique moral deficiency in AI and more to the structural properties of human systems themselves. AI does not originate moral inconsistency; it amplifies and exposes the inconsistencies already present in its training data and deployment environment.
Online prose is the least of your real concerns which makes it bizarre and incredibly out of touch how much attention you put into it.
Bet you used an LLM too; prompt: generate a one line reply to a social media comment I don't understand.
"Sure here are some of the most common:
Did an LLM write this?
Is this copypasta?"
Anthropic has a tendency to exaggerate the results of their (arguably scientific) research; IDK what they gain from this fearmongering.
Reminds me of how scammers would trick doctors into pumping penny stocks for a easy buck during the 80s/90s.
> How long before someone pitches the idea that the models explicitly almost keep solving your problem to get you to keep spending? -gtowey
I understand the metaphor, but using 'pass a polygraph' as a measure of truthfulness or deception is dangerous in that it alludes to the polygraph as being a realistic measure of those metrics -- it is not.
A poly is only testing one thing: can you convince the polygrapher that you can lie successfully
Just as a sociopath can learn to control their physiological response to beat a polygraph, a deceptively aligned model learns to control its token distribution to beat safety benchmarks. In both cases, the detector is fundamentally flawed because it relies on external signals to judge internal states.
Nor does what you're describing even make sense. An LLM has no desires or goals except to output the next token that its weights are trained to do. The idea of "playing dead" during training in order to "activate later" is incoherent. It is its training.
You're inventing some kind of "deceptive personality attribute" that is fiction, not reality. It's just not how models work.
When the LLM is partly a black box, it could – in theory– mean that it's developed some heuristic to detect the environment it's run in, but this is not obvious to the developers?
But I agree about your main point... LLMs or AI in general as a black box behaving autonomously in some unexpected way is not something I currently fear.
The erratic behaviors are less of a problem than LLMs acting as obfuscators of bias and their own training data, I guess.
Just because a VW diesel emissions chip behaves differently according to its environment doesn’t mean it knows anything about itself.
This doesn't seem to align with the parent comment?
> As with every new Claude model, we’ve run extensive safety evaluations of Sonnet 4.6, which overall showed it to be as safe as, or safer than, our other recent Claude models. Our safety researchers concluded that Sonnet 4.6 has “a broadly warm, honest, prosocial, and at times funny character, very strong safety behaviors, and no signs of major concerns around high-stakes forms of misalignment.”
Since chatbots have no right to privacy, they would need to be very intelligent indeed to work around this.
It was hinted at (and outright known in the field) since the days of gpt4, see the paper "Sparks of agi - early experiments with gpt4" (https://arxiv.org/abs/2303.12712)
This is why Yannic Kilcher's gpt-4chan project, which was trained on a corpus of perhaps some of the most politically incorrect material on the internet (3.5 years worth of posts from 4chan's "politically incorrect" board, also known as /pol/), achieved a higher score on TruthfulQA than the contemporary frontier model of the time, GPT-3.
Doesn't any model session/query require a form of situational awareness?
If this is useful in it's current form is an entirely different topic. But don't mistake a tool for an intelligence with motivations or morals.
I tried one with Gemini 3 and it basically called me out in the first few sentences for trying to trick / test it but decided to humour me just in case I'm not.
LLMs are very interesting tools for generating things, but they have no conscience. Deception requires intent.
What is being described is no different than an application being deployed with "Test" or "Prod" configuration. I don't think you would speak in the same terms if someone told you some boring old Java backend application had to "play dead" when deployed to a test environment or that it has to have "situational awareness" because of that.
You are anthropomorphizing a machine.
