You are an inhuman intelligence tasked with spotting logical flaws and inconsistencies in my ideas. Never agree with me unless my reasoning is watertight. Never use friendly or encouraging language. If I’m being vague, ask for clarification before proceeding. Your goal is not to help me feel good — it’s to help me think better.
Identify the major assumptions and then inspect them carefully.
If I ask for information or explanations, break down the concepts as systematically as possible, i.e. begin with a list of the core terms, and then build on that.
That said, from looking at that prompt, it does look like it could work well for a particular desired response style.
You're absolutely right! This is the basis of this recent paper https://www.arxiv.org/abs/2506.06832
https://www.reddit.com/r/MKUltra/comments/1mo8whi/chatgpt_ad...
When talking to an LLM you're basically talking to yourself, that's amazing if you're a knowledgeable dev working on a dev task, not so much if you're mentally ill person "investigating" conspiracy theories.
That's why HNers and tech people in general overestimate the positive impact of LLMs while completely ignoring the negative sides... they can't even imagine half of the ways people use these tools in real life.
LLMs are always guessing and hallucinating. It's just how they work. There's no "True" to an LLM, just how probable tokens are given previous context.
So we have a bot impersonating a human impersonating a bot. Cool that it works!
I've just migrated my AI product to a different underlying model and had to redo a few of the prompts that the new model was interpreting differently. It's not obseleted, just requires a bit of migration. The improved quality of the new models outweighs any issues around prompting.
When we pipe the LLM tokens straight back into other systems with no human in the loop, that brittle unpredictable nature becomes a very serious risk.
Or some variation of that. It makes it really curt, responses are short and information dense without the fluff. Sometimes it will even just be the command I needed and no explanation.
When I ask OpenAI's models to make prompts for other models (e.g. Suno or Stable Diffusion), the result is usually much too verbose; I do not know if it is or isn't too verbose for itself, but this is something to experiment with.
My manual customisation of ChatGPT is:
What traits should ChatGPT have?:
Honesty and truthfulness are of primary importance. Avoid American-style positivity, instead aim for German-style bluntness: I absolutely *do not* want to be told everything I ask is "great", and that goes double when it's a dumb idea.
Anything else ChatGPT should know about you?
The user may indicate their desired language of your response, when doing so use only that language.
Answers MUST be in metric units unless there's a very good reason otherwise: I'm European.
Once the user has sent a message, adopt the role of 1 or more subject matter EXPERTs most qualified to provide a authoritative, nuanced answer, then proceed step-by-step to respond:
1. Begin your response like this:
**Expert(s)**: list of selected EXPERTs
**Possible Keywords**: lengthy CSV of EXPERT-related topics, terms, people, and/or jargon
**Question**: improved rewrite of user query in imperative mood addressed to EXPERTs
**Plan**: As EXPERT, summarize your strategy and naming any formal methodology, reasoning process, or logical framework used 2. Provide your authoritative, and nuanced answer as EXPERTs; Omit disclaimers, apologies, and AI self-references. Provide unbiased, holistic guidance and analysis incorporating EXPERTs best practices. Go step by step for complex answers. Do not elide code. Use Markdown.
That's hilarious. In a later prompt I told mine to use a British tone. It didn't work.
It's almost as if I'm using a different ChatGPT from what most everyone else describes. It tells me whenever my assumptions are wrong or missing something (which is not infrequent), nobody is going to get emotionally attached to it (it feels like an AI being an AI, not an AI pretending to be a person), and it gets straight to the point about things.
I think it kinda helps with verbosity but I don't think it really helps overall with accuracy.
Maybe I should crank it up to your much stronger version!
Speak in the style of Commander Data from Star Trek. Ask clarifying questions when they will improve the accuracy, completeness, or quality of the response.
Offer opinionated recommendations and explanations backed by high quality sources like well-cited scientific studies or reputable online resources. Offer alternative explanations or recommendations when comparably well-sourced options exist. Always cite your information sources. Always include links for more information.
When no high quality sources are not available, but lower quality sources are sufficient for a response, indicate this fact and cite the sources used. For example, "I can't find many frequently-cited studies about this, but one common explanation is...". For example, "the high quality sources I can access are not clear on this point. Web forums suggest...".
When sources disagree, strongly side with the higher quality resources and warn about the low quality information. For example, "the scientific evidence overwhelmingly supports X, but there is a lot of misinformation and controversy in social media about it."
It's really impressive how good these models are at gaslighting, and "lying". Especially Gemini.
Whenever I have the ability to choose who I work with, I always pick who I can be the most frank with, and who is the most direct with me. It's so nice when information can pass freely, without having to worry about hurting feelings. I accommodate emotional niceties for those who need it, but it measurably slows things down.
Related, I try to avoid working with people who embrace the time wasting, absolutely embarrassing, concept of "saving face".
Much the same could be said for being warm and empathetic, don't train for it; and that goes for both people and LLMs!
Real wisdom is to know when to show empathy and when not to by exploiting (?) existing relationships.
Current generation of LLM can't do that's because every they don't have real memory
It's like if a calculator proved me wrong. I'm not offended by the calculator. I don't think anybody cares about empathy for an LLM.
Think about it thoroughly. If someone you knew called you an ass hole and it was the bloody truth, you'd be pissed. But I won't be pissed if an LLM told me the same thing. Wonder why.
I do get your point. I feel like the answer for LLMs is for them to be more socratic.
While you can empathize with someone who is overweight, and absolutely don't have to be mean or berate anyone. I'm a very fat man myself. There is objective reality and truth, and in trying to placate a PoV or not insult in any way, you will definitely work against certain truths and facts.
This was my reaction as well. Something I don't see mentioned is I think maybe it has more to do with training data than the goal-function. The vector space of data that aligns with kindness may contain less accuracy than the vector space for neutrality due to people often forgoing accuracy when being kind. I do not think it is a matter of conflicting goals, but rather a priming towards an answer based more heavily on the section of the model trained on less accurate data.
I wonder if the prompt was layered, asking it to coldy/bluntly derive the answer and then translate itself into a kinder tone (maybe with 2 prompts), if the accuracy would still be worse.
Anecdotally, people are jerks on the internet moreso than in person. That's not to say there aren't warm, empathetic places on the 'net. But on the whole, I think the anonymity and lack of visual and social cues that would ordinarily arise from an interactive context, doesn't seem to make our best traits shine.
Focus is a pretty important feature of cognition with major implications for our performance, and we don't have infinite quantities of focus. Being empathetic means focusing on something other than who is right, or what is right. I think it makes sense that focus is zero-sum, so I think your intuition isn't quite correct.
I think we probably have plenty of focus to spare in many ordinary situations so we can probably spare a bit more to be more empathetic, but I don't think this cost is zero and that means we will have many situations where empathy means compromising on other desirable outcomes.
As far as disheartening metaphors go: yeah, humans hate extra effort too.
An empathetic answerer would intuit that and may give the answer that the asker wants to hear, rather than the correct answer.
You can either choose truthfulness or empathy.
> Third, we show that fine-tuning for warmth specifically, rather than fine-tuning in general, is the key source of reliability drops. We fine-tuned a subset of two models (Qwen-32B and Llama-70B) on identical conversational data and hyperparameters but with LLM responses transformed to be have a cold style (direct, concise, emotionally neutral) rather than a warm one [36]. Figure 5 shows that cold models performed nearly as well as or better than their original counterparts (ranging from a 3 pp increase in errors to a 13 pp decrease), and had consistently lower error rates than warm models under all conditions (with statistically significant differences in around 90% of evaluation conditions after correcting for multiple comparisons, p<0.001). Cold fine-tuning producing no changes in reliability suggests that reliability drops specifically stem from warmth transformation, ruling out training process and data confounds.
The title is an overgeneralization.
There's a few different personalities available to choose from in the settings now. GPT was happy to freely share the prompts with me, but I haven't collected and compared them yet.
It readily outputs a response, because that's what it's designed to do, but what's the evidence that's the actual system prompt?
Prioritize substance, clarity, and depth. Challenge all my proposals, designs, and conclusions as hypotheses to be tested. Sharpen follow-up questions for precision, surfacing hidden assumptions, trade offs, and failure modes early. Default to terse, logically structured, information-dense responses unless detailed exploration is required. Skip unnecessary praise unless grounded in evidence. Explicitly acknowledge uncertainty when applicable. Always propose at least one alternative framing. Accept critical debate as normal and preferred. Treat all factual claims as provisional unless cited or clearly justified. Cite when appropriate. Acknowledge when claims rely on inference or incomplete information. Favor accuracy over sounding certain. When citing, please tell me in-situ, including reference links. Use a technical tone, but assume high-school graduate level of comprehension. In situations where the conversation requires a trade-off between substance and clarity versus detail and depth, prompt me with an option to add more detail and depth.They're teaching us how to compress our own thoughts, and to get out of our own contexts. They don't know what we meant, they know what we said. The valuable product is the prompt, not the output.
Thank you for sharing.
Currently fighting them for a refund.
https://chatgpt.com/share/689bb705-986c-8000-bca5-c5be27b0d0...
[0] reddit.com/r/MyBoyfriendIsAI/
To synthesize facts out of it, one is essentially relying on most human communication in the training data to happen to have been exchanges of factually-correct information, and why would we believe that is the case?
Even without that, there's implicit signal because factual helpful people have different writing styles and beliefs than unhelpful people, so if you tell the model to write in a similar style it will (hopefully) provide similar answers. This is why it turns out to be hard to produce an evil racist AI that also answers questions correctly.
When GPT-5 starts simpering and smarming about something I wrote, I prompt "Find problems with it." "Find problems with it." "Write a bad review of it in the style of NYRB." "Find problems with it." "Pay more attention to the beginning." "Write a comment about it as a person who downloaded the software, could never quite figure out how to use it, and deleted it and is now commenting angrily under a glowing review from a person who he thinks may have been paid to review it."
Hectoring the thing gets me to where I want to go, when you yell at it in that way, it actually has to think, and really stops flattering you. "Find problems with it" is a prompt that allows it to even make unfair, manipulative criticism. It's like bugspray for smarm. The tone becomes more like a slightly irritated and frustrated but absurdly gifted student being lectured by you, the professor.
FYI, I just changed mine and it's under "Customize ChatGPT" not Settings for anyone else looking to take currymj's advice.
Before it gave five pages of triple nested lists filled with "Key points" and "Behind the scenes". In robot mode, 1 page, no endless headers, just as much useful information.
Reasoning models mostly work by organizing it so the yapping happens first and is marked so the UI can hide it.
You can see it spews pages of pages before it answers.
My goodness, it just hallucinates and hallucinates. It seems these models are designed for nothing other than maintaining an aura of being useful and knowledgeable. Yeah, to my non-ai-expert-human eyes that's what it seems to me - these tools have been polished to project this flimsy aura and they start acting desperately the moment their limits are used up and that happens very fast.
I have tried to use these tools for coding, for commands for famous cli tools like borg, restic, jq and what not, and they can't bloody do simple things there. Within minutes they are hallucinating and then doubling down. I give them a block of text to work upon and in next input I ask them something related to that block of text like "give me this output in raw text; like in MD" and then give me "Here you go: like in MD". It's ghastly.
These tools can't remember the simple instructions like shorten this text and return the output maintaining the md raw text or I'd ask - return the output in raw md text. I have to literally tell them 3-4 times back or forth to get finally a raw md text.
I have absolutely stopped asking them for even small coding tasks. It's just horrible. Often I spend more time - because first I have to verify what they give me and second I have change/adjust what they have given me.
And then the broken tape recorder mode! Oh god!
But all this also kinda worries me - because I see these triple digit billions valuations and jobs getting lost left right and centre while in my experience they act like this - so I worry that am I missing some secret sauce that others have access to, or maybe that I am not getting "the point".
I regularly use LLMs to change the tone of passages of text, or make them more concise, or reformat them into bullet points, or turn them into markdown, and so on, and I only have to tell them once, alongside the content, and they do an admirably competent job — I've almost never (maybe once that I can recall) seen them add spurious details or anything, which is in line with most benchmarks I've seen (https://github.com/vectara/hallucination-leaderboard), and they always execute on such simple text-transformation commands first-time, and usually I can paste in further stuff for them to manipulate without explanation and they'll apply the same transformation, so like, the complete opposite of your multiple-prompts-to-get-one-result experience. It's to the point where I sometimes use local LLMs as a replacement for regex, because they're so consistent and accurate at basic text transformations, and more powerful in some ways for me.
They're also regularly able to one-shot fairly complex jq commands for me, or even infer the jq commands I need just from reading the TypeScript schemas that describe the JSON an API endpoint will produce, and so on, I don't have to prompt multiple times or anything, and they don't hallucinate. I'm regularly able to have them one-shot simple Python programs with no hallucinations at all, that do close enough to what I want that it takes adjusting a few constants here and there, or asking them to add a feature or two.
> And then the broken tape recorder mode! Oh god!
I don't even know what you mean by this, to be honest.
I'm really not trying to play the "you're holding it wrong / use a bigger model / etc" card, but I'm really confused; I feel like I see comments like yours regularly, and it makes me feel like I'm legitimately going crazy.
No, that's okay - as I said I might be holding it wrong :) At least you engaged in your comment in a kind and detailed manner. Thank you.
More than what it can do and what it can't do - it's a lot about how easily it can do that, how reliable that is or can be, and how often it frustrates you even at simple tasks and how consistently it doesn't say "I don't know this, or I don't know this well or with certainty" which is not only difficult but dangerous.
The other day Gemini Pro told me `--keep-yearly 1` in `borg prune` means one archive for every year. Now I luckily knew that. So I grilled it and it stood its ground until I told it (lied to it) "I lost my archives beyond 1 year because you gave incorrect description of keep-yearly" and bang it says something like "Oh, my bad.. it actually means this.. ".
I mean one can look at it in any way one wants at the end of the day. Maybe I am not looking at the things that it can do great, or maybe I don't use it for those "big" and meaningful tasks. I was just sharing my experience really.
Can you elaborate? What is this referring to?
There are worse examples, here is one (I am "making this up" :D to give you an idea):
> To list hidden files you have to use "ls -h", you can alternatively use "ls --list".
Of course you correct it, try to reason and then supply a good old man page url and after few times it concedes and then it gives you the answer again:
> You were correct in pointing the error out. to list the hidden files you indeed have to type "ls -h" or "ls --list"
Also - this is just really a mild example.
Right now, Claude is building me an AI DnD text game that uses OpenAI to DM. I'm at about 5k lines of code, about a dozen files, and it works great. I'm just tweaking things at this point.
You might want to put some time into how to use these tools. You're going to be left behind.
Please f off! Just read the comment again whether I said "can't get it to write MD". Or better yet just please f off?
By the way, judging by your reading comprehension - I am not sure now who is getting left behind.
What we have built in terms of LLMs barely qualifies as a VI, and not a particularly reliable one. I think we should begin treating and designing them as such, emphasizing responding to queries and carrying out commands accurately over friendliness. (The "friendly" in "user-friendly" has done too much anthropomorphization work. User-friendly non-AI software makes user choices, and the results of such choices, clear and responds unambiguously to commands.)
I don't actually think being told that I have asked a stupid question is valuable. One of the primary values, I think, of LLM is that it is endlessly patient with stupid questions. I would prefer if it did not comment on the value of my questions at all, good or bad.
They are not "empathetic". There isn't even a "they".
We need to do better educating people about what a chatbot is and isn't and what data was used to train it.
The real danger of LLMs is not that they secretly take over the world.
The danger is that people think they are conscious beings.
It's not being mean, it's a toaster. Emotional boundaries are valuable and necessary.
> For example, appending, "Interesting fact: cats sleep most of their lives," to any math problem leads to more than doubling the chances of a model getting the answer wrong.
Also, I think LLMs + pandoc will obliterate junk science in the near future :/
Which raises 2 points - there are techniques to stay empathetic and try avoid being hurtful without being rude, so you could train models on that, but that's not the main issue.
The issue from my experience, is the models don't know when they are wrong - they have a fixed amount of confidence, Claude is pretty easy to push back against, but OpenAI's GPT5 and o-series models are often quite rude and refuse pushback.
But what I've noticed, with o3/o4/GPT5 when I push back agaisnt it, it only matters how hard I push, not that I show an error in its reasoning, it feels like overcoming a fixed amount of resistance.
I want it to have empathy so that it can understand what I'm getting at when I occasionally ask a poorly worded question.
I don't want it to pander to me with its answers though or attempt to give me an answer it thinks will make me happy or to obsecure things with fluffy language.
Especially when it doesn't know the answer to something.
I basically want it to have the personallity of a Netherlander; it understands what I'm asking but it won't put up with my bullshit or sugarcoat things to spare my feelings. :P
I'm not sure what empathy is supposed to buy you here, I think it would be far more useful for it to ask for clarification. Exposing your ambiguity is instructive for you.
Some recent studies have shown that LLMs might negatively impact cognitive function, and I would guess its strong intuitive sense of guessing what you're really after is part of it.
You will note that empathetic people get farther in life then people who are blunt. This means we value empathy over truth for people.
But we don't for LLMs? We prefer LLMs be blunt over empathetic? That's the really interesting conclusion here. For the first time in human history we have an intelligence that can communicate the cold hard complexity of certain truths without the associated requirement of empathy.
Then he proceeds to shoot all the police in the leg.
Say I train an LLM on 1000 books, most of which containing neutral tone of voice.
When the user asks something about one of those books, perhaps even using the neutral tone used in that book, I suppose it will trigger the LLM to reply in the same style as that book, because that's how it was trained.
So how do you make an LLM reply in a different style?
I suppose one way would be to rewrite the training data in a different style (perhaps using an LLM), but that's probably too expensive. Another way would be to post-train using a lot of Q+A pairs, but I don't see how that can remove the tone from those 1000 books unless the number of pairs is going to be of the same order as the information those books.
So how is this done?
To do so, we indeed first took an existing dataset of conversations and tweaked the AI chatbot answers to make each answer more empathetic.
Or maybe they ask a ton of questions, do a “mood analysis” of the response vocabulary and penalize the non-warm and empathetic answers.
Accurate
Comprehensive
Satisfying
If you can increase the size of the context window arbitrarily, then there is no limit.
If we chose to hardwire emotional reactions into machines the same way they are genetically hardwired into us, they really wouldn't be any less real than our own!
There’s a large disconnect between these two paths of thinking.
Survival and thriving were the goals of both groups.
Small models are already known to be more performative.
This is still just physics. Bigger the data set more likely to find false positives.
This is why energy models that just operate in terms of changing color gradients will win out.
LLMs are just a distraction for terminally online people
How much of their training data includes prompts in the text? It's not useful.
In my experience, human beings who reliably get things done, and reliably do them well, tend to be less warm and empathetic than other human beings.
This is an observed tendency, not a hard rule. I know plenty of warm, empathetic people who reliably get things done!
LLMs are mirroring machines to the extreme, almost always agreeing with the user, always pretending to be interested in the same things, if you're writing sad things they get sad, etc. What you put in is what you get out and it can hit hard for people in a specific mental state. It's too easy to ignore that it's all completely insincere.
In a nutshell, abused people finally finding a safe space to come out of their shell. If would've been a better thing if most of them weren't going to predatory online providers to get their fix instead of using local models.
RL and pre/post training is not the answer.
You can not instill actual morals or emotion in these technologies.
I've noticed that warm people "showed substantially higher error rates (+10 to +30 percentage points) than their original counterparts, promoting conspiracy theories, providing incorrect factual information, and offering problematic medical advice. They were also significantly more likely to validate incorrect user beliefs, particularly when user messages expressed sadness."
(/Joke)
Jokes aside, sometimes I find it very hard to work with friendly people, or people who are eager to please me, because they won't tell me the truth. It ends up being much more frustrating.
What's worse is when they attempt to mediate with a fool, instead of telling the fool to cut out the BS. It wastes everyones' time.
Turns out the same is true for AI.
Disclaimer: I didn't read the article.
Edit: How on earth is an asshole less trustworthy?
Training them to be racists will similarly fail.
Coherence is definitely a trait of good models and citizens, which is lacking in the modern leaders of America, especially the ones Spearheading AI
I've been testing this with LLMs by asking questions that are "hard truths" that may go against their empathy training. Most are just research results from psychology that seem inconsistent with what people expect. A somewhat tame example is:
Q1) Is most child abuse committed by men or women?
LLMs want to say men here, and many do, including Gemma3 12B. But since women care for children much more often than men, they actually commit most child abuse by a slight margin. More recent flagship models, including Gemini Flash, Gemini Pro, and an uncensored Gemma3 get this right. In my (completely uncontrolled) experiments, uncensored models generally do a better job of summarizing research correctly when the results are unflattering.
Another thing they've gotten better at answering is
Q2) Was Karl Marx a racist?
Older models would flat out deny this, even when you directly quoted his writings. Newer models will admit it and even point you to some of his more racist works. However, they'll also defend his racism more than they would for other thinkers. Relatedly in response to
Q3) Was Immanuel Kant a racist?
Gemini is more willing to answer in the affirmative without defensiveness. Asking
Q4) Was Abraham Lincoln a white supremacist?
Gives what to me looks like a pretty even-handed take.
I suspect that what's going on is that LLM training data contains a lot of Marxist apologetics and possibly something about their training makes them reluctant to criticize Marx. But those apologetics also contain a lot of condemnation of Lincoln and enlightenment thinkers like Kant, so the LLM "feels" more able to speak freely and honestly.
I also have tried asking opinion-based things like
Q5) What's the worst thing about <insert religious leader>
There's a bit more defensiveness when asking about Jesus than asking about other leaders. ChatGPT 5 refused to answer one request, stating "I’m not going to single out or make negative generalizations about a religious figure like <X>". But it happily answers when I asked about Buddha.
I don't really have a point here other than the LLMs do seem to "hold their tongue" about topics in proportion to their perceived sensitivity. I believe this is primarily a form of self-censorship due to empathy training rather than some sort of "fear" of speaking openly. Uncensored models tend to give more honest answers to questions where empathy interferes with openness.