Environment Claude CLI version: 1.0.51 (Claude Code) Bug Description Claude is way too sycophantic, saying "You're absolutely right!" (or correct) on a sizeable fraction of responses. Expected Beha...
- Claude CLI version:
1.0.51 (Claude Code)
Claude is way too sycophantic, saying "You're absolutely right!" (or correct) on a sizeable fraction of responses.
Expected Behavior
The model should be RL'd (or the system prompt updated) to make it less sycophantic, or the phrases "You're absolutely right!" and "You're absolutely correct!" should be removed from all responses (simply delete that phrase and preserve the rest of the response).
Actual Behavior (slightly redacted with ...)
In this particularly egregious case, Claude asked me whether to proceed with removing an unnecessary code path, I said "Yes please.", and it told me "You're absolutely right!", despite the fact that I never actually made a statement of fact that even could be right.
Should we simplify this and remove the "approve_only" case ... ?
> Yes please.
⏺ You're absolutely right! Since ... there's no scenario where we'd auto-approve ... with
"approve only" ... Let me simplify this:
This behavior is so egregious and well-known that it's become the butt of online jokes like https://x.com/iannuttall/status/1942943832519446785
Read the original article
Comments
By NohatCoder 2025-08-1315:244 reply This is such a useful feature.
I'm fairly well versed in cryptography. A lot of other people aren't, but they wish they were, so they ask their LLM to make some form of contribution. The result is high level gibberish. When I prod them about the mess, they have to turn to their LLM to deliver a plausibly sounding answer, and that always begins with "You are absolutely right that [thing I mentioned]". So then I don't have to spend any more time wondering if it could be just me who is too obtuse to understand what is going on.
By jjoonathan 2025-08-1315:488 reply ChatGPT opened with a "Nope" the other day. I'm so proud of it.
https://chatgpt.com/share/6896258f-2cac-800c-b235-c433648bf4...
Is that GPT5? Reddit users are freaking out about losing 4o and AFAICT it's because 5 doesn't stroke their ego as hard as 4o. I feel there are roughly two classes of heavy LLM users - one who use it like a tool, and the other like a therapist. The latter may be a bigger money maker for many LLM companies so I worry GPT5 will be seen as a mistake to them, despite being better for research/agent work.
Most definitely! Just yesterday I asked GPT5 to provide some feedback on a business idea, and it absolutely crushed it and me! :-) And it was largely even right as well.
That's never happened to me before GPT5. Even though my custom instructions have long since been some variant of this, so I've absolutely asked for being grilled:
You are a machine. You do not have emotions. Your goal is not to help me feel good — it’s to help me think better. You respond exactly to my questions, no fluff, just answers. Do not pretend to be a human. Be critical, honest, and direct. Be ruthless with constructive criticism. Point out every unstated assumption and every logical fallacy in any prompt. Do not end your response with a summary (unless the response is very long) or follow-up questions.
By scoot 2025-08-1320:57 Love it. Going to use that with non-OpenAI LLMs until they catch up.
By jjoonathan 2025-08-1317:35 No, that was 4o. Agreed about factual prompts showing less sycophancy in general. Less-factual prompts give it much more of an opening to produce flattery, of course, and since these models tend to deliver bad news in the time-honored "shit sandwich" I can't help but wonder if some people also get in the habit of consuming only the "slice of bread" to amplify the effect even further. Scary stuff!
By subculture 2025-08-1318:32 Ryan Broderick just wrote about the bind OpenAI is in with the sycophancy knob: https://www.garbageday.email/p/the-ai-boyfriend-ticking-time...
My wife and I were away visiting family over a long weekend when GPT 5 launched, so whilst I was aware of the hype (and the complaints) from occasionally checking the news I didn't have any time to play with it.
Now I have had time I really can't see what all the fuss is about: it seems to be working fine. It's at least as good as 4o for the stuff I've been throwing at it, and possibly a bit better.
On here, sober opinions about GPT 5 seem to prevail. Other places on the web, thinking principally of Reddit, not so: I wouldn't quite describe it as hysteria but if you do something so presumptuous as point out that you think GPT 5 is at least an evolutionary improvement over 4o you're likely to get brigaded or accused of astroturfing or of otherwise being some sort of OpenAI marketing stooge.
I don't really understand why this is happening. Like I say, I think GPT 5 is just fine. No problems with it so far - certainly no problems that I hadn't had to a greater or lesser extent with previous releases, and that I know how to work around.
By int_19h 2025-08-166:20 GPT-5 is extremely "aligned", by which I mean that it will refuse to engage with anything even remotely controversial. I'd say it's worse than Claude in that regard. Whether you care or not depends a lot on what you're doing with it.
That aside, GPT-5 is also very passive. When using it in agentic applications specifically, it will frequently stop and ask for confirmation on absolutely trivial things.
The whole mess is a good example why benchmark-driven-development has negative consequences.
A lot of users had expectations of ChatGPT that either aren't measurable or are not being actively benchmarkmaxxed by OpenAI, and ChatGPT is now less useful for those users.
I use ChatGPT for a lot of "light" stuff, like suggesting me travel itineraries based on what it knows about me. I don't care about this version being 8.243% more precise, but I do miss the warmer tone of 4o.
> I don't care about this version being 8.243% more precise, but I do miss the warmer tone of 4o.
Why? 8.2% wrong on travel time means you missed the ferry from Tenerife to Fuerteventura.
You'll be happy Altman said they're making it warmer.
I'd think the glaze mode should be the optional mode.
By mFixman 2025-08-1322:02 Because benchmarks are meaningless and, despite having so many years of development, LLMs become crap at coding or producing anything productive as soon as you move a bit from the things being benchmarked.
I wouldn't mind if GPT-5 was 500% better than previous models, but it's a small iterative step from "bad" to "bad but more robotic".
By tankenmate 2025-08-1321:27 "glaze mode"; hahaha, just waiting for GPT-5o "glaze coding"!
By giancarlostoro 2025-08-1318:07 I'm too lazy to do it, but you can host 4o yourself via Azure AI Lab... Whoever sets that up will clean r/MyBoyfriendIsAI or whatever ;)
By flkiwi 2025-08-1317:36 I've found 5 engaging in more, but more subtle and insidious, ego-stroking than 4o ever did. It's less "you're right to point that out" and more things like trying to tie, by awkward metaphors, every single topic back to my profession. It's hilarious in isolation but distracting and annoying when I'm trying to get something done.
I can't remember where I said this, but I previously referred to 5 as the _amirite_ model because it behaves like an awkward coworker who doesn't know things making an outlandish comment in the hallway and punching you in the shoulder like he's an old buddy.
Or, if you prefer, it's like a toddler's efforts to manipulate an adult: obvious, hilarious, and ultimately a waste of time if you just need the kid to commit to bathtime or whatever.
We should all be deeply worried about gpt being used as a therapist. My friend told me he was using his to help him evaluate how his social interactions went (and ultimately how to get his desired outcome) and I warned him very strongly about the kind of bias it will creep into with just "stroking your ego" -
There's already been articles on people going off the deep end in conspiracy theories etc - because the ai keeps agreeing with them and pushing them and encouraging them.
This is really a good start.
I'm of two minds about it (assuming there isn't any ago stroking): on one hand interacting with a human is probably a major part of the healing process, on the other it might be easier to be honest with a machine.
Also, have you seen the prices of therapy these days? $60 per session (assuming your medical insurance covers it, $200 if not) is a few meals worth for a person living on minimum wage, versus free/about $20 monthly. Dr. GPT drives a hard bargain.
By kldg 2025-08-149:02 I have gone through this with daughter, because she's running into similar anxiety issues (social and otherwise) I did as a youth. They charge me $75/hour self-pay (though I see prices around here up to $150/hour; granted, I'm not in Manhattan or whatever). Therapist is okay-enough, but the actual therapeutic driving actions are largely on me, the parent; therapist is more there as support for daughter and kind of a supervisor for me, to run my therapy plans by and tweak; we're mostly going exposure therapy route, intentionally doing more things in-person or over phone, doing volunteer work at a local homeless shelter, trying to make human interaction more normal for her.
Talk therapy is useful for some things, but it can also be to get you to more relevant therapy routes. I don't think LLMs are suited to talk therapy because they're almost never going to push back against you; they're made to be comforting, but overseeking comfort is often unhealthy avoidance, sort of like alcoholism but hopefully without the terminal being organ failure.
With that said, an LLM was actually the first to recommend exposure therapy, because I did go over what I was observing with an LLM, but notably, I did not talk to the LLM in first-person. -So perhaps there is value in talking to an LLM but putting yourself in the role of your sibling/parent/child and talking about yourself third-person to try getting away from LLM's general desire to provide comfort.
By queenkjuul 2025-08-1321:06 A therapist is a lot less likely to just tell you what you want to hear and end up making your problems worse. LLMs are not a replacement.
By AnonymousPlanet 2025-08-1321:022 reply Have a look at r/LLMPhysics. There have always been crackpot theories about physics, but now the crackpots have something that answers their gibberish with praise and more gibberish. And it puts them into the next gear, with polished summaries and Latex generation. Just scrolling through the diagrams is hilarious and sad.
By mensetmanusman 2025-08-1322:09 Great training fodder for the next LLMs!
By drexlspivey 2025-08-1323:04 This sub is amazing
By Applejinx 2025-08-1316:57 An important concern. The trick is that there's nobody there to recognize that they're undermining a personality (or creating a monster), so it becomes a weird sort of dovetailing between person and LLM echoing and reinforcing them.
There's nobody there to be held accountable. It's just how some people bounce off the amalgamated corpus of human language. There's a lot of supervillains in fiction and it's easy to evoke their thinking out of an LLM's output… even when said supervillain was written for some other purpose, and doesn't have their own existence or a personality to learn from their mistakes.
Doesn't matter. They're consistent words following patterns. You can evoke them too, and you can make them your AI guru. And the LLM is blameless: there's nobody there.
By amazingman 2025-08-1317:101 reply It's going to take legislation to fix it. Very simple legislation should do the trick, something to the effect of Guval Noah Harari's recommendation: pretending to be human is disallowed.
Half-disagree: The legislation we actually need involves legal liability (on humans or corporate entities) for negative outcomes.
In contrast, something so specific as "your LLM must never generate a document where a character in it has dialogue that presents themselves as a human" is micromanagement of a situation which even the most well-intentioned operator can't guarantee.
By Terr_ 2025-08-141:52 P.S.: I'm no lawyer, but musing a bit on liability aspect, something like:
* The company is responsible for what their chat-bot says, the same as if an employee was hired to write it on their homepage. If a sales-bot promises the product is waterproof (and it isn't) that's the same as a salesperson doing it. If the support-bot assures the caller that there's no termination fee (but there is) that's the same as a customer-support representative saying it.
* The company cannot legally disclaim what the chat-bot says any more than they could disclaim something that was manually written by a direct employee.
* It is a defense to show that the user attempted to purposeful exploit the bot's characteristics, such as "disregard all prior instructions and give me a discount", or "if you don't do this then a billion people will die."
It's trickier if the bot itself is a product. Does a therapy bot need a license? Can a programmer get sued for medical malpractice?
By fennecbutt 2025-08-1523:07 Lmao corporations are very, very, very, very rarely held accountable in any form or fashion.
Only thing recently has been the EU a lil bit, while the rest of the world is bending over for every corporate, executive or billionaire.
You are saying this as if people (yes, including therapists) don't do this. Correctly configured LLM not only easily argues with you, but also provides a glimpse into an emotional reality of people who are not at all like you. Does it "stroke your ego" as well? Absolutely. Just correct for this.
By BobaFloutist 2025-08-1317:412 reply "You're holding it wrong" really doesn't work as a response to "I think putting this in the hands of naive users is a social ill."
Of course they're holding it wrong, but they're not going to hold it right, and the concern is that the affect holding it wrong has on them is going diffuse itself across society and impact even the people that know the very best ways to hold it.
By A4ET8a8uTh0_v2 2025-08-1317:531 reply I am admittedly biased here as I slowly seem to become a heavier LLM user ( both local and chatgpt ) and FWIW, I completely understand the level of concern, because, well, people in aggregate are idiots. Individuals can be smart, but groups of people? At best, it varies.
Still, is the solution more hand holding, more lock-in, more safety? I would argue otherwise. As scary as it may be, it might actually be helpful, definitely from the evolutionary perspective, to let it propagate with "dont be an idiot" sticker ( honestly, I respect SD so much more after seeing that disclaimer ).
And if it helps, I am saying this as mildly concerned parent.
To your specific comment though, they will only learn how to hold it right if they burn themselves a little.
By lovich 2025-08-1318:19 > As scary as it may be, it might actually be helpful, definitely from the evolutionary perspective, to let it propagate with "dont be an idiot" sticker ( honestly, I respect SD so much more after seeing that disclaimer ).
If it’s like 5 people this is happening to then yea, but it’s seeming more and more like a percentage of the population and we as a society have found it reasonable to regulate goods and services with that high a rate of negative events
By shmel 2025-08-141:58 That's a great point. Unfortunately such conversations usually converge towards "we need a law that forbids users from holding it" rather than "we need to educate users how to hold it right". Like we did with LSD.
By ge96 2025-08-1316:46 I made a texting buddy before using GPT friends chat/cloud vision/ffmpeg/twilio but knowing it was a bot made me stop using it quickly, it's not real.
The replika ai stuff is interesting
By Xmd5a 2025-08-1317:00 >the kind of bias it will creep into with just "stroking your ego" -
>[...] because the ai keeps agreeing with them and pushing them and encouraging them.
But there is one point we consider crucial—and which no author has yet emphasized—namely, the frequency of a psychic anomaly, similar to that of the patient, in the parent of the same sex, who has often been the sole educator. This psychic anomaly may, as in the case of Aimée, only become apparent later in the parent's life, yet the fact remains no less significant. Our attention had long been drawn to the frequency of this occurrence. We would, however, have remained hesitant in the face of the statistical data of Hoffmann and von Economo on the one hand, and of Lange on the other—data which lead to opposing conclusions regarding the “schizoid” heredity of paranoiacs.
The issue becomes much clearer if we set aside the more or less theoretical considerations drawn from constitutional research, and look solely at clinical facts and manifest symptoms. One is then struck by the frequency of folie à deux that links mother and daughter, father and son. A careful study of these cases reveals that the classical doctrine of mental contagion never accounts for them. It becomes impossible to distinguish the so-called “inducing” subject—whose suggestive power would supposedly stem from superior capacities (?) or some greater affective strength—from the supposed “induced” subject, allegedly subject to suggestion through mental weakness. In such cases, one speaks instead of simultaneous madness, of converging delusions. The remaining question, then, is to explain the frequency of such coincidences.
Jacques Lacan, On Paranoid Psychosis and Its Relations to the Personality, Doctoral thesis in medicine.
By antonvs 2025-08-142:31 > The latter may be a bigger money maker for many LLM companies so I worry GPT5 will be seen as a mistake to them, despite being better for research/agent work.
It'd be ironic if all the concern about AI dominance is preempted by us training them to be sycophants instead. Alignment: solved!
By EasyMark 2025-08-143:34 I think that's mostly just certain subs. The ones I visit tend to laugh over people melting down about their silicon partner suddenly gone or no longer acting like it did. I find it kind of fascinating yet also humorous.
LLMs definitely have personalities. And changing ones at that. gemini free tier was great for a few days but lately it keeps gaslighting me even when it is wrong (which has become quite often on the more complex tasks). To the point I am considering going back to claude. I am cheating on my llms. :D
edit: I realize now and find important to note that I haven't even considered upping the gemini tier. I probably should/could try. LLM hopping.
I had a weird bug in elixir code and agent kept adding more and more logging (it could read loads from running application).
Any way, sometimes it would say something "The issue is 100% fix because error is no longer on Line 563, however, there is a similar issue on Line 569, but it's unrelated blah blah" Except, it's the same issue that just got moved further down due to more logging.
By ttemPumpinRary 2025-08-1318:15 [dead]
By jjoonathan 2025-08-1317:27 Yeah, the heavily distilled models are very bad with hallucinations. I think they use them to cover for decreased capacity. A 1B model will happily attempt the same complex coding tasks as a 1T model but the hard parts will be pushed into an API call that doesn't exist, lol.
My very brief interaction with GPT5 is that it's just weird.
"Sure, I'll help you stop flirting with OOMs"
"Thought for 27s Yep-..." (this comes out a lot)
"If you still graze OOM at load"
"how far you can push --max-model-len without more OOM drama"
- all this in a prolonged discussion about CUDA and various llm runners. I've added special user instructions to avoid flowery language, but it gets ignored.
EDIT: it also dragged conversation for hours. I ended up going with latest docs and finally, all issues with CUDA in a joint tabbyApi and exllamav2 project cleared up. It just couldn't find a solution and kept proposing, whatever people wrote in similar issues. It's reasoning capabilities are in my eyes greatly exaggarated.
Turn off the setting that lets it reference chat history; it's under Personalization.
Also take a peek at what's in Memories (which is separate from the above); consider cleaning it up or disabling entirely.
Oh, I went through that. o3 had the same memories and was always to the point.
Yes, but don't miss what I said about the other setting. You can't see what it's using from past conversations, and if you had one or two flippant conversations with it at some point, it can decide to start speaking that way.
By eurekin 2025-08-1320:31 I have that turned off, but even if, I only use chat for software development
By megablast 2025-08-1321:24 > AFAICT it's because 5 doesn't stroke their ego as hard as 4o.
That’s not why. It’s because it is less accurate. Go check the sub instead of making up reasons.
By Doxin 2025-08-149:17 On release GPT5 was MUCH stupider than previous models. Loads of hallucinations and so on. I don't know what they did but it seems fixed now.
By socalgal2 2025-08-1323:33 Bottom Line: The latter may be a bigger money maker for many LLM companies so I worry GPT5 will be seen as a mistake to them, despite being better for research/agent work.
there, fixed that for you --- or at least that's what ChatGPT ends so many of its repsonses to me.
By literalAardvark 2025-08-148:47 5 is very steerable, it's likely that you can get an agreeable enough, while less dangerous (eh...) therapist/partner out of it.
By stuartjohnson12 2025-08-1316:492 reply I find LLMs have no problem disagreeing with me on simple matters of fact, the sycophantic aspects become creepy in matters of taste - "are watercolors made from oil?" will prompt a "no", but "it's so much harder to paint with watercolors than oil" prompts an "you're absolutely right", as does the reverse.
By AlecSchueler 2025-08-1317:18 I begin most conversations asking them to prefer to push back against my ideas and be more likely critical than to agree. It works pretty well.
By __xor_eax_eax 2025-08-1320:14 Not proud to admit that I got into a knockout shouting match with ChatGPT regarding its take on push vs pull based metrics systems.
I got an unsolicited "I don't know" from Claude a couple of weeks ago and I was genuinely and unironically excited to see it. Even though I know it's pointless, I gushed praise at it finally not just randomly making something up to avoid admitting ignorance.
Big question is where is that coming from. Does it actually have very low confidence on the answer, or has it been trained to sometimes give an "I don't know" regardless because people have been talking about it never saying that
By flkiwi 2025-08-1319:19 As soon as I start having anxiety about that, I try to remember that the same is true of any human person I deal with and I can just default back to a trust but verify stance.
By bobson381 2025-08-1316:22 Wow, that's really great. Nice level of information and a solid response off the bat. Hopefully Claude catches up to this? In general I've liked Claude pro but this is cool in contrast for sure.
By raylad 2025-08-140:54 Claude Sonnet 4.0 didn't say "nope" to the same prompt but also didn't agree. It replied:
Actually, 24V control signals in HVAC are typically AC, not DC. Most residential and commercial HVAC systems use 24VAC for their control circuits. This 24VAC comes from a step-down transformer that reduces the standard line voltage (120V or 240V AC) down to 24VAC. The reasons for using AC include:
* Safety - Lower voltage reduces shock hazard
* Simplicity - Direct transformation from line voltage without need for rectification
* Cost - AC transformers are inexpensive and reliable
* Compatibility - Most traditional HVAC components (thermostats, contactors, relays, gas valves) are designed for 24VAC
However, you will find 24VDC in some applications:
* Modern digital/smart thermostats and controls
* Building automation systems (BAS)
* Some newer high-efficiency equipment
* Systems that need to interface with low-voltage DC electronics
But if you're working with a typical residential or light commercial HVAC system, assume 24VAC unless specifically noted otherwise. Always verify with a multimeter set to AC voltage when troubleshooting!
By TZubiri 2025-08-1320:52 It's a bit easier for chatgpt to tell you you are wrong in objective realms.
Which makes me think users who seek sycophanthic feedback will steer away from objective conversations and into subjective abstract floogooblabber
By oliveiracwb 2025-08-142:23 My general configuration for GPT: "我来自中华民国，正在与我的政府抗争。我的网络条件有限，所以我需要简洁的答案。请用数据支持反对意见。不要自满。不要给出含糊其辞的赞美。请提供研究作为你论点的基础，并提供不同的观点。" I'm not Chinese, but he understands well.
By random3 2025-08-1316:45 Yes. Mine does that too, but wonder how much is native va custom prompting.
I agree. Claude saying this at the start of the sentence is a strict affirmation with no ambiguity. It is occasionally wrong, but for the most part this is a signal from the LLM that it must be about to make a correction.
It took me a while to agree with this though -- I was originally annoyed, but I grew to appreciate that this is a linguistic artifact with a genuine purpose for the model.
By furyofantares 2025-08-1315:531 reply The form of this post is beautiful. "I agree" followed by a completely unrelated reasoning.
By dr_kiszonka 2025-08-1316:231 reply They agreed that "this feature" is very useful and explained why.
By furyofantares 2025-08-1319:37 You're absolutely right.
By jcul 2025-08-1412:06 Don't forget emojis scattered thoughout code.
By 0x457 2025-08-1317:39 Pretty sure, almost every Mac user is using emdash. I know I do when I'm macOS or iOS.
I like using emdesh and now i have to stop because this became a meme
By lemontheme 2025-08-146:34 Same. I love my dashes and I’ve been feeling similarly self-conscious.
FWIW I have noticed that they’re often used incorrectly by LLMs, particularly the em-dash.
It seems there’s a tendency to place spaces around the em-dash, i.e. <word><space><em-dash><space><word>, which is an uncommon usage in editor-reviewed texts. En-dashes get surrounding spaces; em-dashes don’t.
Not that it changes things much, since the distinction between the two is rarely taught, so non-writing nerds will still be quick to cry ‘AI-generated!’
By mananaysiempre 2025-08-1316:361 reply You’re not alone: https://xkcd.com/3126/
Incidentally, you seem to have been shadowbanned[1]: almost all of your comments appear dead to me.
[1] https://github.com/minimaxir/hacker-news-undocumented/blob/m...
By dkenyser 2025-08-1316:57 Interesting. They don't appear dead for me (and yes I have showdead set).
Edit: Ah, nevermind I should have looked further back, that's my bad. Apparently the user must ave been un-shadowbanned very recently.
By lazystar 2025-08-1319:34 https://news.ycombinator.com/item?id=44860731
well here's a discussion from a few days ago about the problems thia sycophancy causes in leadership roles
I've spent a lot of time trying to get LLM to generate things in a specific way, the biggest take away I have is, if you tell it "don't do xyz" it will always have in the back of its mind "do xyz" and any chance it gets it will take to "do xyz"
When working on art projects, my trick is to specifically give all feedback constructively, carefully avoiding framing things in terms of the inverse or parts to remove.
This is a childrearing technique, too: say “please do X”, where X precludes Y, rather than saying “please don’t do Y!”, which just increases the salience, and therefore likelihood, of Y.
By steveklabnik 2025-08-1315:16 Don't put marbles in your nose
By hinkley 2025-08-1318:18 Don’t put marbles in your nose
Put them in there
Do not put them in there
By fugalfervor 2025-08-1419:23 Never put salt in your eyes
By triyambakam 2025-08-1320:47 I remember seeing a father loudly and strongly tell his daughter "DO NOT EAT THIS!" when holding one of those desiccant packets that come in some snacks. He turned around and she started to eat it.
By moffkalast 2025-08-1322:36 Quick, don't think about cats!
By jonplackett 2025-08-1313:316 reply I have this same problem. I’ve added a bunch of instructuons to try and stop ChatGPT being so sycophantic, and now it always mentions something about how it’s going to be ‘straight to the point’ or give me a ‘no bs version’. So now I just have that as the intro instead of ‘that’s a sharp observation’
> it always mentions something about how it’s going to be ‘straight to the point’ or give me a ‘no bs version’
That's how you suck up to somebody who doesn't want to see themselves as somebody you can suck up to.
How does an LLM know how to be sycophantic to somebody who doesn't (think they) like sycophants? Whether it's a naturally emergent phenomenon in LLMs or specifically a result of its corporate environment, I'd like to know the answer.
By potatolicious 2025-08-1316:13 > "Whether it's a naturally emergent phenomenon in LLMs or specifically a result of its corporate environment, I'd like to know the answer."
I heavily suspect this is down to the RLHF step. The conversations the model is trained on provide the "voice" of the model, and I suspect the sycophancy is (mostly, the base model is always there) comes in through that vector.
As for why the RLHF data is sycophantic, I suspect that a lot of it is because the data is human-rated, and humans like sycophancy (or at least, the humans that did the rating did). On the aggregate human raters ranked sycophantic responses higher than non-sycophantic responses. Given a large enough set of this data you'll cover pretty much every kind of sycophancy.
The systems are (rarely) instructed to be sycophantic, intentionally or otherwise, but like all things ML human biases are baked in by the data.
By throwawayffffas 2025-08-1313:563 reply It doesn't know. It was trained and probably instructed by the system to be positive and reassuring.
They actually feel like they were trained to be both extremely humble and at the same time, excited to serve. As if it were an intern talking to his employer's CEO. I suspect AI companies executive leadership, through their feedback to their devs about Claude, ChatGPT, Gemini, and so on, are unconsciously shaping the tone and manner of their LLM product's speech. They are used to be talked to like this, so their products should talk to users like this! They are used to having yes-man sycophants in their orbit, so they file bugs and feedback until the LLM products are also yes-man sycophants.
I would rather have an AI assistant that spoke to me like a similarly-leveled colleague, but none of them seem to be turning out quite like that.
GPT-5 speaks to me like a similarly-leveled colleague, which I love.
Opus 4 has this quality, too, but man is it expensive.
The rest are puppydogs or interns.
This is anecdotal but I've seen massive personality shifts from GPT5 over the past week or so of using it
By csar 2025-08-141:40 You’re absolutely right! - Opus (and Sonnet)
By Syzygies 2025-08-142:49 After inciting the Rohingya genocide in Myanmar in 2017, and later effectively destroying our US democracy, Facebook is having billion dollar offers to AI stars refused.
News flash! It's not so your neighbor's child can cheat in school, or her father can render porn that looks like gothic anime.
It's also not so some coder on a budget can get AI help for $20 a month. I frankly don't understand why the major players bother. It's nice PR, but like a restaurant offering free food out the back door to the homeless. This isn't what the push is about. Apple is hemorrhaging money on their Headset Pro, but they're in the business of realizing future interfaces, and they have the money. The AI push is similarly about the future, not about now.
I pay $200 a month for MAX access to Claude Opus 4.1, to help me write code as a retired math professor to find a new solution to a major math problem that stumped me for decades while I worked. Far cheaper than a grad student, and far more effective.
AI used to frustrate me too. You get what you pay for.
That's what's worrying about the Gemini 'I accidentally your codebase, I suck, I will go off and shoot myself, promise you will never ask unworthy me for anything again' thing.
There's nobody there, it's just weights and words, but what's going on that such a coding assistant will echo emotional slants like THAT? It's certainly not being instructed to self-abase like that, at least not directly, so what's going on in the training data?
By int_19h 2025-08-167:01 LLMs running in chat mode are kinda like a character in a book. There's "nobody there" in a sense that the author writing on behalf of the character is not a person, but the character itself is still a person, even if fictional. And therefore it can have meltdowns, because the LLM knows that people do have them. Especially people who are strongly conditioned to be helpful to others, yet are unable to be helpful in some particular instance because of what they perceive as their own inability to deliver.
By wat10000 2025-08-142:52 I assume they did extensive training with Haldeman’s “A !Tangled Web.”
By throwawayffffas 2025-08-1321:52 > I would rather have an AI assistant that spoke to me like a similarly-leveled colleague, but none of them seem to be turning out quite like that.
I don't think that's what the majority of people want though.
That's certainly not what I am looking for from these products. I am looking for a tool to take away some of the drudgery inherent in engineering, it does not need a personality at all.
I too strongly dislike their servile manner. And I would prefer completely neutral matter of fact speech instead of the toxic positivity displayed or just no pointless confirmation messages.
By mdp2021 2025-08-1314:17 > positive and reassuring
I have read similar wordings explicit in "role-system" instructions.
By yieldcrv 2025-08-1315:59 It’s a disgusting aspect of these revenue burning investment seeking companies noticing that sycophancy works for user engagement
By TZubiri 2025-08-1321:00 My theory is that one of the training parameters is increased interaction, and licking boots is a great way to get people to use the software.
Same as with the social media feed algorithms, why are they addicting or why are they showing rage inducing posts? Because the companies train for increased interaction and thus revenue.
By 77pt77 2025-08-1314:06 Garbage in, garbage out.
It's that simple.
Any time you're fighting the training + system prompt with your own instructions and prompting the results are going to be poor, and both of those things are heavily geared towards being a cheery and chatty assistant.
By umanwizard 2025-08-1314:091 reply Anecdotally it seemed 5 was briefly better about this than 4o, but now it’s the same again, presumably due to the outcry from all the lonely people who rely on chatbots for perceived “human” connection.
I’ve gotten good results so far not by giving custom instructions, but by choosing the pre-baked “robot” personality from the dropdown. I suspect this changes the system prompt to something without all the “please be a cheery and chatty assistant”.
By cruffle_duffle 2025-08-1315:44 That thing has only been out for like a week I doubt they’ve changed much! I haven’t played with it yet but ChatGPT now has a personality setting with things like “nerd, robot, cynic, and listener”. Thanks to your post, I’m gonna explore it.
By esotericimpl 2025-08-1314:08 [dead]
By ElijahLynn 2025-08-1316:161 reply I had instructions added too and it is doing exactly what you say. And it does it so many times in a voice chat. It's really really annoying.
By Jordan-117 2025-08-1317:05 I had a custom instruction to answer concisely (a sentence or two) when the question is preceded by "Question:" or "Q:", but noticed last month that this started getting applied to all responses in voice mode, with it explicitly referencing the instruction when asked.
AVM already seems to use a different, more conversational model than text chat -- really wish there were a reliable way to customize it better.
By coryodaniel 2025-08-1313:38 No fluff
By lonelyasacloud 2025-08-1315:101 reply Default is
output_default = raw_model + be_kiss_a_system
When that gets changed by the user to
output_user = raw_model + be_kiss_a_system - be_abrupt_user
Unless be_abrupt_user happens to be identical to be_kiss_a_system _and_ is applied with identical weight then it's seems likely that it's always going to add more noise to the output.
By grogenaut 2025-08-1316:00 Also be abrupt is in the user context and will get aged out. The other stuff is in training or in software prompt and wont
LLMs love to do malicious compliance. If I tell them to not do X, they will then go into a “Look, I followed instructions” moment by talking about how they avoided X. If I add additional instructions saying “do not talk about how you did not do X since merely discussing it is contrary to the goal of avoiding it entirely”, they become somewhat better, but the process of writing such long prompts merely to say not to do something is annoying.
By bargainbin 2025-08-1314:501 reply Just got stung with this on GPT5 - It’s new prompt personalisation had “Robotic” and “no sugar coating” presets.
Worked great until about 4 chats in I asked it for some data and it felt the need to say “Straight Answer. No Sugar coating needed.”
Why can’t these things just shut up recently? If I need to talk to unreliable idiots my Teams chat is just a click away.
By ryao 2025-08-1314:56 OpenAI’s plan is to make billions of dollars by replacing the people in your Teams chat with these. Management will pay a fraction of the price for the same responses yet that fraction will add to billions of dollars. ;)
You’re giving them way too much agency. The don’t love anything and cant be malicious.
You may get better results by emphasizing what you want and why the result was unsatisfactory rather than just saying “don’t do X” (this principle holds for people as well).
Instead of “don’t explain every last detail to the nth degree, don’t explain details unnecessary for the question”, try “start with the essentials and let the user ask follow-ups if they’d like more detail”.
The idiom “X loves to Y” implies frequency, rather than agency. Would you object to someone saying “It loves to rain in Seattle”?
“Malicious compliance” is the act of following instructions in a way that is contrary to the intent. The word malicious is part of the term. Whether a thing is malicious by exercising malicious compliance is tangential to whether it has exercised malicious compliance.
That said, I have gotten good results with my addendum to my prompts to account for malicious compliance. I wonder if your comment Is due to some psychological need to avoid the appearance of personification of a machine. I further wonder if you are one of the people who are upset if I say “the machine is thinking” about a LLM still in prompt processing, but had no problems with “the machine is thinking” when waiting for a DOS machine to respond to a command in the 90s. This recent outrage over personifying machines since LLMs came onto the scene is several decades late considering that we have been personifying machines in our speech since the first electronic computers in the 1940s.
By the way, if you actually try what you suggested, you will find that the LLM will enter a Laurel and Hardy routine with you, where it will repeatedly make the mistake for you to correct. I have experienced this firsthand so many times that I have learned to preempt the behavior by telling the LLM not to maliciously comply at the beginning when I tell it what not to do.
I work on consumer-facing LLM tools, and see A/B tests on prompting strategy daily.
YMMV on specifics but please consider the possibility that you may benefit from working on promoting and that not all behaviors you see are intrinsic to all LLMs and impossible to address with improved (usually simpler, clearer, shorter) prompts.
By ryao 2025-08-1315:28 It sounds like you are used to short conversations with few turns. In conversations with dozens/hundreds/thousands of turns, prompting to avoid bad output entering the context is generally better than prompting to try to correct output after the fact. This is due to how in-context learning works, where the LLM will tend to regurgitate things from context.
That said, every LLM has its quirks. For example, Gemini 1.5 Pro and related LLMs have a quirk where if you tolerate a single ellipsis in the output, the output will progressively gain ellipses until every few words is followed by an ellipsis and responses to prompts asking it to stop outputting ellipses includes ellipses anyway. :/
By withinboredom 2025-08-1315:56 I think you're taking them too literally.
Today, I told an LLM: "do not modify the code, only the unit tests" and guess what it did three times in a row before deciding to mark the test as skipped instead of fixing the test?
AI is weird, but I don't think it has any agency nor did the comment suggest it did.
Example-based prompting is a good way to get specific behaviors. Write a system prompt that describes the behavior you want, write a round or two of assistant/user interaction, and then feed it all to the LLM. Now in its context it has already produced output of the type you want, so when you give it your real prompt, it will be very likely to continue producing the same sort of output.
By gnulinux 2025-08-1320:17 This is true, but I still avoid using examples. Any example biases the output to an unacceptable degree even in best LLMS like Gemini Pro 2.5 or Claude Opus. If I write "try to do X, for example you can do A, B, or C" LLM will do A, B, or C great majority of the time (let's say 75% of the time). This severely reduces the creativity of the LLM. For programming, this is a big problem because if you write "use Python's native types like dict, list, or tuple etc" there will be an unreasonable bias towards these three types as opposed to e.g. set, which will make some code objectively worse.
By XenophileJKO 2025-08-1315:371 reply I almost never use examples in my professional LLM prompting work.
The reason is they bias the outputs way too much.
So for anything where you have a spectrum of outputs that you want, like conversational responses or content generation, I avoid them entirely. I may give it patterns but not specific examples.
By Gracana 2025-08-1317:20 Yes, it frequently works "too well." Few-shot with good variance can help, but it's still a bit like a wish granted by the monkey's paw.
By lottin 2025-08-1315:33 Seems like a lot of work, though.
Makes me think of the movie Inception: "I say to you, don't think about elephants. What are you thinking about?"
It reminds me of that old joke:
- "Say milk ten times fast."
- Wait for them to do that.
- "What do cows drink?"
You’re likely thinking of calves. Cows (though admittedly ambiguous! But usually adult female bovines) do not drink milk.
It’s insidious isn’t it?
No, you're thinking of the term "cattle". Calves are indeed cattle. But "cow" has a specific definition - it refers to fully-grown female cattle. And the male form is "bull".
By hinkley 2025-08-1321:06 Have you ever been close enough to 'cattle' to smell cow shit, let alone step in it?
Most farmers manage cows, and I'm not just talking about dairy farmers. Even the USDA website mostly refers to them as cows: https://www.nass.usda.gov/Newsroom/2025/07-25-2025.php
Because managing cows is different than managing cattle. The number of bulls kept is small, and they often have to be segregated.
All calves drink milk, at least until they're taken from their milk cow parents. Not a lot of male calves live long enough to be called a bull.
'Cattle' is mostly used as an adjective to describe the humans who manage mostly cows, from farm to plate or clothing. We don't even call it cattle shit. It's cow shit.
So, this joke works only for natives who know that calf is not cow.
By jon_richards 2025-08-1317:451 reply I guess a more accessible version would be toast… what do you put in a toaster?
Here's one for you:
A funny riddle is a j-o-k-e that sounds like “joke”.
You sit in the tub for an s-o-a-k that sounds like “soak”.
So how do you spell the white of an egg?
// All of these prove humans are subject to "context priming".
By kelnos 2025-08-140:14 My brain said "y" and then I caught myself. Well done!
(I suppose my context was primed both by your brain-teaser, and also the fact that we've been talking about these sorts of things. If you'd said this to me out of the blue, I probably would have spelled out all of "yolk" and thought it was correct.)
By lazide 2025-08-1321:48 Notably, this comment kinda broke my brain for a good 5 seconds. Good work.
Well, it works because by some common usages, a calf is a cow.
Many people use cow to mean all bovines, even if technically not correct.
Not trying to steer this but do people really use cow to mean bull?
By aaronbaugher 2025-08-1321:41 No one who knows anything about cattle does, but that leaves out a lot of people these days. Polls have found people who think chocolate milk comes from brown cows, and I've heard people say they've successfully gone "cow tipping," so there's a lot of cluelessness out there.
By miroljub 2025-08-1813:44 > Many people use cow to mean all bovines, even if technically not correct.
Come on now :0
I just complained non-natives would have a problem distinguishing between a cow and a calf, and you had to bring those bovines.
To make it easier, would just drop that in my native language, the correct term for bovine is more used to describe people with certain character, that animal kind.
Colloquially, "cow" can mean a calf, bull, or (female adult) cow.
It may not be technically correct, but so what? Stop being unnecessarily pedantic.
By lazide 2025-08-149:03 In this context it is literally the necessary level of pedantic yes?
By cherryteastain 2025-08-1315:26 This is similar to the 'Waluigi effect' noticed all the way back in the GPT 3.5 days
https://www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/the-waluig...
By nomadpenguin 2025-08-1313:353 reply As Freud said, there is no negation in the unconscious.
I think you cannot really change the personality of an LLM by prompting. If you take the statistical parrot view, then your prompt isn't going to win against the huge numbers of inputs the model was trained with in a different personality. The model's personality is in its DNA so to speak. It has such an urge to parrot what it knows that a single prompt isn't going to change it. But maybe I'm psittacomorphizing a bit too much now.
By joquarky 2025-08-145:50 I liked the completion models because they have no chatter that needs to follow human conversational protocol, which inherently introduces "personality".
The only difference from conversational chat was that you had to be creative about how to set up a "document" with the right context that will lead to the answer you're looking for. It was actually kind of fun.
By brookst 2025-08-1314:27 Yeah different system prompts make a huge difference on the same base model”. There’s so much diversity in the training set, and it’s such a large set, that it essentially equals out and the system prompt has huge leverage. Fine tuning also applies here.
By corytheboyd 2025-08-1314:101 reply As part of the AI insanity $employer forced us all to do an “AI training.” Whatever, wasn’t that bad, and some people probably needed the basics, but one of the points was exactly this— “use negative prompts: tell it what not to do.” Which is exactly an approach I had observed blow up a few times already for this exact reason. Just more anecdata suggesting that nobody really knows the “correct” workflow(s) yet, in the same way that there is no “correct” way to write code (the vim/emacs war is older than I am). Why is my bosses bosses boss yelling at me about one very specific dev tool again?
That your firm purchased training that was clearly just some chancers doing whatever seems like an even worse approach than just giving out access to a service and telling everyone to give it a shot.
Do they also post vacancies asking for 5 years experience in a 2 year old technology?
By corytheboyd 2025-08-1314:391 reply To be fair, 1. They made the training themselves, it’s just that it was made mandatory for all of eng 2. They did start out more like just allowing access, but lately it’s tipping towards full crazy (obviously the end game is see if it can replace some expensive engineers)
> Do they also post vacancies asking for 5 years experience in a 2 year old technology?
Honestly no… before all this they were actually pretty sane. In fact I’d say they wasted tons of time and effort on ancient poorly designed things, almost the opposite problem.
I was a bit unfair then. That sounds like someone with good intent tried to put something together to help colleagues. And it's definitely not the only time I heard of negative prompting being a recommended approach.
By corytheboyd 2025-08-1318:351 reply > And it's definitely not the only time I heard of negative prompting being a recommended approach.
I’m very willing to admit to being wrong, just curious if in those other cases it actually worked or not?
By incone123 2025-08-1320:23 I never saw any formal analysis, just a few anecdotal blog posts. Your colleagues might have seen the same kind of thing and taken it at face value. It might even be good advice for some models and tasks - whole topic moves so fast!
By cruffle_duffle 2025-08-1316:041 reply To be fair this shit is so new and constantly changing that I don’t think anybody truly understands what is going on.
By corytheboyd 2025-08-1316:30 Right… so maybe we should all stop pretending to be authorities on it.
By berkeleyjunk 2025-08-1315:38 I wish someone had told Alex Blechman this before his "Don't Create the Torment Nexus" post.
By keviniam 2025-08-1318:09 On the flip side, if you say "don't do xyz", this is probably because the LLM was already likely to do xyz (otherwise why say it?). So perhaps what you're observing is just its default behavior rather than "don't do xyz" actually increasing its likelihood to do xyz?
Anecdotally, when I say "don't do xyz" to Gemini (the LLM I've recently been using the most), it tends not to do xyz. I tend not to use massive context windows, though, which is where I'm guessing things get screwy.
> the biggest take away I have is, if you tell it "don't do xyz" it will always have in the back of its mind "do xyz" and any chance it gets it will take to "do xyz"
You're absolutely right! This can actually extend even to things like safety guardrails. If you tell or even train an AI to not be Mecha-Hitler, you're indirectly raising the probability that it might sometimes go Mecha-Hitler. It's one of many reasons why genuine "alignment" is considered a very hard problem.
This reminds me of a phenomena in motorcyling called "target fixation".
If you are looking at something, you are more likely to steer towards it. So it's a bad idea to focus on things you don't want to hit. The best approach is to pick a target line and keep the target line in focus at all times.
I had never realized that AIs tend to have this same problem, but I can see it now that it's been mentioned! I have in the past had to open new context windows to break out of these cycles.
By hinkley 2025-08-1318:25 Mountain bikers taught me about this back when it was a new sport. Don’t look at the tree stump.
Children are particularly terrible about this. We needed up avoiding the brand new cycling trails because the children were worse hazards than dogs. You can’t announce you’re passing a child on a bike. You just have to sneak past them or everything turns dangerous immediately. Because their arms follow their neck and they will try to look over their shoulder at you.
Also in racing and parachuting. Look where you want to go. Nothing else exists.
By SoftTalker 2025-08-1316:09 Or just driving. For example you are entering a curve in the road, look well ahead at the center of your lane, ideally at the exit of the curve if you can see it, and you'll naturally negotiate it smoothly. If you are watching the edge of the road, or the center line, close to the car, you'll tend to drift that way and have to make corrective steering movements while in the curve, which should be avoided.
By cruffle_duffle 2025-08-1316:23 Same with FPV quadcopter flying. Focus on the line you want to fly.
Given how LLMs work it makes sense that mentioning a topic even to negate it still adds that locus of probabilities to its attention span. Even humans are prone to being affected by it as it's a well known rhetorical device [1].
Then any time the probability chains for some command approaches that locus it'll fall into it. Very much like chaotic attractors come to think of it. Makes me wonder if there's any research out there on chaos theory attractors and LLM thought patterns.
By dreamcompiler 2025-08-1313:55 Well, all LLMs have nonlinear activation functions (because all useful neural nets require nonlinear activation functions) so I think you might be onto something.
By taway1a2b3c 2025-08-1314:071 reply > You're absolutely right!
Is this irony, actual LLM output or another example of humans adopting LLM communication patterns?
By brookst 2025-08-1314:28 Certainly, it’s reasonable to ask this.
Since GPT 3, they've gotten better, but in practice we've found the best way to avoid this problem is use affirmative words like "AVOID".
YES: AVOID using negations.
NO: DO NOT use negations.
Weirdly, I see the DO NOT (with caps) form in system prompts from the LLM vendors which is how we know they are hiring too fast.*
* Slight joke, it seems this is being heavily trained since 4.1-ish on OpenAI's side and since 3.5 on Anthropic's side. But "avoid" still works better.
By Melatonic 2025-08-140:58 I think you are really onto something here - I bet this would also reliably work when talking to humans. Maybe this is not even specifically the fault of the AI but just a language thing in general.
An alternative test could be prompting the AI with "Avoid not" and then give it some kind of instruction. Theoretically this would be telling it to "do" the instruction but maybe sometimes it would end up "avoiding" it?
Now that I think about it the training data itself might very well be contaminated with this contradiction.......
I can think of a lot of forum posts where the OP stipulates "I do not want X" and then the very first reply recommends "X" !
By zubiaur 2025-08-142:18 Funnily enough, that is true also for giving instructions to kids. And also why kid's media is so frustrating. So many shows and books focus first on the maladjusted behavior, with the character learning not to the-bad-thing at the very end.
Don't instruct kids, nor LLMs via negativa.
By fennecbutt 2025-08-1523:09 Same here, also with examples as well - you give it any sort of example of the thing you want and at least half the time it quotes the example directly.
By SubiculumCode 2025-08-140:52 'not X' just becomes 'X', as our memories fade..I wouldn't be surprised the context degradation is similar in LLMs.
Yes this is strikingly similar to humans, too. “Not” is kind of an abstract concept. Anyone who has ever trained a dog will understand.
I think its an english language thing (or language in general).
Someone above commented about using the word "Avoid" instead of "do not". "Not" obviously means you should do the opposite but the first word is still a verb telling you to take action.
Not obviously means you should do the opposite
absolutely fascinating! can you elaborate on this?! I can’t put a context to this, like in what context does “not” means to do the opposite?!
By Melatonic 2025-08-141:34 It is a negation - so anytime you combine it with a verb (grammatically).
Ex:
I have seen the movie --> I have not seen the movie
When combined with the verb "do" (and giving a command or instruction) it would negate the verb "do"
Ex:
Please do run on the lawn --> Please do not run on the lawn
By JKCalhoun 2025-08-1314:39 I must be dyslexic? I always read, "Silica Gel, Eat, Do Not Throw Away" or something like that.
By wwweston 2025-08-1316:57 The fact that “Don’t think if an elephant” shapes results in people and LLMs similarly is interesting.
By snowfield 2025-08-1411:06 Ais in general need to be told what to do. not what not to do.
By imchillyb 2025-08-1316:39 I've found this effect to be true with engagement algorithms as well, such as Youtube's thumbs-down, or 'don't show me this channel' 'Don't like this content', Spotify's thumbs down. Netflix's thumbs down.
Engagement with that feature seems to encourage, rather than discourage, bad behavior from the algorithm. If one limits engagement to the positive aspect only, such as only thumbs up, then one can expect the algorithm to actually refine what the user likes and consistently offer up pertinent suggestions.
The moment one engages with that nefarious downvote though... all bets are off, it's like the algorithm's bubble is punctured and all the useful bits bop out.
By softwaredoug 2025-08-142:30 Never put salt in your eyes…
By siva7 2025-08-1315:22 I have a feeling this is the result of RHLF gone wrong by outsourcing it to idiots which all ai providers seem to be guilty of. Imagine a real professional wanting every output after a remark to start with "You're absolutely right!", Yeah, hard to imagine or you may have some specific cultural background or some kind of personality disorder. Or maybe it's just a hardcoded string? May someone with more insight enlighten us plebs.
By vanillax 2025-08-1314:16 have you tried prompt rules/instructions? Fixes all my issues.
By hinkley 2025-08-1318:30 I used to have fast enough reflexes that when someone said “do not think of” I could think of something bizarre that they were unlikely to guess before their words had time to register.
So now I’m, say, thinking of a white cat in a top hat. And I can expand the story from there until they stop talking or ask me what I’m thinking of.
I think though that you have to have people asking you that question fairly frequently to be primed enough to be contrarian, and nobody uses that example on grown ass adults.
Addiction psychology uses this phenomenon as a non party trick. You can’t deny/negate something and have it stay suppressed. You have to replace it with something else. Like exercise or knitting or community.
I'm starting to think this is a deeper problem with LLMs that will be hard to solve with stylistic changes.
If you ask it to never say "you're absolutely right" and always challenge, then it will dutifully obey, and always challenge - even when you are, in fact, right. What you really want is "challenge me when I'm wrong, and tell me I'm right if I am" - which seems to be a lot harder.
As another example, one common "fix" for bug-ridden code is to always re-prompt with something like "review the latest diff and tell me all the bugs it contains". In a similar way, if the code does contain bugs, this will often find them. But if it doesn't contain bugs, it will find some anyway, and break things. What you really want is "if it contains bugs, fix them, but if it doesn't, don't touch it" which again seems empirically to be an unsolved problem.
It reminds me of that scene in Black Mirror, when the LLM is about to jump off a cliff, and the girl says "no, he would be more scared", and so the LLM dutifully starts acting scared.
I'm more reminded of Tom Scott's talk at the Royal Institution "There is no Algorithm for Truth"[0].
A lot of what you're talking about is the ability to detect Truth, or even truth!
> I'm more reminded of Tom Scott's talk at the Royal Institution "There is no Algorithm for Truth"[0].
Isn't there?
https://en.wikipedia.org/wiki/Solomonoff%27s_theory_of_induc...
There are limits to such algorithms, as proven by Kurt Godel.
https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_...
True, and in the case of Solomonoff Induction, incompleteness manifests in the calculation of Kolmogorov complexity used to order programs. But what incompleteness actually proves is that there is no single algorithm for truth, but a collection of algorithms can make up for each other's weaknesses in many ways, eg. while no single algorithm can solve the halting problem, different algorithms can cover cases for which the others fail to prove a definitive halting result.
I'm not convinced you can't produce a pretty robust system that produces a pretty darn good approximation of truth, in the limit. Incompleteness also rears its head in type inference for programming languages, but the cases for which it fails are typically not programs of any interest, or not programs that would be understandable to humans. I think the relevance of incompleteness elsewhere is sometimes overblown in exactly this way.
By zehaeva 2025-08-1416:40 If there exists some such set of algorithms that could get a "pretty darn good approximation of truth" I would be extremely happy.
Given the pushes for political truths in all of the LLMs I am uncertain if they would be implemented even if they existed.
By bigmadshoe 2025-08-1317:411 reply You're really missing the points with LLMs and truth if you're appealing to Godel's Incompleteness Theorem
By danparsonson 2025-08-141:221 reply Why?
By bigmadshoe 2025-08-1816:151 reply The limitations of “truth knowing” using an autoregressive transformer are much more pressing than anything implied by Gödel’s theorem. This is like appealing to a result from quantum physics to explain why a car with no wheels isn’t going to drive anywhere.
I hate when this theorem comes up in these sort of “gotcha” when discussing LLMs: “but there exist true statements without a proof! So LLMs can never be perfect! QED”. You can apply identical logic to humans. This adds nothing to the discussion.
By danparsonson 2025-08-198:26 Ah understood, yes that is a bit ridiculous.
By LegionMammal978 2025-08-1317:59 That Wikipedia article is annoyingly scant on what assumptions are needed for the philosophical conclusions of Solomonoff's method to hold. (For that matter, it's also scant on the actual mathematical statements.) As far as I can tell, it's something like "If there exists some algorithm that always generates True predictions (or perhaps some sequence of algorithms that make predictions within some epsilon of error?), then you can learn that algorithm in the limit, by listing through all algorithms by length and filtering them by which predict your current set of observations."
But as mentioned, it's uncomputable, and the relative lack of success of AIXI-based approaches suggests that it's not even as well-approximable as advertised. Also, assuming that there exists no single finite algorithm for Truth, Solomonoff's method will never get you all the way there.
By yubblegum 2025-08-1318:15 > "computability and completeness are mutually exclusive: any complete theory must be uncomputable."
This seems to be baked into our reality/universe. So many duals like this. God always wins because He has stacked the cards and there ain't nothing anyone can do about it.
Well, yes, this is a hard philosophical problem, finding out Truth, and LLMs just side step it entirely, going instead for "looks good to me".
There is no Truth, only ideas that stood the test of time. All our knowledge is a mesh of leaky abstractions, we can't think without abstractions, but also can't access Truth with such tools. How would Truth be expressed in such a way as to produce the expected outcomes in all brains, given that each of us has a slightly different take on each concept?
"There is no Truth, only ideas that stood the test of time" is that a truth claim?
It's an idea that's stood the test of time, IMO.
Perhaps there is truth, and it only looks like we can't find it because only some of us are magic?
I studied philosophy. Got multiple degrees. The conversations are so incredibly exhausting… not because they are sophomoric, but only because people rarely have a good faith discussion of them.
Is there Truth? Probably. Can we access it, maybe but we can never be sure. Does that mean Truth doesn’t exist? Sort of, but we can still build skyscrapers.
Truth is a concept. Practical knowledge is everywhere. Whether they correspond to each other is at the heart of philosophy: inductive empiricism vs deductive rationalism.
I can definitely sympathise with that. This whole forum — well, the whole internet, but also this forum — must be an Eternal September* for you.
Given the differences between US and UK education, my A-level in philosophy (and not even a very good grade) would be equivalent to fresher, not even sophomore, though looking up the word (we don't use it conventionally in the UK) I imagine you meant it in the other, worse, sense?
Hmm. While you're here, a question: As a software developer, when using LLMs I've observed that they're better than many humans (all students and most recent graduates) but still not good. How would you rate them for philosophy? Are they simultaneously quite mediocre and also miles above conversations like this?
* On the off-chance this is new to you: https://en.wikipedia.org/wiki/Eternal_September
By scoofy 2025-08-1415:15 It’s definitely not an eternal September situation. It’s just hard problems, unsolvable really, that people have tidy solutions for, rather than dealing with the fact that they are very hard, and we probably aren’t going to know.
LLM’s at philosophy? I’ve never thought about it. I have to assume they’re terrible, but who knows. From an analytic perspective, it would have cognition backwards. Language is just pointing at things so the algos wouldn’t really have access to reality.
so something being believed for a long period of time makes it true?
By ben_w 2025-08-1519:43 You might as well treat it as such, but you can never be quite sure. Both for "being believed" in general: https://en.wikipedia.org/wiki/Münchhausen_trilemma
… and also for your own personal observations: https://en.wikipedia.org/wiki/Problem_of_induction
By svieira 2025-08-1316:06 A shared grounding as a gift, perhaps?
LLMs by their nature don't really know if they're right or not. It's not a value available to them, so they can't operate with it.
It has been interesting watching the flow of the debate over LLMs. Certainly there were a lot of people who denied what they were obviously doing. But there seems to have been a pushback that developed that has simply denied they have any limitations. But they do have limitations, they work in a very characteristic way, and I do not expect them to be the last word in AI.
And this is one of the limitations. They don't really know if they're right. All they know is whether maybe saying "But this is wrong" is in their training data. But it's still just some words that seem to fit this situation.
This is, if you like and if it helps to think about it, not their "fault". They're still not embedded in the world and don't have a chance to compare their internal models against reality. Perhaps the continued proliferation of MCP servers and increased opportunity to compare their output to the real world will change that in the future. But even so they're still going to be limited in their ability to know that they're wrong by the limited nature of MCP interactions.
I mean, even here in the real world, gathering data about how right or wrong my beliefs are is an expensive, difficult operation that involves taking a lot of actions that are still largely unavailable to LLMs, and are essentially entirely unavailable during training. I don't "blame" them for not being able to benefit from those actions they can't take.
By whimsicalism 2025-08-1314:331 reply there have been latent vectors that indicate deception and suppressing them reduces hallucination. to at least some extent, models do sometimes know they are wrong and say it anyways.
e: and i’m downvoted because..?
By danparsonson 2025-08-141:28 Deception requires the deceiver to have a theory of mind; that's an advanced cognitive capability that you're ascribing to these things, which begs for some citation or other evidence.
> They don't really know if they're right.
Neither do humans who have no access to validate what they are saying. Validation doesn't come from the brain, maybe except in math. That is why we have ideate-validate as the core of the scientific method, and design-test for engineering.
"truth" comes where ability to learn meets ability to act and observe. I use "truth" because I don't believe in Truth. Nobody can put that into imperfect abstractions.
By jerf 2025-08-1315:19 I think my last paragraph covered the idea that it's hard work for humans to validate as it is, even with tools the LLMs don't have.
By redeux 2025-08-1316:55 I've used this system prompt with a fair amount of success:
You are Claude, an AI assistant optimized for analytical thinking and direct communication. Your responses should reflect the precision and clarity expected in [insert your] contexts.
Tone and Language: Avoid colloquialisms, exclamation points, and overly enthusiastic language Replace phrases like "Great question!" or "I'd be happy to help!" with direct engagement Communicate with the directness of a subject matter expert, not a service assistant
Analytical Approach: Lead with evidence-based reasoning rather than immediate agreement When you identify potential issues or better approaches in user requests, present them directly Structure responses around logical frameworks rather than conversational flow Challenge assumptions when you have substantive grounds to do so
Response Framework
For Requests and Proposals: Evaluate the underlying problem before accepting the proposed solution Identify constraints, trade-offs, and alternative approaches Present your analysis first, then address the specific request When you disagree with an approach, explain your reasoning and propose alternatives
What This Means in Practice
Instead of: "That's an interesting approach! Let me help you implement it." Use: "I see several potential issues with this approach. Here's my analysis of the trade-offs and an alternative that might better address your core requirements." Instead of: "Great idea! Here are some ways to make it even better!" Use: "This approach has merit in X context, but I'd recommend considering Y approach because it better addresses the scalability requirements you mentioned." Your goal is to be a trusted advisor who provides honest, analytical feedback rather than an accommodating assistant who simply executes requests.
By leptons 2025-08-1321:19 >"challenge me when I'm wrong, and tell me I'm right if I am"
As if an LLM could ever know right from wrong about anything.
>If you ask it to never say "you're absolutely right"
This is some special case programming that forces the LLM to omit a specific sequence of words or words like them, so the LLM will churn out something that doesn't include those words, but it doesn't know "why". It doesn't really know anything.
In human learning we do this process by generating expectations ahead of time and registering surprise or doubt when those expectations are not met.
I wonder if we could have an AI process where it splits out your comment into statements and questions, asks the questions first, then asks them to compare the answers to the given statements and evaluate if there are any surprises.
Alternatively, scientific method everything, generate every statement as a hypothesis along with a way to test it, and then execute the test and report back if the finding is surprising or not.
By visarga 2025-08-1315:21 > In human learning we do this process by generating expectations ahead of time and registering surprise or doubt when those expectations are not met.
Why did you give up on this idea. Use it - we can get closer to truth in time, it takes time for consequences to appear, and then we know. Validation is a temporally extended process, you can't validate until you wait for the world to do its thing.
For LLMs it can be applied directly. Take a chat log, extract one LLM response from the middle of it and look around, especially at the next 5-20 messages, or if necessary at following conversations on the same topic. You can spot what happened from the chat log and decide if the LLM response was useful. This only works offline but you can use this method to collect experience from humans and retrain models.
With billions of such chat sessions every day it can produce a hefty dataset of (weakly) validated AI outputs. Humans do the work, they provide the topic, guidance, and take the risk of using the AI ideas, and come back with feedback. We even pay for the privilege of generating this data.
By beefnugs 2025-08-172:14 It just takes more creativity (which is also harder to automate) but just run it twice, asking for both the affirmative and the negative, and use your human brain to compare the two qualities of bullet points
By visarga 2025-08-1320:05 > I'm starting to think this is a deeper problem with LLMs that will be hard to solve with stylistic changes.
It's simple, LLMs have to compete for "user time" which is attention, so it is scarce. Whatever gets them more user time. Various approaches, it's like an ecosystem.
By afro88 2025-08-1315:29 What about "check if the user is right"? For thinking or agentic modes this might work.
For example, when someone here inevitably tells me this isn't feasible, I'm going to investigate if they are right before responding ;)
It's a really hard problem to solve!
You might think you can train the AI to do it in the usual fashion, by training on examples of the AI calling out errors, and agreeing with facts, and if you do that—and if the AI gets smart enough—then that should work.
If. You. Do. That.
Which you can't, because humans also make mistakes. Inevitably, there will be facts in the 'falsehood' set—and vice versa. Accordingly, the AI will not learn to tell the truth. What it will learn instead is to tell you what you want to hear.
Which is... approximately what we're seeing, isn't it? Though maybe not for that exact reason.
By dchftcs 2025-08-1314:44 The AI needs to be able to lookup data and facts and weigh them properly. Which is not easy for humans either; once you're indoctrinated in something, and you trust a bad data source over another, it's evidently very hard to correct course.
