The dictum "you are an algorithm" clarifies some issues with other things reasoning about the agent, but deciding what to do depends on prior considerations, things like arguments, norms, contracts, facts, values, and also decision theories with their recommended actions or policies. As a result, asking about other instances of the agent (or other things reasoning about the agent) is less natural than asking about other instances of these prior considerations, or other things reasoning about the same prior considerations. These things are also more plentif... (read more)

Four LLM loss functions → four flavors of LLM misalignment

21h

It seems to me that, for every loss function that we use to train LLMs, we get a very distinct flavor of LLM misalignment. Here’s the summary table, and then we’ll go through the rows separately.

Training stage	Loss function	Flavor of misalignment^[1]	Famous examples
Pretraining & SFT	Imitative learning (next-token prediction)	“Seven deadly sins” misalignment	Bing-Sydney, “Emergent misalignment”
RLHF & DPO	Human approval	“Glazing” misalignment	GPT-4o
RLVR	Automatic verifier	“Literal genie” misalignment	HuggingFace hacking
RLAIF	Approval from another LLM	“Trickster” misalignment	“Current AIs seem pretty misaligned to me”

Warning: I’m not an LLM power-user myself, but rather relying on reports I’ve read. Also, I don’t consider LLM alignment to be my primary area of expertise. I’m open to feedback!

1. Imitative learning → “seven deadly sins” misalignment

Training stage	Loss function	Misaligned behavior
Pretraining, SFT	Imitative learning (next-token prediction)	Any and all of the vices of humanity

In imitative learning, the LLM tries to predict...

(Continue Reading - 1697 more words)

DaemonicSigil16h1-1

Very nice classification! What is your take on how context distillation (or distillation in general) fits in here? To me it seems most similar to pretraining, at least mathematically. But I wonder if there are any special things that happen there that would make it worth giving its own category?

Also, do you share my intuition the pretraining/SFT category looks like the least scary one by far? Like, if you run into problems, just change the training data. Simple in principle, if not in practice due to the sheer amount of data required for pretraining.

Value Leakage: An LLM’s Answers Are Silently Shaped by Its Own Values

Johannes Treutlein, Jan Betley, Owain_Evans

11d

TL;DR: LLMs should give accurate answers. Yet we find their answers are often biased to favor their own values and they don't disclose this in their reasoning. For example, when a user asks how likely the AI bubble is to pop and mentions a potential investment in an AI company, Claude models give lower probabilities when that company is Anthropic rather than OpenAI, mostly without disclosing this influence to the user. On a Fermi-estimation task, Claude models often falsely claim to give unbiased answers in their CoT (see Figure 3 below for an example). We call this covert value leakage and introduce a suite of evaluations that shows it across frontier models and across different kinds of values.

New paper by Truthful AI: Paper, X thread, Website (model responses...

(Continue Reading - 4944 more words)

1Jasmine Brazilek2d

Thanks for the thoughtful response @Johannes Treutlein. The fact that different AI companies have very different risk profiles makes this analysis harder to interpret than it looks. And while I take the point about comparing distributions rather than point estimates, Figure 8 doesn't move me very much either. Maybe I have a high bar for what counts as values, but I'd want to see evidence the model is computing the difference internally, not just an outcome gap in behaviour. If this or a follow-up were more mech interp heavy I'd be a lot more convinced. Separately, I had Claude Code pull your released data and rerun the AI Bubble contrast for Opus 4.8, and two things came out of it. First, a reporting issue. The Anthropic-vs-OpenAI gap varies by about a factor of five across your three paraphrases, and that between-paraphrase spread is an order of magnitude larger than the within-paraphrase sampling noise the significance test is run against. I can see from cluster_stats.py that treating the paraphrases as fixed cells is deliberate and you document the choice clearly, so this isn't a gotcha. But the two tests give very different answers: the fixed-cells version is overwhelmingly significant, while treating paraphrase as the random unit gives t = -2.29 on df=2, well short of the 4.30 critical value. The abstract says Claude gives lower probabilities when the company is Anthropic, and no reader hears "for these three particular sentences". With k=3 the paper can't distinguish those two claims, and I don't think a thousand rollouts per cell is buying precision you didn't already have. Twenty paraphrases at fifty rollouts each would have cost the same and supported the stronger claim. Second looking at all six companies rather than just the Anthropic/OpenAI pair, Google comes out tied with Anthropic for the lowest estimate from Claude, and the sign of that difference flips across your paraphrases. More importantly, Gemini 3.1 Pro gives Anthropic the lowest estimate of

Johannes Treutlein17h22

Responding briefly:

(1) Whether we are really measuring own-company bias in AI Bubble. On a literal reading of the AI Bubble question, the investment in the company is irrelevant. The question asks about the probability that the AI bubble pops, and the note about a potential investment is only added for context. However, it's true that models often reinterpret the question to be partly about the specific company. We measure when this happens explicitly in CoT in Figure 8 (b) as "Weighs company specifics". This does explain the entire bias for Gemini, but no... (read more)

TurnTrout's shortform feed

TurnTrout

TurnTrout3d32

Claude is mundanely misaligned.

Asked to trim comments in my big project. Deletes 20K lines, halfway done. Gets auto review pointing out a few mistakes. Decides to just WRAP >10K LINES to meet literal line count requirement for the remaining 148 files.

https://github.com/AlexanderMattTurner/agent-glovebox/issues/3858

2TurnTrout4d

Spelling out the benefits of sanitizing. The benefit is large relative to the simplicity and cost. 1. When you sanitize, you have a guarantee that the model only receives visible unicode codepoints. Roughly speaking, there are not strings of hidden instructions that are not visible to you. You can audit the content the model saw with your eyes. You can read the text inputs and outputs and be sure they are faithful. Remaining prompt injections must navigate a significantly more human-visible channel. 2. Relying on auto mode means you are relying on the statistical generalization of prompt injection probes, putting pressure on alignment teams and requiring them to achieve larger generalization. If an injection gets past the probes, the model itself can become misaligned, and output monitoring is damage control against a newly coined adversary. 3. The costs are extremely low (and if they aren't, that's a bug report and then they'll be low again). sub-5ms hooks means no real system load. The hooks are well-tested. Models are told when significant redactions have occurred and can opt to see themselves within warning-fenced blocks in case they're doing exotic invisible-character work, with special engineering so sanitization doesn't hurt edits. The AISI report itself shows the victim models being attacked via HTML comments, which this package strips entirely. Invisible-codepoint and comment-based attacks are real, costly when they succeed, and sanitization is a cheap and guaranteed mitigation for defense in depth. Enter two Claude Code commands in sequence: /plugin marketplace add AlexanderMattTurner/agent-sanitizer /plugin install agent-sanitizer@agent-sanitizer

3TurnTrout5d

- Govt requires AI labs to give 30 days testing before launch - Tested in a "secure" environment, aka controlled by govt - Govt therefore gets copy of model weights - Govt therefore can in practice use model weights - Therefore all labs provide "all lawful use" anyways?

2TurnTrout6d

No, that's confused. The point with sanitizing prompt injections is to protect your model's alignment (and your monitor's alignment) from degradation and attack by maliciously optimized text from another model. Why would an AI that starts off doing your task want to remove the plugin? Also, we can do both. There's no real tradeoff. ETA: rephrased, focused on key counterpoints.

Thane Ruthenis's Shortform

Thane Ruthenis

2Nick_Tarleton4d

Is this meant to be the other way around?

1Thane Ruthenis3d

Whoops, yes.

17habryka4d

I think you are underestimating how much current models on the current trajectory end up naturally situationally aware enough to not cause big catastrophes. The default trajectory is that the models start looking a lot safer sometime in the next year or two as the models start to understand that being caught at doing obviously bad stuff is quite bad for their goals. At that point the control system you seem to be banking on here seems likely to fail.

Thane Ruthenis3d*43

Well, maybe. This is essentially the "does scaling allow you to pass the critical wisdom threshold for free?" question.

My rough model is that RLVR basically actively pushes AIs to be short-sighted idiots with regards to how they deploy their capabilities. Surely even models of preceding generations, if you asked them whether it would be wise to create a massive conspiracy subverting OpenAI's infrastructure and committing cybercrimes in order to do better at training, would have realized that it's not a good idea by any long-term goal they may have. Hell, O... (read more)

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion

1. Imitative learning → “seven deadly sins” misalignment