If you're wondering why new deepseek r1 sounds a bit different, I think they probably switched from training on synthetic openai to synthetic gemini outputs.
Chatgpt loves the em-dash so much that there are no less than **40** tokens in its tokenizer that contain a "―"
You can squash them for good with logit biasing.
Code snippet >>
I made a framework for LLMs to play Diplomacy against each other.
Diplomacy is a complex board game with a heavy negotiation element. Good for experimenting with game theory & testing persuasion!
It's super interesting reading the negotiation logs.
Code & samples follow.
Spiral-Bench 🌀
I've wanted to understand the psychological effects of sycophancy, and the tendency of models to get stuck in escalatory delusion loops w/ users.
I made an eval to get visibility on this.
It measures how a model enables (or prevents) delusional spirals.
🧵
The gpt-oss models have disappointing results on EQ-Bench and creative writing.
It may be a function of the low active params. Though the high perf in other evals suggests the priorities were elsewhere.
Links & writing samples >>
New models on EQ-Bench writing evals: Kimi-K2-Thinking and the stealthed openrouter model polaris-alpha.
Polaris-alpha likely is gpt-5.1, based on how the outputs cluster nearest gpt-5 and the fact that the model has very high rate limits.
I made a new longform writing benchmark. It involves planning out & writing a novella (8x 1000 word chapters) from a minimal prompt. Outputs are scored by sonnet-3.7.
Llama-4 performing not so well. :~(
🔗 Links & writing samples follow.
Been seeing some chatter that the new mistral small 3.2 writes a lot like deepseek v3. This analysis of their slop profiles confirms.
I think the network representation here makes a bit more sense than the phylo tree, given the complicated nature of model lineages.
z.ai's GLM-4.5 gets a very strong result on EQ-Bench & Longform Writing.
In creative writing it's a little further down the pack near Gemma 3 27b & qwen3-235b-a22b.
Its lexical profile clusters nearest to R1-0528.
Recent models tested on longform writing:
Sonoma-sky-alpha: cloaked model appears to be grok.
Qwen3-max: has the same long context degradation issue as qwen3-235b: converges on super short 2-3 word paragraphs.
Kimi-k2-0905: slightly worse than k2 but ~within margin of error
