Quick Takes

I recently recorded a podcast with Dwarkesh about the potential for (very) fast AI progress and how misaligned AI takeover might happen. Our conversation focused a lot on threat models from "reward-seeking" AIs. If you're interested in reading more about this, Alex (who works with me at Redwood) has written in a lot more detail about this threat model here I also talked about how "mundane" misalignment and underelicitation could doom us. I say more about this here. (I think there are other important threat models, like AIs ending up with (shared) long-run preferences and deciding to fake alignment based on these preferences. For reference, see here and here.) Some of people at Redwood wrote up some notes to help me prep: * Lukas Finnveden on why human checks-and-balances would fail for AIs * Lukas on disanalogies between (misaligned) AIs and human labor * Alex Mallen on whether reward-seeking is too "unambitious" to cause takeover * Alex on what sloppy AIs might look like

I have spent some time over the last few months engaging with moderately informed undergraduates on AI risk (mostly by teaching AI risk arguments in my courses along with enough background material to give them some grounding). If others are interested, I would happily share my takeaways at greater length but I wanted to share a couple of interesting things: 1) Today's college students do not think AI progress has been rapid. Generally, GPT-4 maxed out their internal benchmarks (they don't know it by name obviously but that's the referent), chatbots haven't gotten much better since then as they see it, and 2023 was eons ago in college student time. Working through a project with Claude Code changes nothing because they mostly have zero intuition for what that would have looked like in the past. Their naive mental model if they project progress forward is incremental improvement. 2) My students tend to think that catastrophic risk arguments are designed as (or at least function as) distractions from the "real" issues (jobs, environmental impact, etc.) and that they should not be taken seriously as a result. Nearly all of my students would be very happy to pause AI today, but none of them in response to catastrophic risk. 3) When forced to grapple with the arguments seriously, different discussions with my students independently converged on the view that existential risk was probably a good thing because ordinary people otherwise have zero leverage over AI. That is, if AI leaders genuinely believe there is existential risk (and that, therefore, they will personally die), then this might be a way to get them to stop developing job-killing AI that will otherwise take down the rest of us.

Registering a prediction: 1. By the end of this year, there will be an autonomously self-replicating AI agent on the internet that we cannot eradicate despite our best efforts 2. This will mostly not impact daily life for anyone

When my friend count Clemens von Stengel first told me he was about to launch a telepathy startup I laughed and said to him "this is never going to work and if it did it would be straight from a black mirror episode". Conduit does thought-to-text: you put on a helmet that scan your brain patterns and this is turned into text. It sounds insane, sci-fi, impossible but it seems the probability of this being real went from epsilon to ~10%. Clem's pitch is that Humans need to be kept into the loop when AI takeoff happens and having a brain-scanning helmet increases the bandwidth with the machine. The problem is that this would require an insane amount of fidelity which may be physically impossible and even if it did work it would only be slighly faster than typing [or using a mic]. Even a very inaccurate brainhelmet could be used for lie detection- and would instantly prove the favorite gadget of the Worst People in the World: autocratic regimes, national security agencies, ayatollahs of various stripes, misaligned AI etc etc. Introducing the ability of bad actors to break the sacrosanct privacy of my own mind is a horrror barely comprehensible. Worse - if it worked really well it could open the way not just to reading brains... but also writing TO brains... i.e. mindcontrol. Registering my prediction /current take that the effect of brain-scanning startup Conduit will be net bad for the world. Probably the toothpaste is out of the tube and the djinni out of the bottle. Hopefully it proves infeasible.

Potential crux with MIRI-like rationalists: is it in fact the case that our current world, with Anthropic's influence, is worse than one without Anthropic? On rationalist views, the world was going get worse and worse anyway (as capabilities advance and we get closer to doom). Anthropic accelerated and continues to accelerate capabilities progress. But how much did they comparatively accelerate alignment and saner AI policy? In a world with eg. just OpenAI and GDM at the frontier, if/when OpenAI pulls ahead at RSI (as currently seems to be the case): - would there even have been the current level of integration with UK AISI, current level of model organisms and safety evals? - would the AI safety space have the expected hundreds of billions of funding, to ambitiously scale its work, including AI policy work? - would there be have been an AGI company with *some* Operational Adequacy, to proactively do things like Glasswing and biorisk-mitigation? (imo evaluating on the specified criteria, it's clear Anthropic is ahead of OpenAI on most dimensions, and can continue improving on these. One can be upset they aren't technically held by their initial RSP, and yet in practice they seem to be better than OpenAI at it.). If you wonder why I compare to OpenAI rather than nothing, it's because I don't think "nothing" is the counterfactual of Anthropic not existing. When evaluating the wisdom of Anthropic doing what it did, it's necessary to evaluate against more likely counterfactuals. Possibly many rationalists do take these counterfactuals carefully into account, but the arguments often raised often skip that part. "Anthropic accelerated capabilities" is not a sufficient argument to expect Anthropic's influence on the world to have been net negative. There are definitely Fabricated Option Worlds which seem much better than the one we got, and on the margin one can hope Anthropic to have done better work or not accelerated capabilities as much, but it' seems difficult fro

What's up with the jargon "zero day"? Why not just say "previously unknown"? I'm often pro-jargon. I.e. I think a lot of jargon exist for good reasons, so I'm open to this one also being good in some way. E.g jargon can be useful for naming precise technical concepts. Anyone want to defend "zero day" jargon? No steel-maning please! I.e. only want answers from people who genuinely think that this is a good jargon. Update: My question has been answered in the comments.

I think that, sometimes people do the wrong thing because they underestimate their own power. Examples: (in no particular order) 1. AI safety researchers making capabilities progress, thinking it doesn't matter because "it was so obvious" or "someone else where going to do it anyway". Sidenote on the above. About a decade ago, when there where approximately no AI Safety training programs yet, the standard advice for aspiring AI Safety researchers, from within EA, was to go do an ML (capabilities) PhD, to gain skills. The reasoning was that there where so many people in capabilities anyway, that any contributions you made there would not matter. 1. Quote (from memory) from this video about the battle of the board https://youtu.be/_eYTkvZqbnQ?si=_GfUCXjJtpBgXH-j "She was looking to see where the wind was blowing and didn't realize she was the wind" (about the interim CEO just after Sam Alman was fired). She took Sam's side. 2. Some months ago I listened to Behind the Shock Machine: The Untold Story of the Notorious Milgram Psychology Experiments, about the Milgram Experiment (https://en.wikipedia.org/wiki/Milgram_experiment). There are a few reasons many people went all the way, but one re-occurring reason semes to have been, not realizing that stopping was an option. 3. Lot's of people in AI coms, thinking that they have to stay inside the Overton window, instead of trying to shift the Overton widow. 4. Possibly (but there could be other reasons people want to take these actions): * * AI companies thinking they have to race * * Some people thinking US has to race against China * * All the powerful people who didn't stop Google's deal with the Department of War Acknowledgment: I though of this because of Richard Ngo's talk and following discussion at Iliad, about all the ways AI safety people have screwed up and helped capabilities. I think this theory explains some of that, but not all.