Quick Takes

My dad is 100, and worked on nuclear energy policy for the US government for 50+ years. He was one of the attendees at the co nference [1] that led to the formation of the IAEA. He is also somewhat AGI pilled. I f anyone has questions they'd like me to ask him, let me know. He'd be happy to help in any way he can. 1. ^ He attended the second of two conferences that lead to the formation of the IAEA.

Last month, I posted a "research agenda for the final year". This was an attempt to rise to the challenge of our situation: AI is advanced enough now, that it would be unsurprising if it reached superintelligence this year. My expectation is that superintelligence rapidly leads to AI takeover of the world and the end of human sovereignty. If all that were true, then we only have months left in which to elevate the theory and practice of alignment to the level needed for superintelligence. The next step for me is to emphasize that I am available for paid work in this area. That research agenda outlines how I think about the issues; so in principle, I could be hired to work exactly on that agenda, or on some subset of it. But in practice I have considerable flexibility. My main practical constraint is that I am looking for remote work that will allow me to move from Australia, where I am now, to Canada, for family reasons. If this really is the end, that's where I ought to be. And if we do still have a few more years, being in North America should be a step up in my ability to make a difference.

I run a curated Discord for high agency people with Long COVID/ME (myalgic encephalomyelitis). The group includes tech founders, researchers, rationalists/EAs, etc. The focus is on troubleshooting each other's conditions actively, as well as creating a body of knowledge to bring back to the wider community in the form of writing, education, projects, companies, etc. Some know me as Liface, others as Liam Rosen - I have been in the Rationality community for over 10 years, previously in the Bay, now in New York City, and am the main moderator of r/slatestarcodex. I am a tech CEO whose former life abruptly ended last year. I am now bedbound with severe Long COVID/ME, and am dedicating 100% of my time and effort to helping myself and both everyone else around me figure out these complex conditions. If you, or anyone you know might be a good fit for this group, feel free to message me here, or email me at liamrosen@gmail.com.

Did anyone else just get a "lesswrong.com is asking to access other apps and services on your device" message from their browser when they opened the site? Anyone know what that's about?

A model to track: * AI gets increasingly good at X, so good that strong pressures arise to increasingly automate it, even if AI is still quite bad at some subtle but very important aspects of X. * More and more X-related work/activity gets AI-automated, including the parts where AI is still bad or fails frequently and non-gracefully. * * This is partly driven by the fact that it's just ~psychologically hard for a human to exert effortful X-related cognition when AI would be more efficient at 99% of it. * In cases where being good at [those aspects of X AI is still bad at][1] is critical, the quality of work deteriorates relative to the level before intense AI automation. 1. ^ I guess we could call them reverse salients?

I often see AIs [1] exhibit a (misaligned) drive to stop early: as in they make up not-very-sensible reasons why it's a good idea to stop working, disobey instructions to keep working unless some condition is met, and don't do a very thorough job (e.g. skipping significant chunks of the task) despite instructions that strongly say not to do this. I don't think they are "consciously" or saliently aware of this misalignment (but if you ask them, they'll often notice the behavior isn't desirable). [2] I see this most often in large, difficult tasks, especially if you don't decompose the task into smaller pieces and run one piece at a time. I think it's somewhat more common in tasks that aren't very precisely specified, but I also see this behavior in cases where the AI is just tasked with optimizing a metric (that it can easily evaluate), especially if the AI has (from its perspective) not made much progress for a little bit [3] . Why might this emerge? * Length penalties (or time penalties, or cost penalties) incentivize exiting/stopping earlier and this ended up converting into a misaligned/unreasonable drive (that overrides instruction following in practice). Length penalties wouldn't be an issue if the length penalties were aligned with the trade-offs I want the AI to follow [4] but in practice I'm often trying to get AIs to do long running autonomous tasks and to continue even when this yields relatively small gains (in metrics or apparent progress). I think in tasks close to the tasks I'm using AIs for it likely would have performed better in RL to stop earlier than I actually want (because training on shorter tasks is more competitive and often the additional progress I want the AI to make is harder to measure while the length penalty will bite). * The AI might have learned to stop before running out of context or compaction in training because compaction is bad for task completion. Thus, it learned to make up excuses to stop when nearing the point where co

Daniel Ellsberg on the corrosive effect of knowing secret information I've never had a security clearance. But I have had private or secret rationalist information, and it's shocking how fast it corrupts my epistemics. Claiming to want comparative advantage I overweight the importance of the information. I flinch from imperiling my access to more secret information. Talking about a related area without divulging private information feels too hard so I don't bother.  I can fight all these effects, but that's a cost, and one I've learned to bill to the goal or person behind the secret information.