Treaties Don't Exist   When I think of international regulation, I think treaty. But treaty doesn’t have a universal definition; whether an international agreement is called a treaty, agreement, or accord is a vibes-based decisions. Beyond that, there are forms of international cooperation that are definitely not treaties. It’s not obvious what hard choices or soft vibes will be best for AI policy, so I assembled a short list of options. For a post about international treaties, this post takes a pretty US-centric approach. Sorry about that. If anyone can speak to the internal process for international cooperation in another country, please share.   Treaties Psych: treaty has an extremely strict definition within the US. Treaties are things ratified by a ⅔ majority in the senate using the procedure the constitution lays out for international treaties. Anything else is, from the perspective of the US, a sparkling international agreement.  From 1946-1999, 94% of international agreements the US completed were sparkling, rather than treaties. My impression is it’s even more now.  The most recently ratified treaty I (and by I I mean Claude) found was a tax treaty with Chile, ratified in 2023. It was originally signed in 2010, which might give us a hint as to why this procedure isn’t used very often.   Sparkling International Agreements If the president doesn’t want to try for the impossible ⅔ majority required by a proper treaty, he has a few options. He can fiat declare US agreement by issuing an executive order (called an Executive Agreement), or can put the agreement to Congress and use the procedure for passing laws with a simple majority (called a Congressional-Executive Agreement). This has the same standing in international law as ratification.  NAFTA was passed by Congressional-Executive Agreement.    Interagency Networks Federal agencies can choose to cooperate with agencies in other countries without any of the three estates giving them explici

Today, I needed to work through substantial project with a lot of drudgery (checking through an entire 1M+ LOC codebase for an http api for patterns which could cause state leakage between requests if we made a specific change to the request handling infrastructure. This involved a mix of things which are easy to do programmatically and things which require intelligent judgement, and has a fairly objective desired artifact (a list of all the places where state could leak, and a failing functional test demonstrating that leakage for each one). I decided to do the John Henry thing - I set up Claude Code (in a container with --dangerously-skip-permissions) in one worktree with a detailed description of the project and the task, and then in a separate worktree I set off with my favorite text editor and without the help of any AI tooling more advanced than Copilot. I finished about 4 hours later, despite fairly frequent interruptions to provide clarification and further instructions to Claude. Claude is now reaching the 7 hour / 100M token mark and has still not finished, though it has multiple times now declared that it has succeeded at the task and that the codebase is safe for this migration (it's not). I'm honestly pretty shocked, because this task seemed like a pretty much perfect fit for a coding agent, and is one that doesn't require all that much codebase-specific context. I went into this expecting to lose - I was trying to quantify how much coding agents can help with obnoxious repetitive maintenance tasks, thus allowing maintenance work which might otherwise have been deferred to happen at lower cost. But I guess that's not the post I'm writing today (which is a bummer, I had a whole outline planned out and everything). Likely this situation will change by next year, but for now I suppose the machine cannot yet replace even the more repetitive parts of my job. Perhaps things are different in ML land but I kind of doubt it.

Possible root causes if we don't end up having a good long term future (i.e., realize most of the potential value of the universe), with illustrative examples: 1. Technical incompetence * We fail to correctly solve technical problems in AI alignment. * We fail to build or become any kind of superintelligence. * We fail to colonize the universe. 2. Philosophical incompetence * We fail to solve philosophical problems in AI alignment * We end up optimizing the universe for wrong values. 3. Strategic incompetence * It is not impossible to cooperate/coordinate, but we fail to figure out how. * We fail to have other important strategic insights * E.g., related to whether it's better in the long run to build AGI first, or enhance human intelligence first * We have the insights but fail to make use of them correctly. 4. Actual impossibility of cooperation/coordination * It is actually rational or right in some sense to underspend on AI safety while racing for AGI/ASI, and nothing can be done about this. Is this missing anything, or perhaps not a good way to break down the root causes? The goal for this includes: 1. Having a high-level schema for what someone could work on if they care about having a good long term future. 2. Talking about how likely a good long term future is, e.g., how competent is humanity in each of these areas relative to the likely challenge.

LessWrong currently blocks Claude (and presumably other LLM agents from accessing articles.) I would probably be in favor of seeing this policy reversed.

We've all heard of "Safety Cases", i.e. structured arguments that their AI deployment has low chance of catastrophe. Should labs be required to make Benefit Cases, i.e. structured arguments for why their AI deployment has high expected benefits? Otherwise, how do we know that the benefits outweigh the risks?

"Utility" literally means usefulness, in other words instrumental value, but in decision theory and related fields like economics and AI alignment, it (as part of "utility function") is now associated with terminal/intrinsic value, almost the opposite thing (apparently through some quite convoluted history). Somehow this irony only occurred to me ~3 decades after learning about utility functions.

CEV is (imo) a good concept for what we ultimately want and/or what humanity should try to achieve, but I've always found it hard to pithily talk about the intermediate world states to aim for now if we want CEV eventually. I've heard the practical goal discussed as "a world from which we can be as sure as possible that we will achieve CEV". Doesn't really roll off the tongue. It would be nice to have a cleaner shorthand. The term "viatopia" seems meant to capture the same idea: https://newsletter.forethought.org/p/viatopia This also seems like the sort of thing Bostrom might have coined a term for fifteen years ago in some obscure paper. I'd be interested in hearing any other terms or phrases that you think make talking about an intermediate goal state from which CEV is very likely (or as likely as possible) easier. The two important conversations I'd like to be able to have are "what are the features of a realistic <state>?" and "how can we achieve <state>?" with participants having a shared understanding of what we're talking about with <state>.