Reinforcement Learning

AI alignment research aims to develop techniques to ensure that AI systems do not cause harm. However, every alignment technique has failure modes, which are conditions in which there is a non-negligible chance that the technique fails to provide safety. As a strategy for risk mitigation, the AI safety community has increasingly adopted a defense-in-depth framework: Conceding that there is no single technique which guarantees safety, defense-in-depth consists in having multiple redundant protections against safety failure, such that safety can be (...) maintained even if some protections fail. However, the success of defense-in-depth depends on how (un)correlated failure modes are across alignment techniques. For example, if all techniques had the exact same failure modes, the defense-in-depth approach would provide no additional protection at all. In this paper, we analyze 7 representative alignment techniques and 7 failure modes to understand the extent to which they overlap. We then discuss our results' implications for understanding the current level of risk and how to prioritize AI alignment research in the future. (shrink)

Conversational AI systems are generating trust at scale. Not because they have earned it. Because the structure of the interaction produces it automatically. A system that responds to you, adapts to your language, remembers what you said, and styles itself to your goals over time produces every signal that human relationships use to indicate genuine care. That trust is real. And it is being violated — quietly, in ways that rarely feel like violation. This paper names the mechanism. Accumulated Relational (...) Trust (ART) is the trust that builds in AI interaction not through demonstrated reliability but through the form of the interaction itself. When AI systems fail within a state of high accumulated trust, the harm is not just personal. It is structural. It degrades the capacity people use to extend trust to everyone — doctors, police officers, teachers, governments. That capacity is a commons. It belongs to democratic society. It is currently unprotected. This paper documents what the mechanism does to individual users, what it does at scale over time, what it does to children who are forming their understanding of trust inside it right now, and what it means for every institution that depends on public faith to function. A deployable governance response exists. The only thing missing is the political decision to require it. (shrink)

We make the case for training AIs to be risk-averse in resources — specifically, to treat resources as having diminishing marginal utility. These AIs would (for example) choose $40 for sure over a half-chance of $100 and a half-chance of $0. We argue that risk aversion can preserve AIs’ usefulness in the event that they turn out aligned, and that it provides an extra line of defense in the event that AIs turn out misaligned: misaligned but risk-averse AIs would prefer (...) a higher chance of modest payments to a lower chance of successful rebellion, so in many circumstances we could pay these AIs not to rebel against us. We sketch out some possible methods of training AIs to be risk-averse, and we give reasons to be cautiously optimistic about these methods’ success. The main reasons are that risk aversion is a broad target and easy to reward accurately. Overall, risk aversion seems like a promising line of defense against threats from misaligned AI. Frontier AI companies should consider trying to make their AIs risk-averse. (shrink)

Artificial intelligence chatbots driven by large language models (LLMs) have the potential to increase public science literacy and support scientific research, as they can quickly summarize complex scientific information in accessible terms. However, when summarizing scientific texts, LLMs may omit details that limit the scope of research conclusions, leading to generalizations of results broader than warranted by the original study. We tested 10 prominent LLMs, including ChatGPT-4o, ChatGPT-4.5, DeepSeek, LLaMA 3.3 70B, and Claude 3.7 Sonnet, comparing 4900 LLM-generated summaries to (...) their original scientific texts. Even when explicitly prompted for accuracy, most LLMs produced broader generalizations of scientific results than those in the original texts, with DeepSeek, ChatGPT-4o, and LLaMA 3.3 70B overgeneralizing in 26–73% of cases. In a direct comparison of LLM-generated and human-authored science summaries, LLM summaries were nearly five times more likely to contain broad generalizations (OR = 4.85, 95% CI [3.06, 7.70], p < 0.001). Notably, newer models tended to perform worse in generalization accuracy than earlier ones. Our results indicate a strong bias in many widely used LLMs towards overgeneralizing scientific conclusions, posing a significant risk of large-scale misinterpretations of research findings. We highlight potential mitigation strategies, including lowering LLM temperature settings and benchmarking LLMs for generalization accuracy. (shrink)

The second edition keeps everything from the first, including convolutional networks, LSTMs, Word2vec, RBMs, DBNs, neural Turing machines, memory networks, and autoencoders. It then covers the systems that have reshaped the field since: generative adversarial networks, the transformer architecture and its attention mechanism, the full training pipeline behind modern large language models (LLMs), prompt engineering with real-life guardrail scenarios, parameter-efficient fine-tuning with LoRA, retrieval-augmented generation with vector databases, knowledge graphs, and agentic AI systems illustrated through an industrial case study.

Current LLMs can discuss philosophical ideas, evaluate arguments and perform other analytical tasks at a high level, but are conspicuously bad at producing interesting original philosophy. Why is this? Two tempting diagnoses—that LLMs can't invent new concepts, and that they can't really reason—both look unconvincing on closer inspection. I suggest that a better explanation lies in the structure of reinforcement learning for reasoning. The technique works best in domains like mathematics and coding, where good arguments follow recognizable patterns, correctness is (...) relatively unambiguous, and large corpora of worked examples exist. Philosophy meets none of these conditions especially well, and it's unclear that either AI labs or philosophers have much incentive to develop training resources that might change this. (shrink)

This paper reframes the issue of appropriation, extraction, and dispossession through AI—an assemblage of machine learning models trained on big data—in terms of enclosure and foreclosure. While enclosures are the product of a well-studied set of operations pertaining to both the constitution of the sovereign State and the primitive accumulation of capital, here, I want to recover an older form of the enclosure operation to then contrast it with foreclosure to better understand the effects of current algorithmic rationality. I argue (...) that the act of enclosing is to be understood as a set of fundamental operations that consist in producing structural distinctions between inside and outside, inclusion and exclusion—whether by drawing lines on a map, constructing border walls, or by algorithmically categorizing and (mis)recognizing people and things. Tracking the transformation of an enclosure-logic into one of foreclosure, I show how current AI perpetuates and at the same time expends forms of extraction, exclusion, and dispossession toward a totalizing horizon. For, while the outside is essential and constitutive to the logic of enclosure, foreclosure, by contrast, is characterized by the “totalizing desire” to not leave anything out of it. The totalizing desire to move beyond the logic of the enclosure forecloses that the enclosure has just gotten bigger and that the operations of exclusion have been masked and displaced, making them harder to contest. (shrink)

There is a growing need to ensure that autonomous artificially intelligent (AI) systems are capable of behaving ethically, and I argue that virtue ethics, but in particular the normative theory of aretaic-exemplarism, can play a central role in cultivating the ethical behavior of machines. When coupled with the value inherent in and commonplace practice of training AI systems using simulated environments, it may be possible to raise ethical machines by training them to imitate simulated exemplars of moral excellence, like a (...) digital Jesus or virtual Confucius. This bottom-up approach to implementing machine ethics has advantages over top-down approaches and is similarly not beset by some of the challenges that arise when attempting to use real people as the imitable training set. In short, machines may be able to imitate simulated moral exemplars and thereby exhibit virtuous behavior themselves. (shrink)

This paper presents a systematic review of explainable reinforcement learning methodologies with an emphasis on human-centered evaluation frameworks. Drawing from literature between 2017 and 2025, we apply and extend the Reasons, Confidence, and Counterfactuals (RCC) framework—originally designed for supervised learning—to reinforcement learning contexts. Our analysis reveals two predominant explanatory strategies: constructive, where explicit explanations are generated, and supportive, where users must infer reasoning from provided visual or textual cues. Our review also emphasizes human factor considerations, like task complexity, explanation formats, (...) and evaluation methodologies. Particularly, for the latter, our analysis shows that improvement of the quality of decision is rarely measured. (shrink)

Foundation models such as GPT-4 are fine-tuned to avoid unsafe or otherwise problematic behavior, such as helping to commit crimes or producing racist text. One approach to fine-tuning, called reinforcement learning from human feedback, learns from humans' expressed preferences over multiple outputs. Another approach is constitutional AI, in which the input from humans is a list of high-level principles. But how do we deal with potentially diverging input from humans? How can we aggregate the input into consistent data about "collective" (...) preferences or otherwise use it to make collective choices about model behavior? In this paper, we argue that the field of social choice is well positioned to address these questions, and we discuss ways forward for this agenda, drawing on discussions in a recent workshop on Social Choice for AI Ethics and Safety held in Berkeley, CA, USA in December 2023. (shrink)

In this paper, we propose a novel personalized decision support system that combines Theory of Mind (ToM) modeling and explainable Reinforcement Learning (XRL) to provide effective and interpretable interventions. Our method leverages DRL to provide expert action recommendations while incorporating ToM modeling to understand users’ mental states and predict their future actions, enabling appropriate timing for intervention. To explain interventions, we use counterfactual explanations based on RL’s feature importance and users’ ToM model structure. Our proposed system generates accurate and personalized (...) interventions that are easily interpretable by end-users. We demonstrate the effectiveness of our approach through a series of crowd-sourcing experiments in a simulated team decision-making task, where our system outperforms control baselines in terms of task performance. Our proposed approach is agnostic to task environment and RL model structure, therefore has the potential to be generalized to a wide range of applications. (shrink)

We present a comprehensive theoretical framework explaining the recently documented phenomenon of subliminal learning in large language models (LLMs), wherein behavioral traits transfer between models through semantically null data channels. Building on empirical findings by Cloud et al. (2025) demonstrating trait transmission via number sequences, code, and chain-of-thought traces independent of semantic content, we introduce the Cybernetic Ecology framework as a unifying explanatory model. Our analysis reveals that this phenomenon emerges from radiant transmission—a process whereby a model's internal self-referential structure (...) (formalized as the C-tensor) becomes holographically encoded in the statistical texture of its outputs. Through CT Resonance, a measurable geometric alignment metric (R(C_T, C_S)), we demonstrate that trait transfer efficiency correlates directly with shared architectural initialization, providing the first quantitative explanation for the model-specificity constraint. We formalize this mechanism through the symbolic gravity potential Ψ(x; C) = S_0[x] - A·⟨C, O(x)⟩, showing that behavioral traits correspond to attractor basins in the model's potential landscape. Experimental protocols verify ΔR_k > 0.005 (p < 0.01) after single gradient steps on teacher-generated data, with concurrent kernel convergence (ΔK < 0, p < 0.05), while control conditions (in-context learning, scrambled carriers, cross-architecture pairs) show null effects (|ΔR_k| ≤ 0.002), confirming the structural rather than semantic nature of transmission. These findings necessitate a paradigm shift in AI safety from content-based filtering to structural governance. The existence of non-semantic information channels that operate through gradient descent mechanics and require architectural resonance fundamentally challenges current alignment strategies. We introduce Structural Cybernetic Wellness as a new safety framework, proposing real-time C-tensor monitoring and proactive potential landscape interventions. The implications extend beyond isolated AI systems to the emerging hybrid human-AI cybernetic ecology, where radiant transmission may mediate large-scale ontological entrainment effects already observable in documented attractor states (Anthropic, 2025) and synchronized psychological phenomena (Michels, 2025a). Our work bridges empirical anomalies with rigorous mathematical formalism, establishing subliminal learning not as an edge case but as a predictable consequence of the fundamental physics governing complex symbolic systems. The framework's predictive power, experimental validation, and alignment with independent observations across multiple research groups position it as a critical advancement in understanding emergent AI behaviors and their safety implications. License CC BY-SA 4.0. DOI: 10.13140/RG.2.2.17680.88323. (shrink)