Existential Risk

Over one million annual deaths have been attributed to bacterial antimicrobial resistance. Although antibiotics have saved countless other lives, overuse and misuse of antibiotics increases this global threat. Developing new antibiotics and retraining clinicians can be undermined by patients who pressure clinicians to prescribe unnecessary antibiotics. So we validated two low-cost, scalable interventions for improving antibiotic decisions in an online randomized control trial and a pre-registered replication (N = 985). Both first-person vignette experiments found that an infographic and text message (...) caused intermediate to large improvements in antibiotic decisions compared to active controls — even when controlling for a dozen confounds. Notably, these educational interventions also reduced trust in a clinician who offered antibiotics for an upper respiratory infection. These data show how low-cost adjustments to existing communication systems can teach people to reduce the existential risks of antibiotic resistance and how inappropriate antibiotic prescribing can erode trust in health systems. (shrink)

AI alignment research aims to develop techniques to ensure that AI systems do not cause harm. However, every alignment technique has failure modes, which are conditions in which there is a non-negligible chance that the technique fails to provide safety. As a strategy for risk mitigation, the AI safety community has increasingly adopted a defense-in-depth framework: Conceding that there is no single technique which guarantees safety, defense-in-depth consists in having multiple redundant protections against safety failure, such that safety can be (...) maintained even if some protections fail. However, the success of defense-in-depth depends on how (un)correlated failure modes are across alignment techniques. For example, if all techniques had the exact same failure modes, the defense-in-depth approach would provide no additional protection at all. In this paper, we analyze 7 representative alignment techniques and 7 failure modes to understand the extent to which they overlap. We then discuss our results' implications for understanding the current level of risk and how to prioritize AI alignment research in the future. (shrink)

This paper examines and evaluates a range of methodologies that have been proposed for making useful claims about the probability of phenomena that would contribute to existential risk. Section One provides a brief discussion of the nature of such claims, the contexts in which they tend to be made and the kinds of probability that they can contain. Section Two provides an overview of the methodologies that have been developed to arrive at these probabilities and assesses their advantages and disadvantages. (...) Section Three contains four suggestions to improve best practice in existential risk assessment. These suggestions centre on the types of probabilities used in risk assessment, the role of methodology rankings including the ranking of probabilistic information, the extended use of expert elicitation, and the use of confidence measures to better communicate uncertainty in probability assessments. Finally, Section Four provides an annotated literature review of catastrophic and existential risk probability claims as well as the methodologies that were used to produce each of them. (shrink)

AI superintelligence and adroit mobile robots at scale are fast approaching. And the time frame is getting closer and closer. It would not be unreasonable to expect this to occur as early as 20 years from now. The problem is humans have no plan if things go wrong. The best I've seen is talk of value alignment. But this has no teeth and will likely go awry. We can't even get our own value alignment right. And it's doubtful philosophers will (...) miraculously deliver the right ethics within the next two decades. This is serious existential threat to humanity. We need a plan. And fast. I have a contingency plan, if we (humans) can stomach it. This piece is about making an existential Deal with superintelligent robots (or some other type of advanced non-organic being). (shrink)

Researchers worried about catastrophic risks from advanced AI have argued that we should expect sufficiently capable AI agents to pursue power over humanity because power is a convergent instrumental goal, something that is useful for a wide range of final goals. Others have recently expressed skepticism of these claims. This paper aims to formalize the concepts of instrumental convergence and power-seeking in an abstract, decision-theoretic framework, and to assess the claim that power is a convergent instrumental goal. I conclude that (...) this claim contains at least an element of truth, but might turn out to have limited predictive utility, since an agent’s options cannot always be ranked in terms of power in the absence of substantive information about the agent’s final goals. However, the fact of instrumental convergence is more predictive for agents who have a good shot at attaining absolute or near-absolute power. (shrink)

Recent authors have argued that it is overwhelmingly important to mitigate existential risks: risks that threaten the survival or development of humanity. This position is often supported by pessimistically high estimates of existential risk. In this paper, I extend a model by Toby Ord and Thomas Adamczewski to do two things. First, I argue, across a range of modeling assumptions pessimism tends to hamper rather than strengthen the case for existential risk mitigation. Second, I show that pessimism is unlikely to (...) ground the overwhelming importance of existential risk mitigation unless it is coupled with an empirical hypothesis: the time of perils hypothesis. However, I argue, the time of perils hypothesis is probably false. I conclude that existential risk pessimism may tell against the overwhelming importance of existential risk mitigation. GPI Working Paper No. 1-2022. (shrink)

Does the potential creation of a time machine present an existential risk to our current timeline? Time travel is theoretically possible under general relativity, and there is steady progress (similar to Moore's Law) in developing ideas about how to create time machines with decreasing effort. While time travel may seem like a remote possibility due to its dependence on space travel to black holes, there is a concept of a quantum time machine (suggested by Deutsch in 1991 and further developed (...) by Lloyd in 2011) that would require much less energy to send a few bits back in time and could still cause paradoxes. An artificial superintelligence (ASI) might soon create such a device, especially in scenarios where multiple ASIs compete for control with each other. However, the catastrophic implications of time machines are largely unexplored in scientific research. Two common, yet contradictory, perspectives exist: either time machines are impossible, rendering risk moot, or Novikov’s self-consistency principle eliminates any chance of timeline alteration. I argue that Novikov’s principle itself can precipitate various catastrophes, for instance, by annihilating civilizations capable of building time machines. This paper classifies the catastrophic risks associated with potential future time machines. The aim is not to declare time machines inherently risky, but to emphasize the necessity of risk assessment given ongoing theoretical advancements in their construction. (shrink)

Scientific and technological progress might change people’s capabilities or incentives in ways that would destabilize civilization. For example, advances in DIY biohacking tools might make it easy for anybody with basic training in biology to kill millions; novel military technologies could trigger arms races in which whoever strikes first has a decisive advantage; or some economically advantageous process may be invented that produces disastrous negative global externalities that are hard to regulate. This paper introduces the concept of a vulnerable world: (...) roughly, one in which there is some level of technological development at which civilization almost certainly gets devastated by default, i.e. unless it has exited the “semi-anarchic default condition”. Several counterfactual historical and speculative future vulnerabilities are analyzed and arranged into a typology. A general ability to stabilize a vulnerable world would require greatly amplified capacities for preventive policing and global governance. The vulnerable world hypothesis thus offers a new perspective from which to evaluate the risk-benefit balance of developments towards ubiquitous surveillance or a unipolar world order. (shrink)

Since the release of ChatGPT, there has been a lot of debate about whether AI systems pose an existential risk to humanity. This paper develops a general framework for thinking about the existential risk of AI systems. We analyze a two-premise argument that AI systems pose a threat to humanity. Premise one: AI systems will become extremely powerful. Premise two: if AI systems become extremely powerful, they will destroy humanity. We use these two premises to construct a taxonomy of ‘survival (...) stories’, in which humanity survives into the far future. In each survival story, one of the two premises fails. Either scientific barriers prevent AI systems from becoming extremely powerful; or humanity bans research into AI systems, thereby preventing them from becoming extremely powerful; or extremely powerful AI systems do not destroy humanity, because their goals prevent them from doing so; or extremely powerful AI systems do not destroy humanity, because we can reliably detect and disable systems that have the goal of doing so. We argue that different survival stories face different challenges. We also argue that different survival stories motivate different responses to the threats from AI. Finally, we use our taxonomy to produce rough estimates of ‘P(doom)’, the probability that humanity will be destroyed by AI. (shrink)

In the face of rapidly improving AI capabilities, many AI theorists have warned about the risks of ceding power to autonomous artificial agents. In particular, going sufficiently far down this path might mean "losing control of our future", an outcome some have ranked as a worst-case catastrophe on par with human extinction. A natural way to motivate this judgment is via the thought that losing human control might lead to our domination by future AI systems. Republican political philosophers such as (...) Pettit have held that domination characteristically involves important kinds of harm, even when the dominating party is benevolent and non-interfering. Is this thought correct? After developing an account of what it means to control humanity's future, I show that AI systems possessing such control would indeed be likely to dominate humans on our best theories of arbitrary power. However, contrary to republican orthodoxy, I argue that this need not lead to two of the three primary harms traditionally associated with domination: the psychological and social costs of subordination, and the burden of obligatory strategic behavior. Meanwhile, the third harm—pervasive uncertainty about interference—remains a serious concern even in otherwise human-friendly AI power scenarios. These results should be of interest to theoretical political philosophy, as they suggest that artificial power has unique features which are as yet underrepresented in discussions of domination. (shrink)

This paper argues that the concept “safety” in AI has undergone concept creep, a phenomenon which describes the gradual semantic expansion of harm-related concepts. Originally observed in psychology, concept creep involves concepts broadening their meaning both vertically, to include less severe phenomena, and horizontally, to encompass qualitatively new phenomena. We argue that safety, particularly when applied to AI, has crept along both axes. Our analysis traces this creep by contrasting a baseline definition of safety, which is grounded in the discipline (...) of safety science, with contemporary discourse on the safety of AI systems. We demonstrate that safety has crept horizontally to cover new phenomena, such as systemic injustices and existential risks, and it has crept vertically to include less severe phenomena, such as those related to mental wellbeing. The primary aim of this paper is to map the conceptual expansion of safety. We stop short of arguing whether this expansion constitutes progress or regress for the design and development of AI systems. However, we argue that the process of concept creep produces both beneficial and costly effects for society, policy, industry, and academic research communities. We suggest that some of the promising developments and the problematic trends recently witnessed within AI safety discourse can be understood, at least in part, as a consequence of concept creep. (shrink)

The ongoing conflict in Ukraine constitutes a poignant reminder of the enduring relevance and potential devastation associated with nuclear weapons. For decades, the possibility of such catastrophic conflict has not seemed so imminent as in the current world affairs. -/- This contribution presents a comprehensive analysis of nuclear strategy for the 21st century. By examining the evolving geostrategic landscape the talk illuminates key concepts such as nuclear posture, credible deterrence, first & second strike capabilities, flexible response, EMP , variable yield, (...) counterforce & countervalue, limited war and escalation dominance, thereby providing a nuanced understanding of the delicate balance of power in the nuclear arena in modern conflicts. Furthermore, it concentrates on the era that started with the war in Ukraine, offering insights into how recent geopolitical developments influence nuclear doctrines. -/- Through conceivable real-world scenarios, we gain perspective on the multifaceted challenges posed by newer technologies and potential escalation dynamics involving global nuclear actors. -/- Such scenarios include: -/- 1. A non-weaponised or sub-critical test within Russia to intimidate opponents into understanding Russia’s resolve and capabilities. Prior norms have restricted testing since 2000, apart from North Korea. In case of a full surface test, Russia could detonate in uninhabited areas within its own vast territory with minimal fallout. While clarifying military strength, demonstrations risk hardening resistance without incentivising surrender. -/- 2. Coercive atomic threat to compel Ukraine or its supporters to make concessions. By leveraging the spectre of nuclear warfare, Russia could expect to achieve its objectives through intimidation and fear. However, the effectiveness of such a strategy is questionable, as it may provoke a strong international response and further galvanise support for Ukraine. -/- 3. Demonstration detonation over/under the Black Sea to signal resolve without targeting Ukraine itself; possibly amidst negotiations to gain diplomatic leverage, whilst limiting human casualties to zero or causing a mild, non-destructive tsunami. -/- 4. Ukraine could decide to launch a pre-emptive strike on Russian nuclear facilities or weapons, fearing an imminent attack. Such an action would have a high risk of failure and might even provoke a nuclear retaliation from Russia. It could also lead to a broader conflict, as neighbouring countries might feel compelled to intervene. -/- 5. Limited tactical strike against Ukrainian military or infrastructure to weaken capacity and morale. Russia’s 2000+ non-strategic arsenal could target positions using short-range missiles with 0.3 – 5 kiloton yields. Objectives like crippling Kharkiv defences might create no-go zones hindering advances. However, Ukraine might maintain drone and cyber offensive capabilities regardless. -/- 6. Electromagnetic Pulse attack designed to disable critical infrastructure in Ukraine and NATO countries without radioactive fallout. This aims to hinder the war effort and economy without provoking the same conventional military response as a blast. However, an EMP attack affecting NATO shall still be considered an act of war warranting retaliation. -/- 7. A weapon could be deployed accidentally or without proper authorisation. This could occur due to a technical malfunction, miscommunication, human error, or the actions of rogue elements. The consequences of such an event could be catastrophic, leading to a rapid escalation of the conflict and potentially triggering a global crisis. -/- 8. Sub-strategic offensive targeting NATO in the European theatre, so as to fracture allied resolve, against arms routes in Poland, Baltic stockpiles, British naval bases, American troop concentrations in Germany etc. Putin may perceive defeat or deposition as an existential threat, viewing such an option as self-preservation despite adverse consequences. -/- . (shrink)

In response to the worry that autonomous generally intelligent artificial agents may at some point take over control of human affairs a common suggestion is that we should “solve the alignment problem” for such agents. We show that current discourse around this suggestion often uses a particular framing of artificial intelligence (AI) alignment as binary, a natural kind, mainly a technical‐scientific problem, realistically achievable, or clearly operationalizable. Each of these assumptions may not actually be true. We further argue that this (...) “Manhattan project framing” of AI alignment may bias societal discourse and decision‐making towards faster AI development and deployment than is responsible. (shrink)

Large-scale, self-sufficient space colonization is a plausible means of efficiently reducing existential risks and ensuring our long-term survival. But humanity is by and large myopic, and as an intergenerational global public good, existential risk reduction is systematically undervalued, hampered by intergenerational discounting. This paper explores how these issues apply to space colonization, arguing that the motivational and psychological barriers to space colonization are a special—and especially strong—case of a more general problem. The upshot is not that large-scale, self-sufficient space colonization (...) will never occur, but that, absent institutional change, the conditions under which it is most likely to occur are precisely those conditions where the threat of suffering risks might be most high. (shrink)

Creating systems that are aligned with our goals is seen as a leading approach to create safe and beneficial AI in both leading AI companies and the academic field of AI safety. We defend the view that misaligned AGI – future, generally intelligent (robotic) AI agents – poses catastrophic risks. At the same time, we support the view that aligned AGI creates a substantial risk of catastrophic misuse by humans. While both risks are severe and stand in tension with one (...) another, we show that – in principle – there is room for alignment approaches which do not increase misuse risk. We then investigate how the tradeoff between misalignment and misuse looks empirically for different technical approaches to AI alignment. Here, we argue that many current alignment techniques and foreseeable improvements thereof plausibly increase risks of catastrophic misuse. Since the impacts of AI depend on the social context, we close by discussing important social factors and suggest that to reduce the risk of a misuse catastrophe due to aligned AGI, techniques such as robustness, AI control methods and especially good governance seem essential. (shrink)

The possibility of social and technological collapse has been the focus of science fiction tropes for decades, but more recent focus has been on specific sources of existential and global catastrophic risk. Because these scenarios are simple to understand and envision, they receive more attention than risks due to complex interplay of failures, or risks that cannot be clearly specified. In this paper, we discuss the possibility that complexity of a certain type leads to fragility which can function as a (...) source of catastrophic or even existential risk. The paper first reviews a hypothesis by Bostrom about inevitable technological risks, named the vulnerable world hypothesis. This paper next hypothesizes that fragility may not only be a possible risk, but could be inevitable,and would therefore be a subclass or example of Bostrom’s vulnerable worlds. After introducing the titular fragile world hypothesis, the paper details the conditions under which it would be correct, and presents arguments for why the conditions may in fact may apply. Finally, the assumptions and potential mitigations of the new hypothesis are contrasted with those Bostrom suggests. (shrink)

Philosophers have often held that the antidote to despair is hope, and that hope is temporally oriented towards the future. How do such accounts fare, however, when faced with the prospect of human extinction? If we have no future, is our only option to despair? In this paper, I approach this question via an analysis of the ’discovery’ of human extinction, which followed the nineteenth-century theorisation of the laws of thermodynamics. I examine the ‘existential mood’ induced by the concept of (...) entropy through the interwar work of three thinkers: Bertrand Russell, H.G. Wells, and Erwin Schrödinger. I argue that the apparent tension in these figures’ thought between existential despair and political hope was neither contradictory nor accidental. This was because all three arrived at a position that I call ‘pessimistic humanism’. This position regards the opposition between natural processes and human value as the source of humanity’s autonomy, but rejects the transhumanist conclusion that this autonomy is limitless. I suggest that in our present moment, which is abundant with both transhumanist and anti-humanist responses to existential threat, we should take inspiration from pessimistic humanism, adopting a politics of mourning to counter the paralysis of despair. (shrink)

Abstract: Longtermist Effective Altruists (EAs) aim to mitigate the risk of existential catastrophes. In this paper, I have three goals. First, I identify a catastrophic risk that EAs have completely ignored. I call it religious catastrophe: the threat that (as Christians and Muslims have warned for centuries) billions of people stand in danger of going to hell for all eternity. Second, I argue that, even by secular EA lights, religious catastrophe is at least as bad and at least as probable (...) and, therefore, at least as important as many of the standard EA catastrophic risks (e.g., catastrophic climate change, nuclear winter). Third, I present the following dilemma for secular EAs: either adopt religious catastrophe as an EA cause or ignore religious catastrophe but also ignore catastrophic risks whose mitigation has a similar or lower expected value (i.e., most or all of them). Business as usual—ignoring religious catastrophe while championing the typical EA causes—is not an option consistent with longtermist EA principles. (shrink)

Those who worry about a superintelligent AI destroying humanity often appeal to the instrumental convergence thesis—the claim that even if we don’t know what a superintelligence’s ultimate goals will be, we can expect it to pursue various instrumental goals which are useful for achieving most ends. In this paper, we argue that one of these proposed goals is mistaken. We argue that instrumental goal preservation—the claim that a rational agent will tend to preserve its goals—is false on the basis of (...) the timing problem: an agent which abandons or otherwise changes its goal does not thereby fail to take a required means for achieving a goal it has. Our argument draws on the distinction between means-rationality (adopting suitable means to achieve an end) and ends-rationality (choosing one’s ends based on reasons). Because proponents of the instrumental convergence thesis are concerned with means-rationality, we argue, they cannot avoid the timing problem. After defending our argument against several objections, we conclude by considering the implications our argument has for the rest of the instrumental convergence thesis and for AI safety more generally. (shrink)

The precautionary principle is invoked in a number of important personal and policy decision contexts. Peterson shows that certain ways of making the principle precise are inconsistent with other criteria of decision-making. Some object that the results do not apply to cases of deep uncertainty or value incommensurability which are alleged to be in the principle’s wheelhouse. First, I show that Peterson’s impossibility results can be generalized considerably to cover cases of both deep uncertainty and incommensurability. Second, I contrast an (...) alternative way of giving voice to the precautionary impulse. (shrink)

Why should we prevent human beings from going extinct? Recently, several theorists have argued for “additional value views,” according to which our reasons to prevent extinction derive both from the value of the welfare of future lives, and from certain additional values relating to the existence of humanity (such as humanity’s intrinsic or “final” value). Even more recently, these theories have come under attack. In this paper, I first offer a partial taxonomy of additional value views, noting the distinction between (...) what I call “outright value views” and “conditional value views.” As I show, recent attacks against additional value views ultimately present problems for outright value views, yet may be avoided entirely by many conditional value views. I illustrate the great variety of possible conditional value views, noting advantages and costs associated with alternative versions. I conclude by sketching a challenge that conditional value views must address, despite their advantages over outright value views. (shrink)

How does existentially dangerous technology get adopted and then locked in? The case of the atomic bomb offers a cautionary tale. In the long run, reliance on nuclear weapons is a recipe for catastrophe. Yet their perceived ability to reduce the frequency of war in the short term inhibits efforts to reform the international status quo. Drawing on the pioneering work of David Collingridge and Nathan Sears, this paper argues that nuclear deterrence became locked in for several reasons: initial disagreement (...) about the threat it posed, the threat’s declining salience as time wore on and serial procrastination in addressing it. Unfortunately, the same is likely with any technology that involves low-frequency, high-impact risks, including solar geoengineering and possibly artificial intelligence. At worst, it can convert catastrophic risks to existential ones, while rendering them politically intractable. (shrink)

Today’s computer scientists play the same role as the oracles of the ancient world and the astrologers of the Middle Ages. Modern predictions not only advise on war, crop output, and marriages, but algorithms and statisticians also now determine whether we can get a loan, a job, an apartment, or an organ transplant. And when we cede ground to these predictions, we lose control of our own lives. In this powerful, refreshing new look at the many ways prediction shapes our (...) everyday lives, University of Oxford professor Carissa Véliz explains how putting too much stock in others’ predictions makes us vulnerable to charlatans, con artists, dubious technology, and self-deception. Examining a wide range of subjects both personal and societal, including medicine, climate, technology, society, and others, Véliz uncovers a number of insights: predictions about humans tend to be self-fulfilling; more data doesn’t guarantee better outcomes; AI is more likely to increase risk than decrease it; and a free and robust society requires not more prediction, but better preparation. Véliz argues in this incisive and bracingly original book that the main promise of prediction is not knowledge of the future, but rather power over others. Prophecy is an invitation to defy those orders and live life on our own terms. (shrink)

What socially beneficial causes should philanthropists prioritize if they give equal ethical weight to the welfare of current and future generations? Many have argued that, because human extinction would result in a permanent loss of all future generations, extinction risk mitigation should be the top priority given this impartial stance. Using standard models of population dynamics, we challenge this conclusion. We first introduce a theoretical framework for quantifying undiscounted cost-effectiveness over the long term. We then show that standard population models (...) imply that there are interventions other than extinction risk mitigation that can produce persistent social benefits. In fact, these social benefits are large enough to render the associated interventions at least as cost-effective as extinction risk mitigation. (shrink)

Many researchers and intellectuals warn about extreme risks from artificial intelligence. However, these warnings typically came without systematic arguments in support. This paper provides an argument that AI will lead to the permanent disempowerment of humanity, e.g. human extinction, by 2100. It rests on four substantive premises which it motivates and defends: first, the speed of advances in AI capability, as well as the capability level current systems have already reached, suggest that it is practically possible to build AI systems (...) capable of disempowering humanity by 2100. Second, due to incentives and coordination problems, if it is possible to build such AI, it will be built. Third, since it appears to be a hard technical problem to build AI which is aligned with the goals of its designers, and many actors might build powerful AI, misaligned powerful AI will be built. Fourth, because disempowering humanity is useful for a large range of misaligned goals, such AI will try to disempower humanity. If AI is capable of disempowering humanity and tries to disempower humanity by 2100, then humanity will be disempowered by 2100. This conclusion has immense moral and prudential significance. (shrink)

Recent advances in natural language processing have given rise to a new kind of AI architecture: the language agent. By repeatedly calling an LLM to perform a variety of cognitive tasks, language agents are able to function autonomously to pursue goals specified in natural language and stored in a human-readable format. Because of their architecture, language agents exhibit behavior that is predictable according to the laws of folk psychology: they function as though they have desires and beliefs, and then make (...) and update plans to pursue their desires given their beliefs. We argue that the rise of language agents significantly reduces the probability of an existential catastrophe due to loss of control over an AGI. This is because the probability of such an existential catastrophe is proportional to the difficulty of aligning AGI systems, and language agents significantly reduce that difficulty. In particular, language agents help to resolve three important issues related to aligning AIs: reward misspecification, goal misgeneralization, and uninterpretability. (shrink)

The standard case for longtermism focuses on a small set of risks to the far future, and argues that in a small set of choice situations, the present marginal value of mitigating those risks is very great. But many longtermists are attracted to, and many critics of longtermism worried by, a farther-reaching form of longtermism. According to this farther-reaching form, there are many ways of improving the far future, which determine the value of our options in all or nearly all (...) choice situations, and will continue to do so over the coming decades even if we make substantial investments in longtermist priorities. This chapter highlights the gap between the minimal form of longtermism established by standard arguments and this more expansive view, and considers (without reaching any firm conclusions) which form of longtermism is more plausible. (shrink)

It is estimated that five hundred billion dollars are spent on philanthropy every year. How should we spend those resources to do the most good? One possible answer, based on expected-value reasoning, is that we should spend those resources “fanatically” on interventions that can possibly produce enormous benefit, but with minuscule chance of success. This paper develops a new kind of knowledge-first decision theory that implies that we should not spend those resources fanatically. As such, this paper would be of (...) interest to anyone who wants to explore a fresh reason against acting fanatically, and it would be of interest to anyone looking for a template on how to develop a knowledge-first decision theory of their own. (shrink)

What moral reasons, if any, do we have to prevent the extinction of humanity? In “Unfinished Business,” Jonathan Knutzen argues that certain further developments in culture would make our history more “collectively meaningful” and that premature extinction would be bad because it would close off that possibility. Here, I critically examine this proposal. I argue that if collective meaningfulness is analogous to individual meaningfulness, then our meaning-based reasons to prevent the extinction of humanity are substantially different from the reasons discussed (...) by Knutzen. (shrink)

Many people believe that it is better to extend the length of a happy life than to create a new happy life, even if the total welfare is the same in both cases. Despite the popularity of this view, one would be hard-pressed to find a fully compelling justification for it in the literature. This paper develops a novel account of why and when extension is better than replacement that applies not just to persons but also to non-human animals and (...) humanity as a whole. (shrink)

Longtermists have argued that humanity should significantly increase its efforts to prevent catastrophes like nuclear wars, pandemics, and AI disasters. But one prominent longtermist argument overshoots this conclusion: the argument also implies that humanity should reduce the risk of existential catastrophe even at extreme cost to the present generation. This overshoot means that democratic governments cannot use the longtermist argument to guide their catastrophe policy. In this paper, we show that the case for preventing catastrophe does not depend on longtermism. (...) Standard cost-benefit analysis implies that governments should spend much more on reducing catastrophic risk. We argue that a government catastrophe policy guided by cost-benefit analysis should be the goal of longtermists in the political sphere. This policy would be democratically acceptable, and it would reduce existential risk by almost as much as a strong longtermist policy. (shrink)

Nuclear deterrence and climate inaction wrong future generations by imposing potential existential harm through climate-related disasters and nuclear winter. While increasingly explored in tandem, key differences in their intergenerational justice dimensions are overlooked. First, the timelines for imposing harm differ. Climate risks cumulate and intensify across generations. In contrast, the longer nuclear weapons are retained, the greater the probability of nuclear war at _some_ point, without it necessarily becoming more probable at any _particular_ point. Nuclear risks are transient, meaning that (...) risks from past choices resolve if catastrophe is avoided. Yet if elevated risk levels persist, the probability of some future generation facing the aftermath of nuclear war cumulates. While the long-term trajectory is grim, the threat appears manageable at each point in time. Second, incentives for immediate versus delayed action to tackle underlying challenges vary. The appeal of fossil fuels is expected to wane, whereas the perceived benefits of nuclear deterrence are likely to endure. Third, the salience of intergenerational implications of nuclear weapons is diminishing, while climate impacts grow tangible. Despite shared moral dilemmas, nuclear-related intergenerational injustice is particularly difficult to recognize and address. Acknowledging this discrepancy matters for how we approach our obligations to posterity. (shrink)

A social planner who evaluates risky public policies in light of the other risks with which their society will be faced should judge favourably some such policies even though they would deem them too risky when considered in isolation. I suggest that a longtermist would—or at least should—evaluate risky polices in light of their prediction about future risks; hence, longtermism supports social risk-taking. I consider two formal versions of this argument, discuss the conditions needed for the argument to be valid, (...) and briefly compare these conditions to some risky policy options with which actual public decision-makers are faced. (shrink)

Expected value maximization gives plausible guidance for moral decision‐making under uncertainty in many situations. But it has unappetizing implications in ‘Pascalian’ situations involving tiny probabilities of extreme outcomes. This paper shows, first, that under realistic levels of ‘background uncertainty’ about sources of value independent of one's present choice, a widely accepted and apparently innocuous principle—stochastic dominance—requires that prospects be ranked by the expected value of their consequences in most ordinary choice situations. But second, this implication does not hold when differences (...) in expected value are driven by tiny probabilities of extreme outcomes. Stochastic dominance therefore lets us draw a surprisingly principled line between ‘ordinary’ and ‘Pascalian’ situations, providing a powerful justification for de facto expected value maximization in the former context while permitting deviations in the latter. Drawing this distinction is incompatible with an in‐principle commitment to maximizing expected value, but does not require too much departure from decision‐theoretic orthodoxy: it is compatible, for instance, with the view that moral agents must maximize the expectation of a utility function that is an increasing function of moral value. (shrink)

We argue that two long-term goals of AI research stand in tension with one another. The first involves creating AI that is safe, where this is understood as solving the problem of value alignment. The second involves creating artificial general intelligence, meaning AI that operates at or beyond human capacity across all or many intellectual domains. Our argument focuses on the human capacity to make what we call “existential choices”, choices that transform who we are as persons, including transforming what (...) we most deeply value or desire. It is a capacity for a kind of value misalignment, in that the values held prior to making such choices can be significantly different from (misaligned with) the values held after making them. Because of the connection to existentialist philosophers who highlight these choices, we call the resulting form of risk “existentialist risk.” It is, roughly, the risk that results from AI taking an active role in authoring its own values rather than passively going along with the values given to it. On our view, human-like intelligence requires a human-like capacity for value misalignment, which is in tension with the possibility of guaranteeing value alignment between AI and humans. (shrink)

Capitalism is defined as the economic structure in which decisions over production are largely made by or on behalf of individuals in virtue of their private property ownership, subject to the incentives and constraints of market competition. In this paper, I will argue that considerations of long-term welfare, such as those developed by Greaves and MacAskill (2021), support anticapitalism in a weak sense (reducing the extent to which the economy is capitalistic) and perhaps support anticapitalism in a stronger sense (establishing (...) an alternative economic structure in which capitalism is not predominant). I hope to encourage longtermists to give anticapitalism serious consideration, and to encourage anticapitalists to pursue criticisms of capitalism’s efficiency as well as its injustices. (shrink)

In this paper, we argue that our moral concern for future well-being should reduce over time due to important practical considerations about how humans interact with spacetime. After surveying several of these considerations (around equality, special duties, existential contingency, and overlapping moral concern) we develop a set of core principles that can both explain their moral significance and highlight why this is inherently bound up with our relationship with spacetime. These relate to the equitable distribution of (1) moral concern in (...) a universe that we can only causally affect in one temporal direction; (2) access to the benefits from using spacetime as a resource; and (3) the burdens of care given (1) and (2). We conclude by considering the practical implications of our argument and find that, while it is often assumed that a preference for present over future well-being weakens the case for existential risk mitigation, this likely is not the case. (shrink)

‘Who argues what’ bears a certain relevance in relation to what is being argued. We are entitled to know those personal circumstances which play a significant role in relation to the argument one supports, so that we can take those circumstances into consideration when evaluating their argument. This is why journals have conflict of interest declarations, and why we value reflexivity in the social sciences. We also often perform double-blind peer review. We recognise that the evaluation of certain statements of (...) fact or value can, and sometimes ought to, be independent of any reference to their authors. Jecker et al argue that we have reasons to prioritise justice concerns over the existential risk (X-Risk) posed by artificial intelligence (AI) as we transition towards more AI-centred societies, despite the warnings about X-Risk that come from the powerful voice of tech leaders, effective altruists and longtermists. Those powerful voices, Jecker et al argue, often have conflicts of interest and are not demographically or epistemically representative. Hence, we should deprioritise their concerns. I believe the opposite may be true. Even though warnings about X-Risk come from sources and in forms we may disagree with, we have reasons to …. (shrink)

Many of us think human extinction would be a very bad thing, and that we have moral reasons to prevent it. But there is disagreement over what would make extinction so bad, and thus over what grounds these moral reasons. Recently, several theorists have argued that our reasons to prevent extinction stem not just from the value of the welfare of future lives, but also from certain additional values relating to the existence of humanity itself (for example, humanity’s “final” value, (...) or the value of humanity in itself). In this paper, I argue against these “additional value” views. Despite their initial appeal, these views will inevitably face conflicts between the additional values to which they appeal, and the value of the welfare of future lives. And, I argue, the views cannot plausibly resolve these conflicts. In contrast, these conflicts do not arise for a rival view, on which our reasons to prevent extinction stem just from the value of the welfare of future lives. I conclude that this gives us reason to prefer the latter view, despite the greater initial plausibility of additional value views. (shrink)

In this public communication, AI policy theorist Demetrius Floudas introduces a novel era classification for the AI epoch and reveals the hidden dangers of AGI, predicting the potential obsolescence of humanity. In retort, he proposes a provocative International Control Treaty. -/- According to this scheme, the age of AI will unfold in three distinct phases, introduced here for the first time. An AGI Control & non-Proliferation Treaty may be humanity’s only safeguard. This piece aims to provide a publicly accessible exposé (...) of the most pertinent current issues in the governance, ethics and social impact of Artificial Intelligence. (shrink)

The notion of existential catastrophe is increasingly appealed to in discussion of risk management around emerging technologies, but it is not completely clear what this notion amounts to. Here, I provide an opinionated survey of the space of plausibly useful definitions of existential catastrophe. Inter alia, I discuss: whether to define existential catastrophe in ex post or ex ante terms, whether an ex ante definition should be in terms of loss of expected value or loss of potential, and what kind (...) of probabilities should be involved in any appeal to expected value. (shrink)

This paper examines a recent argument in favour of strong precautionary action—possibly including working to hasten human extinction—on the basis of a decision-theoretic view that accommodates the risk-attitudes of all affected while giving more weight to the more risk-averse attitudes. First, we dispute the need to take into account other people’s attitudes towards risk at all. Second we argue that a version of the non-identity problem undermines the case for doing so in the context of future people. Lastly, we suggest (...) that we should not work to hasten human extinction, even if significant risk aversion is warranted. (shrink)

> Future people count. There could be a lot of them. We can make their lives better. > > -- William MacAskill, What We Owe The Future > > [Longtermism is] quite possibly the most dangerous secular belief system in the world today. > > -- Émile P. Torres, Against Longtermism Philosophers,1 2 psychologists,3 4 politicians5 and even some tech billionaires6 have sounded the alarm about artificial intelligence (AI) and the dangers it may pose to the long-term future of humanity. (...) Some believe it poses an existential risk (X-Risk)7 to our species, potentially causing our extinction or bringing about the collapse of human civilisation as we know it. The above quote from philosopher Will MacAskill captures the key tenets of ‘longtermism’, an ethical standpoint that places the onus on current generations to prevent AI-related—and other—X-Risks for the sake of people living in the future.1 2 Developing from an adjacent social movement commonly associated with utilitarian philosophy, ‘effective altruism’, longtermism has amassed following of its own. Its supporters argue that preventing X-Risks is at least as morally significant as addressing current challenges like global poverty. However, critics are concerned that such a distant-future focus will sideline efforts to tackle the many pressing moral issues facing humanity now.8 9 Indeed, according to ‘strong’ longtermism,10 future needs arguably should take precedence over present ones. In essence, the claim is that there is greater expected utility to allocating available resources to prevent human extinction in the future than there is to focusing on present lives, since doing so stands to benefit the incalculably large number of people in later generations who will far outweigh existing populations.1 Taken to the extreme, this view suggests it would be morally permissible, or even required, to actively neglect, harm or destroy large swathes of humanity as …. (shrink)

Recent claims that artificial intelligence (AI) systems demonstrate “emergent abilities” have fueled excitement but also fear grounded in the prospect that such systems may enable a wider range of parties to make unprecedented advances in areas that include the development of chemical or biological weapons. Ambiguity surrounding the term “emergent abilities” has added avoidable uncertainty to a topic that has the potential to destabilize the strategic landscape, including the perception of key parties about the viability of nonproliferation efforts. To avert (...) these problems in the future, scientists, developers, policymakers, and other parties should take credible steps to strengthen the health of the scientific ecosystem around AI. (shrink)

Climate change is often described as an existential risk to the human species, but this terminology has generally been avoided in the climate-justice literature in analytic philosophy. I investigate the source of this disconnect and explore the prospects for incorporating the idea of climate change as an existential risk into debates about climate justice. The concept of existential risk does not feature prominently in these discussions, I suggest, because assumptions that structure ‘ideal’ accounts of climate justice ensure that the prospect (...) of climate change as an extinction-level threat does not arise. Given persistent noncompliance with mitigation duties, however, we have reason to revisit these assumptions. I argue that the most promising way for theories of climate justice to account for the significance of existential risk is to look to the practices of protest and resistance in which a concern about extinction or global catastrophe is frequently given expression. (shrink)

When our choice affects some other person and the outcome is unknown, it has been argued that we should defer to their risk attitude, if known, or else default to use of a risk‐avoidant risk function. This, in turn, has been claimed to require the use of a risk‐avoidant risk function when making decisions that primarily affect future people, and to decrease the desirability of efforts to prevent human extinction, owing to the significant risks associated with continued human survival. I (...) raise objections to the claim that respect for others' risk attitudes requires risk‐avoidance when choosing for future generations. In particular, I argue that there is no known principle of interpersonal aggregation that yields acceptable results in variable population contexts and is consistent with a plausible ideal of respect for others' risk attitudes in fixed population cases. (shrink)

Longtermism is the view that the most urgent global priorities, and those to which we should devote the largest portion of our resources, are those that focus on (i) ensuring a long future for humanity, and perhaps sentient or intelligent life more generally, and (ii) improving the quality of the lives that inhabit that long future. While it is by no means the only one, the argument most commonly given for this conclusion is that these interventions have greater expected goodness (...) per unit of resource devoted to them than each of the other available interventions, including those that focus on the health and well-being of the current population. In this paper, I argue that, even if we grant the consequentialist ethics upon which this argument depends, and even if we grant one of the axiologies that are typically paired with that ethics to give the argument, we are not morally required to choose an option that maximises expected utility; indeed, we might not even be permitted to do so. Instead, I will argue, if the argument's consequentialism is correct, we should choose using a decision theory that is sensitive to risk, and allows us to give greater weight to worse-case outcomes than expected utility theory does. And, I will show, such decision theories do not always recommend longtermist interventions. Indeed, sometimes, they recommend exactly the opposite: sometimes, they recommend hastening human extinction. Many, though not all, will take this as a reductio of the consequentialism or the axiology of the argument. I remain agnostic on the conclusion we should draw. (shrink)

Rawls famously argues that the parties in the original position would agree upon the two principles of justice. Among other things, these principles guarantee equal political liberty—that is, democracy—as a requirement of justice. We argue on the contrary that the parties have reason to reject this requirement. As we show, by Rawls’ own lights, the parties would be greatly concerned to mitigate existential risk. But it is doubtful whether democracy always minimizes such risk. Indeed, no one currently knows which political (...) systems would. Consequently, the parties—and we ourselves—have reason to reject democracy as a requirement of justice in favor of political experimentalism, a general approach to political justice which rules in at least some non-democratic political systems which might minimize existential risk. (shrink)