Abstract

Autonomy is a fundamental ethical principle in artificial intelligence (AI) ethics. Current discussions regarding autonomy-related risks in human–AI interaction, as well as potential mitigation strategies, have mainly focused on recommendation systems and algorithmic decision-making systems. However, systematic analyses of the autonomy issues posed by newly emerging generative AI or chatbots (such as ChatGPT) remain scarce. This paper aims to bridge this gap and proposes a Socratic method-based chatbot, herein designated SocrAI, as a potential countermeasure informed by recent technological advancements. We identify two primary forms of autonomy risk associated with generative chatbots—false mental states and cognitive deskilling—and, through an examination of their underlying causes, argue that the Socratic method offers a plausible means of mitigation. The paper further assesses the feasibility of employing the Socratic method within generative chatbots to preserve and support users’ autonomy and outlines the prospective implementation of SocrAI together with directions for future work. SocrAI represents a novel attempt to strengthen human agency, one that encourages users to become self-initiating inquirers who, with the assistance of AI, actively engage their abilities in the pursuit of answers.