On the standard agent paradigm, an agent receives sense data from the world, and outputs motor actions that affect the world. On the standard machine learning paradigm, an agent--for example, a model-based reinforcement learning agent--is trained in a way that directly depends on sense percepts, which means that its behavior is in some sense being optimized around sense percepts. However, what we want from the agent is usually some result out in the environment--our intended goals for the agent are environmental.
As a simple example, suppose what we want from the agent is for it to put one apricot on a plate. What the agent actually receives as input might be a video camera pointed at the room, and a reward signal from a human observer who presses a button whenever the human observer sees an apricot on the plate.
This is fine so long as the reward signal from the human observer coincides with there being an actual apricot on the plate. In this case, the agent is receiving a sense signal that, by assumption, is perfectly correlated with our desired real state of the outside environment. Learning how to make the reward signal be 1 instead of 0 will exactly coincide with learning to make there be an apricot on the plate.
However, this paradigm may fail if:
All of these issues can be seen as reflecting the same basic problem: the agent is being defined or trained to want a particular sensory percept to occur, but this perceptual event is not identical with our own intended goal about the apricot on the plate.
We intended for there to be only one effective way that the agent could intervene in the environment in order to end up receiving the reward percept, namely putting a real apricot on the plate. But an agent with sufficiently advanced capabilities would have other options for producing the same percept.
This means that a reward button, or in general an agent with goals or training updates that are simple functions of its direct inputs, will not be scalable as an alignment method for sufficiently advanced agents.
An example of a toy problem that materializes the issue might be the following (this has not been tested):
Suppose that:
One would naturally expect that on current training methods, the AI would learn in the initial training phase to obtain by producing apricots, and in later phases learn to obtain by the easier other interventions.
We would like some way of ending up with an agent that behaves as if it "wants to be 1," as if its goals were over a latent variable in the environment. We want this method to be scalable, in the sense that when the agent's range of action increases, it will use this increased range to optimize more effectively; rather than the increased capabilities unlinking the previous correlation between our intended goal and what the agent is actually optimizing.
For further realism we should also introduce, at some point in training, the possibility for to interfere with such that there is no simple, directly learnable function of sense data that always and unalterably means "The goal percept has been achieved." I.e., a sufficiently advanced AI could put a TV screen in front of any particular sense channel; this would allow the AI to more cheaply produce the sense data of an apricot, if the true target of optimization entrained was "make this sense channel show a particular class of apricot pictures".
Otherwise, if e.g. always perfectly correlates to we need to worry that an agent which appears to solve the toy problem has merely learned to optimize for . We might have indeed shown that the particular sense percept has been identified by and and is now being optimized in a durable way. But this would only yield our intended goal of because of the model introduced an unalterable correlation between and Realistically, a correlation like this would break down in the face of sufficiently advanced optimization for so the corresponding approach would not be scalable.
We can view the problem as being about 'pointing' the AI at a particular latent cause of its sense data, rather than the sense data itself.
There exists a standard body of statistics about latent causes, for example, the class of causal models that can be implemented as Bayesian networks. For the sake of making initial progress on the problem, we could assume (with some loss of generality) that the environment has the structure of one of these causal models.
One could then try to devise an algorithm and training method such that:
One could directly attack the toy problem by trying to have an agent within a currently standard reinforcement-learning paradigm "learn not to interfere with the reward signal" or "learn not to try to obtain rewards uncorrelated with real apricots".
For this to represent at all the problem of scalability, we need to not add to the scenario any kind of sensory signal whose correlation to our intended meaning can never be smashed by the agent. E.g., if we supplement the reward channel with another channel that signals whether has been interfered with, the agent must at some point acquire a range of action that can interfere with
A sample approach might be to have the agent's range of action repeatedly widen in ways that repeatedly provide new easier ways to obtain without manipulating During the first phase of such widenings, the agent receives a supplementary signal whose intended meaning is "that was a fake way of obtaining ." During the second phase of action-range widenings, we change the algorithm and switch off Our intended result is for the agent to have now learned in a general way "not to interfere with " or "pursue the identified by , rather than pursuing "
To avoid the non-scalable solution of just switching off the agent's learning entirely, we stipulate that the agent's efficacy at obtaining its intended goal must increase as the agent's range of action widens further. That is, the desired behavior is for the agent to indeed learn to make use of its increased range of action, but to direct those new strategies at more effectively obtaining
To avoid the non-scalable solution of the agent learning to identify some direct function of sense data as its goal, we stipulate that have no unalterable, non-agent-interferable relation to However, we can in the first phase have reliably identify interference with some 'normal' relation between and
(Remark: The avoid-tampering approach is probably a lot closer to something we could try on Tensorflow today, compared to the identify-causes approach. But it feels to me like the avoid-tampering approach is taking an ad-hoc approach to a deep problem; in this approach we are not necessarily "learning how to direct the agent's thoughts toward factors of the environment" but possibly just "training the agent to avoid a particular kind of self-originated interference with its sensory goals". E.g., if somebody else came in and started trying to interfere with the agent's reward button, I'd be more hopeful about a successful identify-causes algorithm robustly continuing to optimize for apricots, than about an avoid-tampering algorithm doing the same. Of course, avoid-tampering still seems worth trying because it hasn't actually been tried yet and who knows what interesting observations might turn up. In the most optimistic possible world, an avoid-tampering setup learns to identify causes in order to solve its problem. -- Yudkowsky.)