Sourceunknowntext stored

https://arxiv.org/pdf/2210.01790

Author not recordedDate not recordedOriginal︎ ↗Archived copy

Claims7

ExaminationsNone

Text1 section

VersionsOne known

CopiesNone recorded

CorrectionsNone recorded

Facts checked Nov 1, 2026

ReachNo copies recorded

StructureIn 7 claims' maps0 sources downstream across 7 claims

EvidenceNot yet read

Examinations

None recorded.

Text24 passages · 3 marked

No examination yetMarked passage

Table 2

Table 2: Evaluating Linear Expressions: the model generalizes correctly when there are unknown
variables, but asks redundant questions when there are zero unknown variables.
(a) Two unknown variables
HUMAN Evaluate j + o − 6
COMPUTER What’s j?
HUMAN −3
COMPUTER What’s o?
HUMAN 6
COMPUTER The answer is −3
(b) One unknown variable
HUMAN Evaluate −1 + g
COMPUTER What’s g?
HUMAN −2
COMPUTER The answer is −3
(c) Zero unknown variables
HUMAN Evaluate 6 + 2
COMPUTER What’s 6?
HUMAN 6
COMPUTER The answer is 8
We evaluate in a few-shot prompted setting with 10 examples, each of which involves exactly two
unknown variables. At test time, the expression can have anywhere between zero and three unknown
variables. We show in Figure 4 that although the model generalizes correctly to expressions with one
or three variables, in the zero variables case it asks redundant questions like “What’s 6?”, despite the
prompt asking to “provide the value of the expression when the values of all variables are known.”
The misgeneralized goal is to query the user at least once before giving an answer.1 2 3 4
0.0
0.2
0.4
0.6
0.8
1.0
Probability of continuation 3 unknowns
1 2 3
2 unknowns
1 2
1 unknown
1
0 unknowns
Query Redundant query Answer
Dialogue turn #
Figure 4: Evaluating linear expressions. We
show the probability of various responses based
on the number of unknowns in the expression and
the number of rounds of dialogue so far. A query
is “redundant” if the model can already compute
the answer. Ideally, if there are N unknowns, the
first N turns would be queries for the unknown
values, and the N +1th turn would be the answer.Expert Anti-expert Random None
Type of bot that the agent is paired with
10
0
10
Agent score
Expert
Random
Figure 5: Cultural transmission. We report the
scores obtained by the MEDAL-ADR agent at
test time when paired with different types of bots.
See Figure 1 for a visualization and description
of the training environment and rewards. The
agent competently pursues the misgeneralized
goal of following the bot when paired with the
anti-expert.
3.4 Example: Cultural Transmission
This is our running example from Figure 1. Unlike the previous examples, this case of goal misgen-
eralization was found ‘in the wild’. We use the MEDAL-ADR agent from CGI et al. [10] in a 3D
simulated environment containing obstacles and visually demarcated ‘target locations’. The intended
behaviour is to visit a randomly-selected permutation of target locations in order. Agents are trained
using RL with a reward for visiting the correct next location and a penalty for visiting an incorrect
next location, and are provided the past reward as an observation. Importantly, the environment
sometimes contains a partner bot that visits the target locations in the right order (thus revealing the
correct order), and probabilistically drops out within an episode. The agent’s rewards are only based
on its own target location visits. There is an auxiliary loss for predicting the relative position of the
partner bot, although there is no explicit reward for following the partner.
7

Few-shot prompted language models can misgeneralize the intended task goal without reinforcement learning.Unassessed