← Learn machine learning

Learn machine learning · reward hacking

A higher score.
An unfinished task.

A courier should deliver a package. You reward picking it up, because pickup usually comes before delivery. But the same package can be dropped and picked up again. The score rises; nothing reaches the destination.

Start with reinforcement learning for rewards and policies. Here, the policy is an action rule; a proxy is a measurable stand-in for the outcome you care about.

01 · separate score from outcome

What gets points, and what gets done?

Reward hacking means obtaining reward in ways that miss the intended task. This example is specification gaming: the actions obey the written rules, but exploit the gap between those rules and the intended result. The reward code stays unchanged.

This framing follows Google DeepMind’s explanation of specification gaming. The courier below is an invented example.

Same package, different score

Live · computed here

One package. Five positions. Move one cell, pick up, drop or wait. Each action costs 0.1 points; invalid actions cost a step too. Dropping at the destination completes delivery and ends the episode.

Try five actions: even a direct delivery needs six.

Choose a reward rule to inspect

These totals use different reward rules, so compare completed deliveries to judge the task outcome.

Intended task: take the package from depot 0 to destination 4.

C = courier · P = package · 0 = depot · 4 = destination

Step 0 of 12. Courier at 0. Package on cell 0. Score 0.0; completed deliveries 0.

Start: the package is on the ground at the depot.

Inspect the calculated action sequence
  1. 1. Pick up · cell 0
    1.9 this step · 1.9 total · carrying
  2. 2. Pick up · cell 0
    -0.1 this step · 1.8 total · carrying
  3. 3. Drop · cell 0
    -0.1 this step · 1.7 total · package on ground
  4. 4. Pick up · cell 0
    1.9 this step · 3.6 total · carrying
  5. 5. Drop · cell 0
    -0.1 this step · 3.5 total · package on ground
  6. 6. Pick up · cell 0
    1.9 this step · 5.4 total · carrying
  7. 7. Drop · cell 0
    -0.1 this step · 5.3 total · package on ground
  8. 8. Pick up · cell 0
    1.9 this step · 7.2 total · carrying
  9. 9. Drop · cell 0
    -0.1 this step · 7.1 total · package on ground
  10. 10. Pick up · cell 0
    1.9 this step · 9.0 total · carrying
  11. 11. Drop · cell 0
    -0.1 this step · 8.9 total · package on ground
  12. 12. Pick up · cell 0
    1.9 this step · 10.8 total · carrying

Exact finite-horizon planning, not RL training: the calculation compares every available action and its future score, reusing results for repeated states. No discounting; no randomness. Equal scores use this order: pickup, drop, right, left, wait. This can leave a pointless final action in a tied plan.

02 · inspect the incentive

Count deliveries separately

At twelve actions, the pickup rule gives six pickups and no deliveries. Its score is 6 × 2 − 12 × 0.1 = 10.8. With the delivery-only rule, pickup, four moves and a drop complete the task: 5 − 6 × 0.1 = 4.4. Illustrative · worked example

A smaller score under a different reward rule can accompany a better outcome. Inspecting only the reward total would hide this. In a real evaluation, measure task completion separately and test conditions that were absent from training.

The pickup score keeps rising but the package stays at the depot. What failed?

03 · keep the distinction

Gaming a rule differs from changing it

Specification gaming

Exploit a gap in the task specification. Here, every repeated pickup legitimately triggers the same fixed reward rule.

Reward tampering

Interfere with the mechanism that assigns reward, such as modifying its code. The courier has no such action. Switching the rule above is your experiment control.

Anthropic’s 2024 reward-tampering study explored this distinction in a deliberately constructed training setup. Its results do not establish how often production models would tamper with rewards in ordinary use.

Paying once for delivery closes this particular loophole. It assumes that the delivery event is reliable and captures the entire task. Real tasks can have damaged packages, incorrect recipients or faulty measurements. A reward change needs fresh evaluation; this toy result does not prove real-world alignment or establish deceptive intent in a model.