Machine learning · extension
Reinforcement learning
A useful action may pay off several steps later. Reinforcement learning uses rewards from interaction to update estimates of which actions lead to larger returns.
This lesson trains a small table in your browser. It needs no neural network: each grid position is a state, and each state has four action values.
Rewards evaluate outcomes; labels specify targets
In supervised learning, an example supplies a target output, such as the correct image category. Here, the environment returns the next state and a numerical reward after an action. It does not provide the correct action at every position. The action changes which experience arrives next.
State → action
Position 14 → move right
Environment response
Goal 15, reward +1, episode ends
Learning update
Change Q(14, right) towards +1
illustrative A terminal transition in the grid below. Rewards and walls are fixed; movement is deterministic.
Inspect one update, then train a policy
Q(state, action) estimates discounted future return after that action. A policy is the rule for choosing actions. This demo trains with epsilon-greedy sampling and evaluates with the largest Q value only. The heatmap shows max Q at each state; the arrows show the greedy policy derived from it.
Try one learning step, then reset and train 100 episodes. Select a square to compare its four action values. Set ε to 1 to sample every training action randomly; evaluation still uses the greedy arrows. Set γ to 0 to remove the future-value term and compare how the policy changes.
live Computed in this browser · synthetic gridworld · seed 42
Changing any setting clears Q and restarts seed 42. α controls update size; γ weights future reward; ε is the probability of sampling a random action, which may also be the greedy action.
Policy arrows and value heatmap live
Wall
Wall
● current position. Green is positive max Q; orange is negative. Arrows select the largest action value. Ties use up, right, down, left, in that order. The goal has no next action.
State 0 action values live
| Action | Q estimate |
|---|---|
| Up | 0.000 |
| Right | 0.000 |
| Down | 0.000 |
| Left | 0.000 |
State 0 is the start. Every move costs −0.1, except entering goal 15 gives +1 and ends the episode. A boundary or wall leaves the position unchanged and still costs −0.1.
live Current episode: 0 steps · undiscounted return 0.000 · completed episodes 0
Each episode stops at the goal or 80 steps. A time limit truncates the episode; it does not turn the last state into a terminal goal. A batch starts fresh episodes and retains Q. The next learning step after an episode ends starts at state 0.
Ready. All Q values are zero; seed 42.
One transition, one update live
Take one step to expose the state, sampled action and numerical update.
Training and evaluation measure different behaviour
Run a batch to see its training success count and mean return.
| Q table | Goal | Steps | Return |
|---|---|---|---|
| Untrained | No | 80 | -8.000 |
| Current | No | 80 | -8.000 |
Return here is the sum of rewards without discounting; Q estimates discounted future return. This deterministic world has one start, so repeating the same greedy evaluation adds no evidence. Finite training can leave a looping or incomplete policy.
Delayed credit moves through estimates
On the first visit to the goal, the action just before it receives a positive update. An earlier action can then receive a better target when its next-state estimate has improved. Repeated visits can carry the influence of a distant reward back towards the start.
The update is Q ← Q + α × (reward + γ × max next Q − Q). α changes how far one update moves. γ discounts future reward. The max uses an estimated value, so an update is not proof that the estimate is correct.
Exploration changes the experience
With probability ε, training samples a random action. Otherwise it uses a greedy action. Q-learning still builds its target from the largest next-state Q value, even when the sampled action came from exploration. This difference makes it an off-policy method.
A fixed learning rate, a finite number of episodes and limited exploration do not guarantee convergence. This small deterministic world leaves out uncertain observations, changing dynamics, continuous actions and the cost of gathering real experience.
Check the mechanism
Reward is part of the specification
This grid rewards reaching one marked goal. In a larger system, a high measured return only shows success against the reward that was defined. The next lesson changes the reward and examines how a policy can score well while missing the intended outcome.
Reference: Sutton and Barto, Reinforcement Learning: An Introduction, second edition, chapters 3 and 6; Q-learning update in section 6.5.