1Reinforcement Learning

8 min read

Reinforcement learning concerns an agent that improves its decisions through interaction rather than labelled examples.

In this section you will

Describe the agent and environment loop, one step at a time
Distinguish a policy from a single action
Compute a discounted return from a sequence of rewards
State the objective a single agent optimizes
Map a one-agent kitchen onto the loop

One agent, one kitchen, one order to serve. Everything in this section is a question about what this agent should do next.

Learning from Interaction

Reinforcement learning is what you do when nobody can tell you the right answer, but somebody can tell you how well things went.

The setup has two parts and nothing else. The agent is the thing that decides. The environment is everything else, in our case the kitchen, the ingredients, the stove and the order. The agent takes an action, the environment changes and reports back, and the agent decides again. Learning happens in that loop and nowhere else.

Two features make this a distinct kind of problem.

The decisions are sequential, and they interfere with each other. Turning on the stove is not good or bad by itself. It is good if you are about to cook and bad if you have not collected the ingredients, and you will not find out which for several steps. Each choice changes the situation the next choice is made in.

The feedback evaluates rather than instructs. This is the sharpest break from supervised learning, and it is worth stating in Sutton and Barto’s terms: reinforcement learning uses evaluative feedback, supervised learning uses instructive feedback. A label says “the answer was 7”. A reward says “that went badly”, without saying what would have gone well.

Paradigm	What the data gives you	What you learn
Supervised	inputs paired with correct outputs	to reproduce the labels
Unsupervised	inputs only	structure in the inputs
Self-supervised	inputs, with part of each hidden	to predict the hidden part
Reinforcement	a scalar score for what you actually did	to act so the score is high

The last row differs from the others in two ways that matter. You only ever learn about the action you took, so you have to try things to find out about them, nobody hands you a dataset of alternatives. And the data you learn from is generated by your own behaviour, so it changes as you improve.

The Reinforcement Learning Loop

Formally, the interaction is a repeating cycle of four things.

At each step $t$ : the agent receives the state $\st$ , describing the situation; it takes an action $a_t$ ; the environment returns a reward $\rew$ , a single number scoring what happened; and the environment moves to the next state $s_{t+1}$ .

Figure 1

\tone{observe}{\st} \;\longrightarrow\; \tone{action}{a_t} \;\longrightarrow\; \tone{reward}{\rew},\ \tone{observe}{s_{t+1}}

$\st$: the state at step t: what the situation is now
$a_t$: the action the agent takes in response
$\rew$: the reward: one number scoring what just happened
$s_{t+1}$: the next state, which is where the following step begins

One step. Repeat this and you have everything reinforcement learning studies.

Run that cycle from a starting situation until the task ends, the order is served, or the kitchen runs out of time, and you have one episode. Learning happens across many episodes.

Policies as Decision Rules

The agent’s behaviour is entirely captured by one object: its policy, the rule that turns a situation into an action. When you say a learner has improved, you mean its policy changed.

Figure 2

\tone{action}{a_t} \sim \tone{policy}{\pi}\bigl(a_t \given \tone{observe}{\st}\bigr)

$\pi$: the policy: the agent’s decision rule
$a_t \sim$: the action is drawn from the distribution the policy defines
$\st$: the situation the policy is given

The policy is the agent. Given this situation, what do I do?

Two flavours, and the distinction returns later. A deterministic policy picks one action for each state: $a_t = \pi(\st)$ . A stochastic policy defines a distribution over actions and samples from it, as written above. Stochastic policies matter because randomness is how an agent tries things it has not tried, and, in the Coordinate chapter, because sometimes a team is genuinely better off flipping a coin than committing.

Return and the Learning Objective

An agent that maximises the reward available right now is not doing the task. The kitchen agent could stand at the stove collecting whatever the stove pays and never serve anything. What matters is the whole episode.

Figure 3

\tone{reward}{G_t} = \sum_{k=0}^{T-t} \ubrace{observe}{\gamma^{k}}{how much later steps count} \; \tone{reward}{r_{t+k}}

$G_t$: the return from step t onward: the total the agent still stands to collect
$\gamma$: the discount factor, between 0 and 1
$r_{t+k}$: the reward k steps into the future
$T$: the last step of the episode

Not this step's reward. Everything from here to the end of the episode, added up.

The discount factor $\gamma$ says how much a reward that arrives later is worth now. At $\gamma = 0$ the agent is completely short-sighted and only the current reward exists. As $\gamma$ approaches 1 a reward ten steps away counts almost as much as one right now, and the agent becomes willing to invest: collect ingredients that pay nothing in themselves, because they lead to a served order.

Because both the environment and the policy can be random, no single episode is the thing to optimise. The objective is the average return the policy achieves.

Figure 4

\tone{policy}{J(\pi)} = \E_{\tau \sim \tone{policy}{\pi}}\bigl[\, \tone{reward}{G(\tau)} \,\bigr]

$J(\pi)$: the objective: how good the policy is
$\tau$: a trajectory: one whole episode of states, actions and rewards
$\E_{\tau \sim \pi}$: average over the episodes that policy produces
$G(\tau)$: the return of that episode

A policy is good if the episodes it produces tend to go well.

Everything else in reinforcement learning, value functions, Q-learning, policy gradients, is machinery for finding a $\pi$ that makes $J(\pi)$ large. This resource introduces those pieces only where a multi-agent idea actually needs them.

The Kitchen as a Reinforcement-Learning Problem

Four abstract objects, one concrete kitchen. This is the whole of single-agent reinforcement learning applied to our example.

Piece	In the kitchen
State $\st$	where the agent is, what it holds, which ingredients are prepared, how far the order has got
Action $a_t$	move, pick up, put down, chop, place on stove, serve
Reward $\rew$	a payment when an order is served
Policy $\pi$	the agent’s habit of what to do in each situation
Episode	one service, from an empty kitchen to a served order or the end of time
Return $G_t$	how much of that service is still to come, discounted

Given all of this, a good policy in the single-agent kitchen is a sequence worked out by one decider: collect, chop, heat, cook, plate, serve. The agent controls every part of it. If something is not done, it is because this agent did not do it.

Knowledge check

The kitchen agent chops a tomato and receives a reward of 0. What has it learned?

Select one answer.

That chopping led to no immediate payment. Not that chopping was the wrong action.That chopping was wrong, since a correct action would have been rewarded.That it should have taken a different action, though not which one.Nothing, because zero rewards carry no information.

Reinforcement Learning Summary

Reinforcement learning is learning from interaction: act, observe the consequence, act again.
Its feedback is evaluative, not instructive. A reward scores what you did; it does not reveal what you should have done.
One step is $\st \rightarrow a_t \rightarrow \rew, s_{t+1}$ . An episode is a run of those steps.
A policy $\pi(a_t \given \st)$ is the agent. Improving means changing it.
The objective is the expected discounted return $J(\pi)$ , not the immediate reward. The discount factor $\gamma$ sets how far ahead the agent cares.
If the correct actions are already labelled, use supervised learning instead.