1.6Centralized Critics

5 min read

By the end of this section you can answer

What changes when the critic sees more than the actor does?

A centralized critic is one way to use global information under CTDE: the critic evaluates actions during training while each actor remains a local policy for execution.

In this section you will

• Separate the roles of the actor and the critic
• Expand the critic’s inputs to the state and the joint action
• Explain how a richer evaluation changes the policy update
• State what remains decentralized at execution

Actor and Critic Roles

Many reinforcement learning methods split the learner in two.

The actor is the policy: the thing that chooses actions. The critic is a learned value or advantage function used to improve that policy. It does not choose anything, it evaluates.

Figure 1

$$\underset{\text{\scriptsize chooses}}{\tone{policy}{\pi_\theta(a \given s)}} \qquad\qquad \underset{\text{\scriptsize evaluates}}{\tone{reward}{V_\phi(s)}}$$

$\pi_\theta$

the actor: the policy, with its own parameters θ

$V_\phi$

the critic: a learned estimate of how good a situation is, parameters φ

Two components with two jobs. Only one of them has to run at deployment.

The critic exists to make the actor’s updates less noisy. Instead of judging an action by the whole messy return that followed it, the actor is nudged according to how that action compared with what the critic expected.

And here is the observation that makes this section work: only the actor is needed at deployment. The critic is training scaffolding. Once the policy is trained, the critic is discarded.

Key takeaway

If the critic is thrown away before deployment, the decentralization constraint never applies to it.

So there is no reason to keep it blind.

Centralized Critic Information

That is the whole idea. Condition the critic on the centralized information $z_t$, and leave the actor exactly as constrained as it has to be.

Figure 2

$$\tone{reward}{V_\ag\bigl(\tone{observe}{h^\ag_t},\ \tone{comm}{z_t}\bigr)} \qquad\text{or}\qquad \tone{reward}{Q_\ag\bigl(\tone{observe}{h^\ag_t},\ \tone{comm}{z_t},\ \tone{action}{\jointact}\bigr)}$$

$h^\ag_t$

agent i’s own history, what it has personally seen and done

$z_t$

centralized information, available because this is training

$\jointact$

the joint action; an action-value critic is conditioned on what everybody did

Two flavours of centralized critic: one evaluates the situation, one evaluates the situation together with what the whole team did.

Both forms appear in the literature. A centralized state-value critic is conditioned on the agent’s history plus the centralized information. A centralized action-value critic is additionally conditioned on the joint action, which, from Coordination, is exactly the form that can tell combinations apart.

Meanwhile the actor is untouched:

Figure 3

$$\tone{policy}{\pol{\ag}\bigl(\tone{action}{\act{\ag}} \given \tone{observe}{h^\ag_t}\bigr)}$$

$\pol{\ag}$

the actor: still local, still deployable, still the only thing that runs

$h^\ag_t$

its own history and nothing else, no z, no partner action

The part that survives deployment never learned to depend on anything it will not have.

The picture is the CTDE picture with one box renamed, and that is the point. A centralized critic is not a different arrangement. It is a specific answer to “what consumes $z_t$?”

Effects on the Learning Signal

The critic’s job is to say how good things were. A blind critic has to do that from one agent’s partial view, which means the same observation gets the same evaluation whether the team’s other half was doing something useful or something disastrous. Its estimates are noisy for a reason that has nothing to do with the agent being evaluated.

Give it $z_t$ and it can see what the team actually did. Its estimates get sharper, the actor’s updates get less noisy, and learning gets more stable, all without changing what the actor is allowed to know.

Common pitfall

Do not oversell this. A centralized critic makes the learning signal better informed; it does not solve coordination.

The actor still chooses from local information, still cannot see its partner’s choice for the current step, and still has to arrive at a policy that happens to fit with its partner’s. A better-informed teacher does not remove the student’s blindfold, it just stops giving misleading feedback.

Knowledge check

Why is it acceptable for a critic to use the global state when the actor may not?

Select one answer.

AThe critic is discarded after training, so it never has to run under the deployment constraint.BThe critic only estimates values, and values are less sensitive than actions.CThe actor indirectly receives the global state through the critic’s gradients, so both effectively see it.DBecause the global state is usually similar to the agent’s observation anyway.

Check answer

Centralized Critics Summary

• An actor chooses actions; a critic evaluates. Only the actor is needed at deployment.
• A centralized critic is conditioned on the agent’s history plus centralized information $z_t$; an action-value version is additionally conditioned on the joint action $\jointact$.
• The actor stays local: $\pol{\ag}(\act{\ag} \given h^\ag_t)$.
• The benefit is a better-informed learning signal, the critic can see what the team did, so its estimates are less confounded by partner behaviour.
• It does not solve coordination. The actor still decides alone, on partial information.