1.13Coordinate Worksheet

3 min read

This five-part digital activity consolidates the chapter’s central coordination ideas. You will first match learning approaches to their applications, then calculate a joint-action-space size and a VDN team value. An interactive graph shows how joint-action counts grow with the number of agents. The final prompt asks you to weigh the training benefits of CTDE against its additional complexity. Every item except that trade-off response is automatically checkable.

What you will be able to do

• Match common MARL training approaches to the problems they address. Understand
• Calculate the size and value of simple joint-action problems. Apply
• Interpret how coordination difficulty changes as agents are added. Analyse
• Compare coordination approaches for a given deployment constraint. Evaluate

1 · Match Approaches to Applications

Drag each block into a slot, or tap one and then tap a slot.

• Independent learning
• CTDE
• Centralized critic
• VDN
• QMIX

1. Each agent learns using only its own experience.

   drop here

2. Global information is available during training, but agents act from local observations at deployment.

   drop here

3. A value function uses global information during training while the actor stays decentralized.

   drop here

4. Individual action values are added to produce a team value.

   drop here

5. Individual values are combined by a monotonic mixing function.

   drop here

CheckClear

2 · Calculate the Joint-Action Space

2.1

Four agents each have five possible actions. How many joint actions are possible?

Answerjoint actionsCheck

Need a hint?

2.2

Now six agents, still five actions each.

Answerjoint actionsCheck

Need a hint?

3 · Calculate a VDN Team Value

Under VDN the team value is the sum of the individual values, $Q_{\text{tot}} = \sum_i Q_i$.

3.1

Given $Q_1 = 3.2$, $Q_2 = 2.6$ and $Q_3 = 4.1$, calculate $Q_{\text{tot}}$.

AnswerCheck

3.2

Agent 2 switches to an action with $Q_2 = 3.4$. What is the new team value?

AnswerCheck

Need a hint?

4 · Explore Joint-Action Growth

Each agent has five actions. Drag the slider to add agents.

Number of agents

4.1

At ten agents, how many joint actions are there?

Answerjoint actionsCheck

5 · Evaluate the CTDE Trade-off

Ten warehouse robots must act independently at deployment. During training, the simulator provides the complete warehouse state.

5.1

Would you prefer fully independent learning or CTDE here? What is the main trade-off in your choice?

AnswersReveal

1

Answer to 1.

Independent learning · CTDE · centralized critic · VDN · QMIX.

If one tripped you up it is probably CTDE against centralized critic: CTDE is the paradigm that permits global information during training, and a centralized critic is one way of using it.

2.1

Answer to 2.1.

$5^4 = 625$.

2.2

Answer to 2.2.

$5^6 = 15{,}625$. Two extra agents multiply by $5^2 = 25$, not by 2.

3.1

Answer to 3.1.

$3.2 + 2.6 + 4.1 = 9.9$.

3.2

Answer to 3.2.

$9.9 - 2.6 + 3.4 = 10.7$. Under a plain sum, one agent changing action moves the team value by exactly its own change.

4.1

Answer to 4.1.

$5^{10} = 9{,}765{,}625$. The tenth agent added more joint actions than the first nine combined.

5.1

Answer to 5.1.

No single right answer. A good one names what it gives up.

CTDE is the strong default: the simulator’s state costs nothing during training, execution is local either way, so refusing it buys nothing. Price: one more component to train and debug.

Independent learning is defensible under time pressure. It needs no new algorithm, keeps per-agent cost flat as the fleet grows, and is the baseline CTDE has to beat. Price: non-stationarity and high variance across seeds.

Review with Flashcards

Revisit the 10 key concepts, equations, and intuitions from this chapter.

Open Coordinate Flashcards