3The Multi-Agent Environment

7 min read

By the end of this section you can answer

Who decides, and what does the environment actually respond to?

A multi-agent environment receives decisions from several agents and advances once in response to their combined action.

In this section you will

• Define the agent set and the environment state
• Distinguish an individual action from a joint action
• Trace one state transition driven by a joint action
• Describe an episode and where it starts

The Agent Set

Start by naming them. The set of agents is

Figure 1

$$\tone{policy}{\mathcal{N}} = \{1, 2, \dots, \nag\}$$

$\mathcal{N}$

the set of agents, each with its own decision to make

$\nag$

how many there are; fixed for the whole episode

A list of deciders. In the kitchen, N = {1, 2}.

Every agent in $\mathcal{N}$ acts at the same time. At each step all of them choose, then the environment moves once. No agent gets to see what another chose for the current step before choosing itself. There is no turn-taking here, and nothing has been combined yet at the moment of deciding.

Simultaneous, not synchronised

“Simultaneous” is a claim about information, not about clocks. It means no agent can condition its choice on another agent’s choice at the same step. Agents that both observe the same kitchen will often choose related actions. That is not a violation; it is just two agents reacting to the same thing.

Environment State

The state $\st \in \mathcal{S}$ is the complete configuration of the environment at step $t$. For the kitchen it would include

• where each agent is standing,
• what each agent is holding,
• where the ingredients are, and which are prepared,
• what is on the stove and how cooked it is,
• what the current order asks for and how far it has got,
• how much time is left.

That list is the whole situation, and it is what the environment’s rules are defined on.

Common pitfall

The state describes the environment. It does not describe what any individual agent knows.

Everything above is true of the kitchen whether or not anybody can see it. An agent facing the counter has no idea how cooked the pot is. The next section is entirely about this gap; it is worth expecting it now, because the most common way to get multi-agent reasoning wrong is to quietly assume every agent knows $\st$.

Individual Actions

Each agent has its own set of things it can do, and picks one:

Figure 2

$$\tone{action}{\act{\ag}} \in \mathcal{A}_{\ag}$$

$\act{i}$

the action agent i takes at step t

$\mathcal{A}_{i}$

agent i’s own action set; different agents may have different sets

One choice per agent, from that agent's own menu.

In the kitchen the menu is small and physical: move, pick up, put down, prepare, interact with whatever is in front of you. Nothing in that list mentions the other agent. An agent cannot choose “make my partner fetch the tomato”; it can only act, and let the consequences reach the partner through the kitchen.

Joint Actions

The environment does not receive two separate actions. It receives one object containing both.

Figure 3

$$\tone{action}{\jointact} = \bigl(\tone{policy}{\act{1}},\ \tone{policy}{\act{2}},\ \dots,\ \tone{policy}{\act{\nag}}\bigr)$$

$\jointact$

the joint action: the whole ordered tuple, in bold

$\act{i}$

agent i’s action, sitting at position i

Everybody's choice for this step, gathered into one input.

The order is not cosmetic: position $i$ always means agent $i$, so agent 1 chops, agent 2 heats is a different tuple from agent 1 heats, agent 2 chops, even though the same two actions appear in both.

And the set of all such tuples is the product of the individual sets:

Figure 4

$$\mathcal{A} = \mathcal{A}_{1} \times \mathcal{A}_{2} \times \cdots \times \mathcal{A}_{\nag}$$

$\mathcal{A}$

the joint-action space

$\mathcal{A}_i$

the local action space available to agent i

$\nag$

the number of agents

The joint-action space is the Cartesian product of the local action spaces.

That word product is doing real work. It is worth feeling the size before reading it.

Predict first

Four agents. Five actions each. How many joint actions can the environment receive?

[25]   [125]   [625]   [1024]

$5^4 = 625$. Not $5 \times 4 = 20$, which is the count of individual choices available across the team, and not $5 + 5 + 5 + 5$. Each agent’s five options multiply against every other agent’s, so the joint action space grows exponentially in the number of agents.

Agents	Actions each	Joint actions
2	5	25
4	5	625
8	5	390,625
4	9	6,561
8	9	43,046,721

Common pitfall

Exponential growth is real and it costs you, in computation and in how much exploring a team must do. But it is not the specifically multi-agent difficulty, and it does not by itself explain coordination failure, 625 things at a fixed state are few enough to simply write out and check.

The harder fact is that nobody chooses among the 625. Four agents each choose among five, from their own separate information, and the outcome depends on all four. Joint Actions and Policies returns to this.

Key takeaway

Even a small kitchen grows quickly. If four agents each choose from nine local actions, the environment must distinguish $9^4 = 6{,}561$ joint actions at a single state, although each agent still selects from only nine.

State Transitions

Once the joint action arrives, the environment moves.

Figure 5

$$\tone{observe}{s_{t+1}} \sim P\bigl(s_{t+1} \given \tone{observe}{\st},\ \tone{action}{\jointact}\bigr)$$

$s_{t+1}$

the next state

$P$

the transition function: the environment’s rules

$\st$

the state the step began in

$\jointact$

the joint action; note P is not defined on any single agent’s action

Where the kitchen ends up depends on what the agents did together.

Read the right-hand side and notice what is missing. There is no $P(s_{t+1} \given \st, \act{1})$ anywhere in this framework. You cannot ask what agent 1’s action does to the kitchen, because the question is not well-formed: the answer depends on what agent 2 did at the same moment.

Here is the same step drawn out.

Each agent decides on its own, on its own information. Those separate decisions are then combined into one input, and the environment transitions once. That single convergence point is the whole geometry of a multi-agent step, and most of this resource is about living with it.

Episodes and Initial States

The rest is bookkeeping, and it works as it does in the single-agent case.

An episode starts from an initial state drawn from a distribution $\mu$, a fresh kitchen, a new order. Then the loop above repeats: everybody acts, the state moves, everybody acts again. It ends on a terminal condition, either success (the order is served) or a step limit (service is over). Learning happens across many episodes, not within one.

Knowledge check

You want to know what effect 'place pot on stove' has on the kitchen. Which statement is correct?

Select one answer.

AThe question is not well-formed on its own. The transition is defined on the joint action, so the effect depends on what the other agent does at the same step.BThe pot ends up on the stove, since that action affects only the pot.CYou can answer it by averaging over the other agent’s actions.DIt depends on the reward function.

Check answer

Multi-Agent Environment Summary

• $\mathcal{N} = \{1, \dots, \nag\}$ agents act simultaneously: no agent conditions on another’s choice for the current step.
• The state $\st$ is the whole configuration of the environment. It is not what any agent knows.
• Each agent picks $\act{\ag}$ from its own set; those choices combine into the joint action $\jointact$.
• The joint action space is the product $\mathcal{A}_1 \times \cdots \times \mathcal{A}_\nag$, so it grows exponentially: four agents with five actions each give 625 joint actions.
• The transition $P(s_{t+1} \given \st, \jointact)$ is defined on the joint action. “What does my action do?” has no answer on its own.