3.10Research Connection: N-Agent Ad Hoc Teamwork

5 min read

By the end of this section you can answer

What if the number of unfamiliar teammates changes too?

N-Agent Ad Hoc Teamwork extends partner generalization to teams in which both the number and behavioural types of uncontrolled agents can change.

In this section you will

• Locate N-Agent Ad Hoc Teamwork between controlling one agent and the whole team
• Explain why variable team composition matters at deployment
• Describe POAM’s partner-representation approach
• Read the reported findings and what they do and do not establish

The N-Agent Ad Hoc Teamwork Setting

Two settings have been well studied, and they are the two extremes.

	How many agents the learner controls
Standard cooperative MARL	all of them
Traditional ad hoc teamwork	one of them

N-Agent Ad Hoc Teamwork (NeurIPS 2024) points out that real systems can sit between these extremes. The authors introduce a setting in which a set of autonomous agents must interact and cooperate with dynamically varying numbers and types of partners.

Both halves of that phrase matter, and the first is the one previous framings did not cover.

Figure 1

$$\underbrace{\tone{policy}{\text{agents we control}}}_{\text{some}} \;+\; \underbrace{\tone{conflict}{\text{agents we do not}}}_{\text{how many? which kinds?}}$$

$\text{controlled agents}$

the policies available to the learner

$\text{uncontrolled agents}$

partners whose number and types may vary

NAHT teams combine controlled policies with a variable set of unfamiliar partners.

Variable Team Composition

Return to the motivating examples from Ad Hoc Teamwork and count.

A warehouse operator deploying twelve of its own robots alongside a subcontractor’s eight controls some of the team. So does a fleet operator whose vehicles meet other manufacturers’ vehicles at a junction: how many of each is at the junction changes minute to minute.

Neither situation is described by controlling one agent, and neither is described by controlling all of them. And the difference is not cosmetic: an agent that can rely on three co-trained colleagues behaves differently from one that is alone among strangers, and in the middle case it does not know in advance which it will be.

Key takeaway

The number of partners is usually treated as a fixed part of the problem.

Once it varies, an agent cannot even count on how much of the team shares its conventions, so a policy has to work across compositions rather than just across partners.

POAM and Partner Representations

The authors propose Policy Optimization with Agent Modelling, or POAM.

The idea connects directly to sections 3.4 and 3.5 rather than arriving as an unrelated algorithm. POAM is a policy gradient method that learns representations of partner behaviours and uses them to adapt.

That is the behaviour embedding from Partner Representations, put to work:

Figure 2

$$\ubrace{observe}{\text{observed partner behaviour}}{what I have seen} \rightarrow \ubrace{policy}{\text{learned representation}}{who these partners are} \rightarrow \ubrace{action}{\text{my action}}{what I should do about it}$$

$\text{observed behaviour}$

locally available evidence about the partners

$\text{representation}$

a learned summary of the variable partner set

$\text{action}$

the controlled agent’s response

POAM conditions controlled actions on learned representations of the observed partner set.

Note what the representation has to survive here. Under NAHT the partners vary in kind and in number, so a representation indexed by partner identity is doubly useless: it has no entry for a stranger, and no way to describe a team of three strangers rather than one.

Reported Experimental Findings

The authors evaluate POAM in multi-agent particle environments and StarCraft II, and report improved cooperative performance over baselines along with generalization to previously unseen partners.

Published results, not ours

Everything in this section is what the paper reports on its own environments and baselines. Nothing here was reproduced in this resource.

The reason to read it is the problem formulation, which reframes ad hoc teamwork as a spectrum rather than a binary, and the confirmation that behaviour representations are the mechanism people reach for.

Implications for Ad Hoc Teamwork

How should an agent represent the behaviour of partners when both their type and their number can change?

Worth sitting with. A representation of one partner is a vector. A representation of a team of unknown size, whose members you may be individually uncertain about, is a harder object, and how to build one is open.

Knowledge check

Why is a partner representation indexed by identity especially useless in the NAHT setting?

Select one answer.

ABecause the partners vary in both kind and number, so an identity index has no entry for a stranger and no way to describe a team of several unknowns.BBecause identities are private information that agents cannot observe.CBecause policy gradient methods cannot use discrete representations.DBecause NAHT requires controlling only one agent.

Check answer

N-Agent Ad Hoc Teamwork Lessons

• Two well-studied extremes: the learner controls all agents (standard cooperative MARL) or one (traditional ad hoc teamwork).
• NAHT is the setting in between, where agents cooperate with dynamically varying numbers and types of partners.
• Real deployments are mostly in the middle: your own fleet alongside somebody else’s, in proportions that change.
• POAM is a policy gradient method that learns representations of partner behaviour and uses them to adapt, which is the Partner Representations mechanism instantiated.
• The authors report improved cooperative performance and generalization to unseen partners on multi-agent particle environments and StarCraft II.

Further reading

N-Agent Ad Hoc Teamwork, Caroline Wang, Arrasy Rahman, Ishan Durugkar, Elad Liebman and Peter Stone. Advances in Neural Information Processing Systems 37 (NeurIPS 2024). Proceedings page.