Evaluation

1 min read

Always greedy, because an evaluation that keeps exploring reports a policy nobody would deploy. Always from a fixed seed, so two systems are compared on the same episodes rather than on different luck.

Import from cooperative_marl_labs.evaluation.

Functions

evaluate_agents

from cooperative_marl_labs.evaluation import evaluate_agents

evaluate_agents(
    env,
    agents: Mapping[str, Any],
    episodes: int = 100,
    seed: int = 42,
    replace: Mapping[str, Any] | None = None,
) -> dict[str, float]

Run episodes greedy episodes and return per-step averages.

Parameters

replace: Swaps in a policy for one agent id, which is how an unfamiliar access point is introduced without retraining anything.

Returns

dict team_reward, total_throughput, mean_throughput, interference, collision_rate, messages_sent, plus avoidable_interference: how much of the interference a better allocation could have removed. That last one matters because with more access points than channels some interference is unavoidable, so raw interference makes every policy look equally bad.

evaluate_partner_policy

from cooperative_marl_labs.evaluation import evaluate_partner_policy

evaluate_partner_policy(
    env,
    ego_act: Callable[[Any, list[int]], int],
    partner,
    episodes: int = 200,
    seed: int = 42,
    switch: tuple[int, Any] | None = None,
) -> float

Mean reward per step for an ego policy against one partner.

ego_act(observation, partner_history) keeps the signature wide enough for both a fixed policy and one that infers from the history.

switch replaces the partner part-way through every episode, which is how the lab tests whether an estimator can notice a change.

crossplay_matrix

from cooperative_marl_labs.evaluation import crossplay_matrix

crossplay_matrix(
    ego_policies: Mapping[str, Any],
    partner_policies: Mapping[str, Any],
    evaluate_fn: Callable[[Any, Any], float],
) -> pd.DataFrame

Returns a DataFrame indexed by ego name, columns by partner name.

evaluate_fn(ego, partner) returns one score.

Constants

Name	Value
`WIRELESS_METRICS`	`('team_reward', 'total_throughput', 'mean_throughput', 'interferenc...`