Grandmaster Level in StarCraft II Using Multi-Agent Reinforcement Learning

StarCraft II is a real-time strategy game where two players compete by gathering resources, building bases, training armies, and battling for control of the map. Three asymmetric races — Protoss, Terran, and Zerg — offer fundamentally different unit rosters, abilities, and strategies. The game runs in real time with no turns, forcing simultaneous micro-management (controlling individual units in combat) and macro-management (economy, production, expansion).

What makes it uniquely hard for AI is the combination of several challenges that rarely appear together. First, imperfect information: fog of war hides most of the map, so the agent must reason about what the opponent might be doing without seeing it. Second, the action space is enormous: at each step the agent chooses an action type, which units to apply it to, and where on the map to target — a combinatorial explosion reaching up to 10²⁶ possible actions. Third, the game is non-transitive: there is no single dominant strategy because strategies form rock-paper-scissors cycles. And fourth, games last thousands of steps, requiring long-horizon planning where early economic decisions determine late-game outcomes.

AlphaStar's neural network has 139 million parameters and processes raw game observations — a list of visible units with their properties, minimap features, and scalar game state — to produce structured actions. The architecture has three major stages, each solving a different aspect of the challenge.

Stage 1 — Encoding: The agent receives the current observation as a set of units (each described by hundreds of features: type, health, position, owner). A Transformer processes these unit features, allowing every unit to attend to every other unit. Scatter connections then integrate spatial information from the minimap (processed by a ResNet) with the non-spatial unit embeddings. This fusion is critical because StarCraft requires reasoning about both what units exist and where they are on the map.

Stage 2 — Memory: The encoded observation feeds into a deep LSTM core that maintains a hidden state across timesteps. This is where temporal reasoning happens: the LSTM learns to track unseen enemy movements, remember scouting information, anticipate build timings, and plan multi-step strategies. Without this memory, the agent would be purely reactive and unable to execute coordinated plans.

Stage 3 — Action: The LSTM output feeds an auto-regressive policy head. Instead of choosing from a flat list of 10²⁶ actions, AlphaStar decomposes each action into a sequence of sub-decisions: what action type (build, move, attack), who to apply it to (unit selection via a pointer network), and where to target (map location). Each sub-decision conditions on all previous ones, forming a chain that makes the combinatorial action space tractable.

At each game step, a StarCraft action consists of multiple structured arguments: the action type (e.g. "move," "attack," "build"), the units involved, a target unit or map location, and queuing flags. If these were chosen independently, the space would be intractable. AlphaStar's key insight is to factor the joint distribution auto-regressively.

Formally, the policy factorizes the action as:

For unit selection, AlphaStar uses a pointer network — a mechanism borrowed from sequence-to-sequence models that can point to elements in a variable-length input set. Since the number of units on the battlefield changes constantly, a fixed-size output layer cannot work. The pointer network dynamically attends to the current unit list and selects which ones should execute the action. Think of it as a spotlight sweeping across the battlefield, highlighting units one by one.

AlphaStar's training unfolds in two major phases, each building on the previous one — like a student who first watches experts play, then enters a competitive league to develop their own style.

Phase 1 — Supervised Learning: The agent trains on 971,000 replays from human players with MMR above 3500 (roughly the top 22%). The policy learns to predict the human action at each timestep, conditioned on a strategy statistic z sampled from the same replay. This gives the agent a solid foundation of micro and macro play and produces a diverse set of initial policies reflecting different human playstyles. After supervised learning alone, the agent reaches approximately mid-Diamond level — competitive but far from professional.

Phase 2 — Reinforcement Learning in the League: Starting from the supervised agents, reinforcement learning fine-tunes the policy to maximize win rate. But naive self-play leads to strategy cycles and catastrophic forgetting. AlphaStar's breakthrough is the league training framework.

The central insight of AlphaStar's league training is that robust play requires diversity, not just strength. In a non-transitive game, being the best against one opponent means nothing if a different opponent can exploit you. The league addresses this with three types of agents.

Main Agents are the core players being trained. They play against all other agents in the league, weighted by how challenging each opponent is. Their objective is to find a strategy that is robust across the entire population — an approximation of a Nash equilibrium.

Main Exploiters target weaknesses specifically in the main agents. They are reset to supervised learning periodically, so they keep discovering new exploits rather than over-specializing. When a main exploiter finds a flaw, the main agent must patch it.

League Exploiters target weaknesses in the entire league, including past versions of all agents. They ensure the league maintains strategic diversity — if the entire league drifts toward one meta, the league exploiter discovers counter-strategies.

All agents are periodically frozen as snapshots and added to the opponent pool, creating a growing archive of diverse strategies. This prevents catastrophic forgetting: even if the current main agent forgets how to counter an old strategy, that strategy still exists in the league as a frozen opponent.

AlphaStar's RL algorithm must handle a difficult setting: the agent generates trajectories asynchronously on many machines, so by the time a trajectory is used for a gradient update, the policy has already changed. This is the off-policy problem.

The solution combines three techniques. First, V-trace provides importance-weighted value estimates that correct for the policy lag, originally developed for the IMPALA architecture. Second, TD(λ) computes multi-step temporal difference targets for the value function. Third, a novel technique called UPGO (Upgoing Policy Gradient) uses returns only when they exceed the value baseline, preventing the agent from being pulled down by unlucky trajectories.

In addition to the win/loss reward, AlphaStar uses pseudo-rewards derived from human statistics. The strategy statistic z — sampled from the human replay data — encodes build orders, unit compositions, and upgrade timings. A reward bonus encourages the agent to follow the sampled z during early training, ensuring strategic diversity. This bonus is annealed to zero over time, so the final agent optimizes purely for winning.

Think of it as training wheels: the human strategy statistics guide the agent's early exploration to sensible parts of the strategy space, preventing it from discovering a single degenerate exploit and getting stuck. Once the agent is strong enough, the training wheels come off.

AlphaStar was evaluated on Battle.net, the official StarCraft II ladder, playing anonymously against human opponents under standard game conditions with no simplifications. The final agent — AlphaStar Final — achieved an MMR above 6200 for all three races, placing it in the Grandmaster league (top 0.2% of active players).

Key results across the development stages show the contribution of each component. Supervised learning alone reaches approximately Diamond level (around MMR 3700). Adding single-agent RL with self-play pushes performance to Master level. But the leap to Grandmaster comes from league training: the multi-agent framework that discovers and patches strategic weaknesses through the interplay of main agents, exploiters, and frozen snapshots.

Crucially, AlphaStar played under human-like constraints: camera movements were restricted to mimic human viewing, and action rates were capped to match high-level human play (about 22 non-duplicated effective actions per 5-second window). The agent was not given any unfair advantages in speed or information.

The AlphaStar league ran for 14 days using 16 TPUs per agent. During training, each agent experienced up to 200 years of real-time StarCraft play. The league ultimately contained around 900 distinct agents (including all frozen snapshots), each a separate neural network with 139 million parameters.

This massive compute was necessary because StarCraft's strategic complexity required the league to discover an enormous variety of strategies and counter-strategies. Each frozen snapshot preserves a unique strategic "species" in the ecosystem. The final AlphaStar agent is robust precisely because it has been tested against this entire zoo of opponents.