Mastering the Game of Go Without Human Knowledge

The original AlphaGo was a landmark — the first program to defeat a professional Go player. But its pipeline had three dependencies on human expertise:

• Supervised pre-training from 30 million positions in expert games to initialize the policy network.
• Hand-crafted features — liberty counts, ladder patterns, stone age — engineered by Go experts.
• Separate networks — a policy network for move selection and a value network for position evaluation, trained independently.

These dependencies meant AlphaGo could refine human strategies but never fundamentally transcend them. Worse, the approach was Go-specific: there was no path to transfer it to chess, protein folding, or any domain lacking millions of expert demonstrations.

AlphaGo Zero's premise is radical: throw away all human data. Start from random weights. Let the system play against itself, and use the game outcomes to improve. The entire training loop has just three steps:

1. Play — the current neural network plays games against itself, using Monte Carlo Tree Search to select moves.
2. Learn — the network is trained to predict the MCTS-improved move probabilities and the eventual game winner.
3. Evaluate — the new network plays the current best; if it wins more than 55% of games, it replaces the best and becomes the new self-play generator.

This is a form of generalized policy iteration: MCTS acts as a policy improvement operator (it finds stronger moves than the raw network), and self-play acts as a policy evaluation operator (game outcomes evaluate how good the policy really is). The network distills the search's wisdom, and the improved network makes the next search even better — a virtuous cycle.

At each turn, AlphaGo Zero doesn't just use the network's raw move predictions. Instead, it runs 1,600 simulations of MCTS — each simulation a journey from the current board state down a search tree. Think of it as mentally rehearsing possible futures before choosing a move.

Each simulation has four stages:

Select — starting from the root (current position), walk down the tree, at each node picking the action that maximizes Q(s,a) + U(s,a). Q is the average value from past simulations; U is an exploration bonus based on the network's prior probability and how rarely this action has been tried. This is the PUCT formula — it balances exploitation (moves that have worked well) with exploration (moves the network thinks are promising but haven't been tested enough).

Expand — when you reach a leaf node (a position never seen before), expand it: feed the board to the neural network, which returns a policy vector p (how promising each move is) and a value v (who's likely to win from here).

Backup — carry the value v back up the path, updating Q for every edge you traversed. After many simulations, Q converges to a reliable estimate of each move's worth.

Play — after all 1,600 simulations, choose the move proportional to how often each action was visited at the root. Visit count, not raw value, is the selection criterion — a subtle but critical detail that ensures robust move selection.

At each node in the search tree, MCTS must decide which branch to explore next. It selects the action that maximizes the sum of two terms: the estimated value Q(s,a) and an exploration bonus U(s,a). The idea is intuitive — follow paths that either have proven high value (exploit) or are promising but under-explored (explore).

Unlike the original AlphaGo, which used two separate networks, AlphaGo Zero uses a single dual-headed residual network. The input is a 19×19×17 tensor: the current board plus the last 8 history positions for each player, plus a color indicator.

The network has three parts:

Residual tower — 19 (or 39) residual blocks, each containing two 3×3 convolutional layers with batch normalization and ReLU. The residual connections (skip connections) allow gradients to flow through the deep stack without vanishing — the same trick that made deep image networks possible.

Policy head — a 1×1 convolution followed by a fully connected layer outputting a probability distribution over all 19×19 + 1 = 362 possible moves (including pass). This answers: "what should I play next?"

Value head — a 1×1 convolution, a fully connected hidden layer with 256 units, and a tanh output giving a single scalar in [-1, +1]. This answers: "who is winning from this position?"

Sharing a residual tower between both heads means the network learns features useful for both tasks simultaneously — a position pattern that helps predict good moves also helps predict who will win.

After each self-play game produces a dataset of (state, MCTS policy, game outcome) triples, the network is trained by minimizing a combined loss:

Read this loss as two teaching signals and a safety net. The value loss says: "get better at predicting who wins." The policy loss says: "learn to mimic the search's move choices — they're smarter than your raw guesses." The regularization says: "don't memorize specific positions — generalize."

As training progresses, the network predicts the search's output more accurately, which makes the search itself more powerful (because it starts from a better network), which produces even better training data — the virtuous cycle again.

A danger of self-play is converging to a narrow strategy — mastering one opening and never discovering that a different opening is even better. AlphaGo Zero guards against this by adding Dirichlet noise to the root node's prior probabilities:

$P(s, a) = (1 - \epsilon) \cdot p_a + \epsilon \cdot \eta_a, \quad \eta \sim \text{Dir}(\alpha)$

With $\epsilon = 0.25$ and $\alpha = 0.03$ for Go's 19×19 board. This means 25% of the root prior comes from random noise rather than the network — enough to ensure the search occasionally tries surprising moves. The noise only applies at the root (not deeper in the tree), so the search remains focused once a direction is chosen.

Think of it as a creative constraint: the network proposes moves, but a random nudge forces it to occasionally consider moves it would otherwise dismiss. Some of these surprises turn out to be genuinely good — and that's how AlphaGo Zero discovers strategies no human ever played.

AlphaGo Zero's training curve is one of the most dramatic in machine learning history. Starting from random play — literally choosing moves uniformly at random — the system:

• After 3 hours: plays basic legal Go, capturing stones and securing territory.
• After 19 hours: rediscovers standard human openings (star-point, 3-4 point).
• After 36 hours: surpasses AlphaGo Lee (the version that defeated Lee Sedol 4-1).
• After 72 hours: defeats AlphaGo Lee 100–0, using only 4 TPUs vs. 48 TPUs.
• After 40 days: achieves an Elo rating of 5,185, surpassing AlphaGo Master (4,858) which had defeated top professionals 60–0 online.

The final system used only a single machine with 4 TPUs. The original AlphaGo Lee was distributed across many machines with 48 TPUs. Superior performance with 1/12th the compute — that's the power of starting from scratch.

Perhaps the most fascinating aspect of AlphaGo Zero is what it learned, not just how well it played. During training, the system independently rediscovered Go concepts that took humans thousands of years to develop: joseki (corner patterns), life-and-death patterns, influence, thickness, and territory. It went through recognizable phases of Go knowledge: early game focused on local fights, then gradually developing whole-board strategy.

But it didn't stop there. AlphaGo Zero also discovered novel strategies that challenged conventional wisdom — moves that human experts initially dismissed as mistakes but later recognized as genuinely superior. It preferred some openings that humans rarely play and discarded others that humans consider standard.

This is the deepest lesson of the paper: a system with no human bias can discover knowledge that human bias actively prevents us from seeing. Centuries of accumulated Go wisdom, while mostly correct, also contained blind spots — strategies dismissed by tradition that were actually optimal.

The paper carefully ablates each design choice to show it matters:

• Dual network vs. separate networks: The single dual-headed network (dual-res) significantly outperforms using separate policy and value networks (sep-res), even though the separate value network predicts expert moves slightly better. Why? Because shared features create a synergy — learning what moves are good also helps learn who's winning.
• Residual vs. plain convolutional: Residual blocks dramatically improve both prediction accuracy and playing strength, especially as depth increases. Without skip connections, deep networks can't effectively learn.
• No human data vs. human data: The tabula rasa version (AlphaGo Zero) ultimately surpasses a version trained with human data (AlphaGo Master). Human data gives a head start but introduces bias that the system must later unlearn.

AlphaGo Zero showed that the self-play + MCTS + neural network recipe is domain-general. AlphaZero proved it by mastering three different games with the same code. The principle continues to influence reinforcement learning, robotics, and scientific discovery today.