AlphaGo: Mastering the Game of Go with Deep Neural Networks and Tree Search

Policy network architecture

A 13-layer convolutional neural network:

Input: 19×19×48 tensor
  └ 19×19 board (stone positions)
  └ 48 feature planes (liberties, ko, recency, color, edge distance)

→ Layer 1: 5×5 filters, 192 channels
→ Layers 2–12: 3×3 filters, 192 channels
→ Layer 13: 1×1 filter → 1 channel
→ Softmax over 361 positions
→ Output: P(a|s) — probability of playing each intersection

No pooling layers — spatial resolution must be preserved because the output is a move on the board. The 48 input features encode domain knowledge: liberties, ladder status, stone recency, ko rules. This is significant manual feature engineering despite the “deep learning” framing.

Supervised learning phase

Training data: 30 million position-move pairs from 160,000 games on the KGS Go server (6–9 dan amateur and professional games).

57% sounds low, but when expanded to top-5 predictions, accuracy exceeds 85%. Multiple moves are often equally strong at any given position.

Fast rollout policy

A separate lightweight model for fast game simulations during MCTS:

Reinforcement learning phase

Starting from the SL policy, RL fine-tunes by self-play using the REINFORCE algorithm:

Repeat:
  1. Select opponent from pool of past RL versions
     (random selection prevents overfitting to one opponent)
  2. Play a full game: current RL policy vs. selected opponent
  3. Reward: z = +1 if won, -1 if lost
  4. Update: increase probability of moves that led to wins,
             decrease moves that led to losses

The opponent pool prevents policy collapse — co-evolving into a narrow strategy that beats only itself.

Value network

Same architecture as the policy network but outputs a single scalar via tanh (−1 to +1). Critically trained on self-play data with one position per game to avoid overfitting:

A single forward pass (~3ms) achieves accuracy comparable to averaging ~10,000 rollouts (~20 seconds) — a ~7,000× speedup.

MCTS details

At each node, the selection formula balances exploitation and exploration (PUCT):

a = argmax[ Q(s,a) + c · P(s,a) · sqrt(sum_b N(s,b)) / (1 + N(s,a)) ]

Q(s,a) = average value from simulations (exploitation)
P(s,a) = prior from policy network (guides exploration)
N(s,a) = visit count (more visits = less exploration bonus)

Leaf evaluation blends value network and rollout: V(s) = (1−λ)·v(s) + λ·z, where λ=0.5.

Subtle detail: MCTS uses the SL policy (not the stronger RL policy) for tree priors. The SL policy models a broader distribution of reasonable human moves, providing better exploration diversity. The RL policy is too peaked toward its own play style.

Ablation — what each component contributes

No single component is sufficient — the synergy is the breakthrough.

Why REINFORCE?

REINFORCE is the simplest policy gradient algorithm — and notoriously high-variance. Why not PPO or A2C?

• PPO didn’t exist yet (published 2017)
• The policy was warm-started from SL — RL only needs to fine-tune, reducing variance concerns
• Binary win/loss rewards eliminate the need for complex reward shaping

The REINFORCE update with baseline: Δθ = α · Σ_t (z_t − v̅) · ∇θ log πθ(a_t|s_t)

The baseline v̅ (typically the value function estimate) reduces variance by crediting moves that performed better than expected, rather than all moves in winning games.

Opponent pool — implicit population-based training

The RL policy maintains a pool of past checkpoints. Each game is played against a random past version, not the current self. Without this:

• The policy develops a narrow strategy exploiting its own weaknesses
• It never learns to handle diverse play styles
• A human playing an unexpected opening could exploit the gap

The pool forces robustness against diverse opponents — an implicit form of population-based training.

Value network overfitting — the full experiment

General principle: When training examples are temporally correlated (adjacent timesteps, consecutive frames, nearby sentences), the network can memorize sequential patterns rather than learning the underlying function. Breaking correlation forces genuine learning. This applies to video understanding, RL, and any sequential prediction task.

Virtual losses in distributed MCTS

The distributed version runs many MCTS threads in parallel. Problem: without coordination, multiple threads explore the same promising branch simultaneously.

Thread 1 starts exploring move A:
  Q(s, A) temporarily decreased
  N(s, A) temporarily increased
  → Other threads see move A as less attractive
  → They explore moves B, C, D instead
  → Thread 1 finishes, removes virtual loss, updates real value

This spreads threads across different parts of the tree without explicit communication — an elegant lock-free parallelism technique.

Time management

AlphaGo uses adaptive time allocation based on position complexity:

• Simple positions (policy confidence >90%): fewer simulations, move quickly
• Complex positions (high policy entropy): more simulations, think carefully
• Critical positions (search instability — best move keeps changing): extend time up to 4× normal

The symmetry trick

Go boards have 8-fold symmetry (4 rotations × 2 reflections). Each network evaluation randomly selects one of 8 transformations, effectively averaging over symmetries without extra compute. During training, all 8 symmetries are used for data augmentation.

Distributed architecture

Master (search control)
  └ 1,920 CPUs: rollouts, tree traversal
  └ 280 GPUs: policy + value network evaluation
  Asynchronous pipeline keeps both fully utilized

~850 Elo improvement from distribution alone — same networks, more search. Equivalent to 2–3 professional ranks.

Historical convergence

AlphaGo required the convergence of several advances: deep CNNs (2012–2015), GPU computing maturity, MCTS (Coulom, 2006), the UCT formula (Kocsis & Szepesvári, 2006), move prediction from CNNs (Clark & Storkey, 2014), and batch normalization (2015). Without any single one, AlphaGo couldn’t have been built.

What AlphaGo got wrong

Human data dependency: The pipeline starts with 30M human expert games, creating a ceiling effect (inherits human blind spots), domain specificity (48 hand-crafted features don’t transfer to chess), limited data availability, and a bootstrap problem (needs human experts to exist first).

Rollout weakness: The fast rollout policy (24.2% accuracy) is most unreliable in tactically sharp positions where one wrong move changes the outcome — precisely when evaluation matters most. AlphaGo Zero later dropped rollouts entirely, and performance improved.

Compute: The Lee Sedol match system used 1,920 CPUs + 280 GPUs, estimated at over $1M total. A research demonstration, not a scalable product.

The AlphaGo family tree

AlphaGo (2016, Nature)
  Human data + hand-crafted features + rollouts + value + policy
  └→ AlphaGo Zero (2017, Nature)
  │     NO human data, NO features, NO rollouts. Beat AlphaGo 100-0.
  └→ AlphaZero (2017/2018, Science)
  │     Generalized to chess + shogi. Same algorithm, three games.
  └→ MuZero (2019/2020, Nature)
        Doesn’t know the rules. Learns world model from interaction.

Each step removed a dependency: human data → game specificity → game rules. AlphaGo Zero surpassed the Lee Sedol version after approximately 3 days (~72 hours) of pure self-play training, proving the human data was not just unnecessary — it was a bottleneck.

Beyond games — the template

AlphaGo’s architecture (neural network evaluation + tree search) became a blueprint for diverse domains:

The insight: any problem frameable as sequential decisions over evaluable states can benefit from the AlphaGo paradigm.

The creativity question — Move 37

AlphaGo’s Move 37 in Game 2 — estimated by the policy network at 1-in-10,000 probability of being played by a human — sparked debate about machine creativity. It emerged from RL self-play, not human imitation. Professional commentators called it a mistake before realizing it was brilliant.

Lee Sedol’s counter-punch — Move 78

In Game 4, Lee Sedol played Move 78 — a wedge that the policy network also estimated at 1-in-10,000 probability. It exploited AlphaGo’s coverage problem: moves with near-zero policy priors are never explored deeply by MCTS, even if they’re brilliant. AlphaGo’s response was erratic. Lee Sedol won decisively.

The search-learning tradeoff

Pure learning (no search):  Fast but limited by network capacity. Strong amateur.
Pure search (no learning):  Slow but theoretically optimal. Intermediate.
Learning + Search:          Search amplifies learned knowledge. Superhuman.

Neural networks provide fast, approximate intuition. Search provides slow, precise verification. The combination is greater than either alone. This principle carries through to AlphaFold, AlphaTensor, AlphaGeometry, and beyond.

Historical significance

The Lee Sedol match (March 2016) generated 200 million Weibo pageviews and tens of millions of live stream viewers. Lee Sedol retired from professional Go in November 2019, citing AI’s dominance. The Ke Jie match (May 2017) saw AlphaGo win 3–0 against the world #1.

Final scorecard