AlphaZero: Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

Architecture deep dive

AlphaZero uses a single deep residual network with two output heads: 20 residual blocks, 256 filters throughout.

Input encoding (game-specific):
  Chess:  8×8×119 planes
  Shogi:  9×9×362 planes
  Go:     19×19×17 planes

→ Conv 3×3, 256 filters, batch norm, ReLU
→ 19 residual blocks (Conv→BN→ReLU→Conv→BN→skip→ReLU)
→ Split into two heads:

Policy Head: Conv 1×1 → BN → ReLU → move planes → softmax
Value Head:  Conv 1×1 → BN → ReLU → FC(256) → ReLU → FC(1) → tanh

Input planes — why the numbers differ

Each “plane” is a binary grid the size of the board encoding one specific fact:

Go’s 17 planes vs. AlphaGo’s 48 is a massive simplification — no hand-crafted features (liberties, ladders, ko). The network learns to compute whatever features matter from raw board state.

Continuous training

AlphaGo Zero trained in discrete cycles (generate games → train → evaluate → repeat). AlphaZero uses continuous training:

Self-play actors: Always generating games with latest network
Trainer: Always training on most recent games
No explicit evaluation step — network always updating

This is simpler and faster — no checkpoint selection overhead. The risk: a bad update could produce bad self-play data that reinforces the bad update (a death spiral). In practice this doesn’t happen because updates are small relative to existing knowledge.

Training infrastructure

The Stockfish controversy

The original December 2017 preprint drew significant criticism:

Chess opening preferences

Without any opening book, AlphaZero develops its own repertoire through self-play. It almost never plays 1.e4, preferring 1.c4 (English Opening) and 1.d4 systems — slower, positional openings where long-term strategic understanding dominates over tactical calculation.

MCTS — the PUCT formula

At each node during tree traversal, AlphaZero selects the action maximizing:

a = argmax [ Q(s,a) + c_puct · P(s,a) · sqrt(ΣN(s,b)) / (1 + N(s,a)) ]

Q(s,a)   = mean value of simulations through this action (exploitation)
P(s,a)   = prior probability from policy network (learned intuition)
N(s,a)   = visit count for this action
c_puct   = exploration constant

Early in search (low N): the prior P(s,a) dominates — network intuition guides exploration. Late in search (high N): Q(s,a) dominates — empirical results override intuition.

Dirichlet noise — forcing exploration

At the root node only, noise is added to the prior probabilities:

P(s,a) = (1 - ε) · p_a + ε · η_a    where η ~ Dir(α), ε = 0.25

75% network prior + 25% random noise. The α parameter is game-specific:

Small α produces sparse noise (boosting a few moves). Large α produces uniform noise. In Go (250 legal moves), you want sparse exploration; in chess (30 moves), denser noise is fine.

Temperature scheduling

Move selection: π(a|s) = N(s,a)^(1/τ) / Σ N(s,b)^(1/τ)

First 30 moves:  τ = 1.0  (proportional to visit count — exploratory)
After move 30:   τ → 0   (approaches argmax — greedy)
Competitive play: τ → 0   (always greedy)

Early-game diversity prevents self-play from collapsing into a single opening line. After move 30, positions are unique enough for greedy play.

Training dynamics — what AlphaZero learns when

Chess Elo curve:
  Hours 0-1:  Random → discovers material values       (~0 → ~1000 Elo)
  Hours 1-2:  Piece development, king safety, tactics  (~1000 → ~2000)
  Hours 2-4:  Positional concepts, pawn structure      (~2000 → ~3000, surpasses Stockfish)
  Hours 4-9:  Refinement, subtle endgames              (~3000 → ~3400+, plateaus)

AlphaZero rediscovers centuries of human chess knowledge in about 3 hours. The remaining 5 hours are refinement.

Concepts AlphaZero independently discovers

GM Matthew Sadler and WIM Natasha Regan analyzed thousands of AlphaZero’s games for Game Changer (2019):

Limitations

• Perfect information only — cannot play poker (hidden cards), StarCraft (fog of war)
• Two-player zero-sum only — no cooperative, multi-player, or negotiation games
• Deterministic only — no backgammon (dice), no card draws
• No explainability — outputs V(s)=0.72 but can’t say why the position favors White

AlphaZero’s legacy — how it changed chess engines

In August 2020, Stockfish integrated NNUE (Efficiently Updatable Neural Network) — a neural network evaluation originally developed for shogi. The result: +90 Elo from swapping the evaluation function alone.

Before NNUE (Stockfish 11):  hand-crafted eval + alpha-beta search
After NNUE (Stockfish 12+):  neural network eval + alpha-beta search (NOT MCTS)

The hybrid: neural evaluation + classical search

Stockfish didn’t adopt MCTS. It kept alpha-beta search but replaced hand-crafted evaluation with a neural network — taking AlphaZero’s key insight (learned evaluation) without its search algorithm. Modern Stockfish is stronger than AlphaZero ever was. AlphaZero’s contribution was proving neural evaluation is superior, which the traditional engine community then integrated.

Leela Chess Zero — the open-source clone

Since AlphaZero was never open-sourced, the community built Leela Chess Zero (Lc0) — a faithful reimplementation trained through distributed volunteer computing. Lc0 has won multiple TCEC seasons, trading titles with Stockfish and pushing both engines far beyond where AlphaZero left off.

Impact on human chess

For the first time, grandmasters studied computer games for strategic inspiration. Previous engines played “correct but uninspiring” chess. AlphaZero’s games were different — GM Sadler and WIM Regan spent a year analyzing them for Game Changer (2019).

• Initiative over material — top players became more willing to sacrifice for dynamic compensation
• 1.d4 resurgence — AlphaZero’s preference influenced both engine and human opening choices
• Exchange sacrifice revival — giving rook for minor piece + positional compensation, previously rejected by engines

The Nim blind spot (2026)

A March 2026 paper (Zhou & Riis, Queen Mary University of London) tested AlphaZero-style self-play on Nim — a simple game with a known perfect strategy based on XOR of heap sizes.

From AlphaZero to MuZero

AlphaZero still requires knowing the rules of the game. MuZero (2019 preprint, 2020 Nature) removes this final crutch:

MuZero learns three networks: representation (observation → hidden state), dynamics (state + action → next state + reward), and prediction (state → policy + value). It achieves comparable performance to AlphaZero on chess, shogi, and Go — without knowing the rules — and masters 57 Atari games.

The meta-lesson — System 1 + System 2

AlphaZero’s deepest insight isn’t about games. It’s about the relationship between fast intuition and slow deliberation:

System 1 (fast):  Neural network — instant pattern recognition
                  “This position looks good for White”

System 2 (slow):  MCTS — deliberate search through possibilities
                  “Let me verify by exploring move sequences…”

Neither alone is sufficient. Pure intuition (network without search) plays strong amateur chess. Pure deliberation (search without network) plays intermediate chess. Together: superhuman. This dual-process paradigm — learning provides intuition, search provides verification — may be the most important architectural insight in modern AI.

Scorecard