Chameleon: Mixed-Modal Early-Fusion Foundation Models

The training recipe — 10 trillion tokens

Total: ~10 trillion tokens — 5× more data than Transfusion’s 2T. The interleaved data is crucial: this teaches the model to naturally weave text and images. Most prior work only trained on paired data (one image, one caption).

Two model sizes

The unified vocabulary

Text and image tokens live in the same embedding space. The model doesn’t know it’s switching modalities — it just predicts the next token from a ~73K vocabulary.

VQ-VAE image tokenizer (Make-A-Scene heritage)

Chameleon uses a VQ-VAE derived from Meta’s Make-A-Scene image tokenizer:

Encoding:
  Input image (512×512×3)
    ↓ Encoder CNN
  Latent grid (32×32×256)
    ↓ Quantize each position → nearest of 8,192 codebook entries
  Token grid (32×32) = 1,024 discrete tokens
    ↓ Flatten row-by-row
  Token sequence [t_1, t_2, ..., t_1024]

Decoding:
  Token sequence → codebook lookup → latent grid → Decoder CNN → image

The VQ-VAE is frozen during Chameleon training. Its reconstruction quality caps the model’s generation quality:

Training stability — the hardest problem

This is arguably Chameleon’s most important contribution. Standard transformer training diverges ~20% into mixed-modal training.

Problem 1: Softmax attention explodes. When training on mixed modalities, the norms of Q and K vectors grow unboundedly. Image tokens and text tokens produce very different activation magnitudes. Once norms get large enough, softmax saturates (one weight → 1.0, all others → 0.0) and gradients vanish. Training collapses.

Problem 2: Layer norm placement. Standard Pre-Norm isn’t enough. Chameleon uses a revised Pre-Norm with QK-Norm applied after the query/key projections. The combination of Pre-Norm + QK-Norm prevents gradient explosions.

Problem 3: Image-text loss ratio instabilities. During training, image loss and text loss oscillate in anti-correlation — when image loss drops, text loss spikes, and vice versa. The modalities compete for model capacity. Fix: careful data scheduling with changing ratios during training (more text early, more interleaved later).

Inference: generating mixed documents

Text generation: Identical to any language model — autoregressive next-token prediction.

Image generation: When the model predicts <image_start>, it generates 1,024 image tokens autoregressively — each sampled from the 8,192-entry image vocabulary using the same softmax as text. No diffusion, no iterative denoising. Just next-token prediction, 1,024 times. Then the tokens are decoded through the VQ-VAE decoder into pixels.

Interleaved generation: The model decides when to insert images based on context, generates them inline, then continues text conditioned on everything before.

Head-to-head: Chameleon vs Transfusion

Benchmarks deep dive

Text benchmarks (Chameleon-34B):

Competitive but doesn’t beat text-only specialists. Small “multimodal tax” — image training doesn’t destroy text capability, but doesn’t help either.

Image captioning (where Chameleon shines):

State-of-the-art on image captioning — early fusion helps the model deeply understand images.

Mixed-modal human evaluation:

Limitations

Mathematical formulations

The unified objective. Chameleon’s beauty is its simplicity — one loss for everything:

L = -Σ_{i=1}^{N} log P_θ(x_i | x_{<i})

Where x_i can be a text token OR an image token. Same cross-entropy, same softmax, same backpropagation path. Compare to Transfusion’s dual loss: L_LM + λ · L_DDPM. No balancing hyperparameter λ, no noise scheduling, no timestep conditioning.

VQ-VAE quantization — the full math

Given encoder output z_e and codebook E = {e_k} with K = 8,192 entries:

Quantization (forward):
  z_q(i,j) = e_{k*}  where  k* = argmin_k ||z_e(i,j) - e_k||_2

Straight-through estimator (gradient hack):
  z_q = z_e + sg(z_q - z_e)

  Forward: equals z_q
  Backward: gradient flows through z_e only (pretend quantization didn't happen)

The argmin is non-differentiable. The straight-through estimator (STE) copies the gradient from z_q directly to z_e — mathematically unjustified but empirically works.

VQ-VAE training loss (3 terms):

L_VQ-VAE = ||x - D(z_q)||^2       // reconstruction: make output look like input
         + ||sg[z_e] - z_q||^2     // codebook: move codebook entries toward encoder outputs
         + β||z_e - sg[z_q]||^2   // commitment: prevent encoder from drifting from codebook

In practice, Chameleon uses EMA codebook updates instead of the gradient-based codebook loss:

e_k ← γ · e_k + (1 - γ) · mean(z_e mapped to k)    (γ = 0.99)

Codebook utilization and collapse

With K = 8,192 entries, typical utilization is only 40–70%. Thousands of entries go unused due to:

• Rich-get-richer dynamics: Popular codes attract more encoder outputs → get updated more → become even more popular
• Dead codes: Entries initialized far from data never get selected
• Redundant codes: Multiple entries converge to similar vectors

Mitigations: code reset (replace dead codes with sampled encoder outputs), EMA decay, entropy regularization. Even with mitigations, the effective information capacity per position is less than the theoretical log₂(8192) = 13 bits.

QK-Norm — the full formulation

Standard multi-head attention computes:

attn_logits = Q · K^T / √(d_h)

When ||Q|| and ||K|| grow large (which happens with mixed modalities), logits explode → softmax saturates → one-hot attention → gradient vanishing.

Chameleon’s fix:

Q_hat = Q / ||Q||_2      // normalize to unit vector
K_hat = K / ||K||_2      // normalize to unit vector

attn_logits = τ_h · Q_hat · K_hat^T / √(d_h)

where τ_h is a learnable temperature per head

Now dot products are bounded to [-1, 1], and τ_h controls sharpness: higher τ = sharper focus on specific positions, lower τ = broader attention.

QK-Norm appeared in ViT-22B and nGPT, but Chameleon’s contribution is proving it’s essential for mixed-modal early fusion at scale — make-or-break, not nice-to-have.

Training dynamics — the modality tug-of-war

Text loss and image loss exhibit anti-correlated oscillations. When gradient updates optimize for image prediction, shared weights shift toward image-favorable representations — text prediction temporarily suffers, and vice versa.

Management: data ratio scheduling (more text early, more interleaved later), gradient norm monitoring, and a two-stage training process (pre-training on all modalities, then alignment with curated safety-filtered data).

Pseudocode: training step

def chameleon_train_step(batch, model, vqvae, optimizer):
    total_loss = 0
    for document in batch:
        token_sequence = []
        for element in document:
            if element.type == "text":
                tokens = bpe_tokenize(element.text)
                token_sequence.extend(tokens)
            elif element.type == "image":
                with torch.no_grad():  # VQ-VAE is frozen
                    z_e = vqvae.encoder(element.pixels)
                    indices = vqvae.quantize(z_e)  # [32,32] ints
                    img_tokens = indices.flatten().tolist()
                img_tokens = [t + TEXT_VOCAB_SIZE for t in img_tokens]
                token_sequence.append(IMAGE_START)
                token_sequence.extend(img_tokens)
                token_sequence.append(IMAGE_END)

        # Standard causal LM - no special attention mask needed
        input_ids  = token_sequence[:-1]
        target_ids = token_sequence[1:]
        logits = model(input_ids)  # [seq_len, 73828]

        # ONE cross-entropy loss over ALL positions
        loss = cross_entropy(logits, target_ids)
        total_loss += loss

    total_loss.backward()
    clip_grad_norm_(model.parameters(), max_norm=1.0)
    optimizer.step()

Note the simplicity compared to Transfusion’s training step: no noise sampling, no timestep conditioning, no dual losses.

Pseudocode: QK-Norm attention

class QKNormAttention(nn.Module):
    def __init__(self, d_model, n_heads):
        self.d_h = d_model // n_heads
        self.W_Q = nn.Linear(d_model, d_model, bias=False)
        self.W_K = nn.Linear(d_model, d_model, bias=False)
        self.W_V = nn.Linear(d_model, d_model, bias=False)
        self.W_O = nn.Linear(d_model, d_model, bias=False)
        self.tau = nn.Parameter(torch.ones(n_heads, 1, 1))

    def forward(self, x, causal_mask):
        Q = self.W_Q(x)  # project
        K = self.W_K(x)
        V = self.W_V(x)
        # Reshape to [B, heads, S, d_h] ...
        Q = F.normalize(Q, dim=-1)  # unit vectors
        K = F.normalize(K, dim=-1)
        logits = self.tau * (Q @ K.T) / sqrt(self.d_h)
        logits = logits.masked_fill(~causal_mask, -inf)
        weights = softmax(logits, dim=-1)
        return self.W_O(weights @ V)

Pseudocode: image generation at inference

def generate_image(model, text_context, vqvae, temp=0.9, top_p=0.95):
    tokens = tokenize(text_context) + [IMAGE_START]
    for i in range(1024):
        logits = model(tokens)[-1]
        # Mask out text tokens - only sample from image vocab
        logits[:TEXT_VOCAB_SIZE] = -inf
        probs = softmax(logits / temp, dim=-1)
        # Nucleus (top-p) sampling
        sorted_p, sorted_idx = sort(probs, descending=True)
        cumsum = cumulative_sum(sorted_p)
        mask = (cumsum - sorted_p) > top_p
        sorted_p[mask] = 0
        sorted_p /= sorted_p.sum()
        next_token = sorted_idx[multinomial(sorted_p, 1)]
        tokens.append(next_token)
    tokens.append(IMAGE_END)

    # Decode via VQ-VAE
    img_indices = [t - TEXT_VOCAB_SIZE for t in tokens[-1025:-1]]
    indices = tensor(img_indices).reshape(32, 32)
    with torch.no_grad():
        z_q = vqvae.codebook_lookup(indices)
        image = vqvae.decoder(z_q)
    return image

Critical analysis

Expert verdict: Chameleon vs Transfusion

Chameleon, alongside Transfusion, kicked off a wave of unified multimodal models. Here’s what it sparked and what comes next.

What happened since May 2024

The evolving landscape

May 2024: Chameleon               Aug 2024: Transfusion
   |  "Tokenize everything"          |  "Continuous + diffusion"
   v                                  v
Sep 2024: Emu3                    Jan 2025: JanusFlow
"Better tokenizer fixes it"       "Decoupled encoders + flow"
   |                                  |
   +----------------+-----------------+
                    |
                    v
             2025-2026: Convergence

   Emerging consensus:
   1. Separate encoders for understanding vs generation
   2. Better tokenizers (continuous OR high-quality discrete)
   3. Unified transformer backbone
   4. Training stability tricks (QK-Norm) are essential

Improvement vectors

1. Fix the tokenizer

Chameleon’s 8,192-entry VQ-VAE with 40–70% utilization is the biggest bottleneck. Paths: larger codebook (32K–64K), SBER-MoVQGAN (Emu3’s approach), Finite Scalar Quantization (FSQ — eliminates codebook collapse by construction), or Lookup-Free Quantization (LFQ — exponential codebook without explicit entries).

2. Decouple understanding and generation encoders

One VQ-VAE serving double duty is a forced compromise. Use SigLIP/CLIP for understanding (optimized for semantics), VQ-VAE/VAE for generation (optimized for pixel reconstruction), share the transformer. Janus-Pro proved this works.

3. Reduce tokens per image

1,024 tokens per image is expensive (quadratic attention, sequential generation). Paths: higher compression VQ-VAE (256 tokens), hierarchical coarse-to-fine (64 + 256), variable-length tokenization (simple images get fewer tokens), or multi-scale with upsampler. Dropping to 256 tokens would make Chameleon compute-competitive with Transfusion.

4. Video generation

At 1,024 tokens per frame and 24 fps: 1 second = 24,576 tokens, 10 seconds = 245,760 tokens. Computationally intractable with current sequence lengths. Needs: temporal compression (1 token-set per keyframe), 3D VQ-VAE for video volumes, sparse attention across frames.

5. Scale to 70B+

Does the multimodal tax shrink at larger scale? Hypothesis: yes — at 70B+, the transformer has enough capacity that text and image objectives stop competing. Only a handful of labs can attempt this.

Scorecard

Compared to Transfusion