DALL-E 3: Improving Image Generation with Better Captions

The captioner — building the foundation

DALL-E 3’s captioner is built on a CoCa-style (Contrastive Captioner) architecture — a vision-language model that combines two training objectives: contrastive learning (CLIP-style image-text matching) and autoregressive captioning (generating text descriptions token by token).

Image → Vision Encoder (ViT) → Cross-attention → Language Decoder → Caption

They took a pre-trained CoCa model and fine-tuned it on a curated dataset of high-quality, human-written image descriptions — not web-scraped alt-text, but detailed descriptions specifically collected for this purpose.

Two caption types

The descriptive captions capture: objects and attributes (wet coat, dark-gold), spatial relationships (stands on, behind the dog), counts (no other people or animals), style (low angle, eye level), and negatives (what’s NOT in the image).

Caption quality matters more than quantity

The captioner was trained on a relatively small curated dataset (hundreds of thousands, not billions). But because each example was a high-quality, detailed description, the captioner learned to generalize this level of detail to any image. Quality of supervision > quantity of supervision.

The three caption types in training

The training mix

They blend synthetic and original captions:

Training batch composition:
└─ 95% synthetic captions (long descriptive)
└─  5% ground truth (original web captions)

Why keep 5% ground truth? Two reasons:

1. Distribution matching. Users write short, casual prompts. Training only on long captions makes the model over-reliant on verbose descriptions and poor on short prompts.
2. Diversity signal. Web captions, despite being noisy, contain cultural context and conceptual associations that synthetic captions miss.

What the captioner learns

Spatial awareness:
  Web caption:   “family photo”
  Synthetic:     “Three adults stand in a row. A woman in a blue
                  dress is on the left, a tall man in a gray suit
                  is in the center...”

Counting:
  Web caption:   “flowers”
  Synthetic:     “Seven sunflowers in a clear glass vase. Five
                  are fully bloomed, two are still partially closed.”

Text recognition:
  Web caption:   “street scene”
  Synthetic:     “A busy city street with a red stop sign
                  reading ‘STOP’. Behind it, a green street
                  sign reads ‘BROADWAY’.”

Prompt upsampling — GPT-4 at inference

User prompt: “a cat”
       ↓
   GPT-4 rewrite
       ↓
Expanded: “A fluffy orange tabby cat sits on a windowsill,
looking out at a rainy day. Soft natural light illuminates
its fur. The window frame is white-painted wood...”
       ↓
   DALL-E 3 diffusion model
       ↓
   Generated image

GPT-4 is instructed to: preserve user intent, add plausible details, be specific, and vary outputs. Users can opt out and see the rewritten prompt. This bridges the distribution gap between how users write and how the model was trained.

Evaluation framework

T2I-CompBench (automated) tests compositional generation:

Human preference studies: Side-by-side comparisons with SDXL, rated by human evaluators for quality and prompt following. DALL-E 3 achieved 71.7% preference.

De-emphasis of FID: FID measures distributional similarity to real images, not prompt following. A model can have great FID but still generate the wrong objects and colors. DALL-E 3 prioritized compositional accuracy and human preference over FID.

Captioner base model selection

The captioner isn’t trained from scratch. OpenAI starts with a pre-trained CoCa model and fine-tunes it:

The contrastive pretraining gives the vision encoder discriminative features — it knows the difference between a golden retriever and a labrador. Without it, the captioner writes fluent but less precise descriptions.

Annotation guidelines

Decoding strategy

At recaptioning time, the captioner uses nucleus sampling (top-p = 0.9) with temperature τ = 0.7, rather than beam search:

Nucleus sampling’s adaptive nucleus size is key: when the model is confident, the nucleus shrinks (near-deterministic). When uncertain, it expands (more creative). Diversity in captions → diversity in what the generator learns.

The diffusion model

The paper is deliberately vague about the generator (proprietary), but the architecture is a U-Net with text cross-attention:

Noisy latent z_t
     ↓
┌─────────────────────────────┐
│         U-Net                  │
│  Down: [Conv → ResBlock → Attn] │
│  Bottleneck                     │
│  Up:   [Conv → ResBlock → Attn] │  ← Cross-attention to
│                                 │    text embeddings
└─────────────────────────────┘
        Predicted noise ε_θ

Training objective: Standard DDPM noise prediction:

L_diffusion = E[||ε - ε_θ(z_t, t, c)||²]

Where:
  z_0 = clean latent (image encoded by VAE)
  z_t = noisy latent at timestep t
  ε  = actual noise added
  c   = text conditioning (caption embedding)

The only change from DALL-E 2 is c. Same architecture, same loss, same noise schedule — but c went from garbage alt-text embeddings to rich descriptive caption embeddings. That’s the entire paper’s contribution to the diffusion model itself.

Classifier-Free Guidance (CFG)

At inference, DALL-E 3 uses CFG to amplify text conditioning:

ε̂_θ = ε_θ(z_t, t, ∅) + s · [ε_θ(z_t, t, c) - ε_θ(z_t, t, ∅)]

Run the model twice — once with the caption, once without. The difference is the “direction” the caption pushes the image. Scale s amplifies that direction (typically 7–15). Higher s = more faithful to the prompt but less diverse. During training, captions are randomly dropped ~10% of the time to enable this.

Recaptioning at scale

Quality filtering: Length filter (10–300 tokens), repetition filter, hallucination detection via CLIP alignment scores. Captions with alignment below a threshold are regenerated or replaced with the original ground truth.

Ablation studies

The generator–captioner flywheel

A better generator enables a better captioner through synthetic training pairs:

Step 1: Use generator to create images from detailed prompts
  Prompt → Generator → [perfectly-aligned image]
  
Step 2: Use (prompt, generated_image) pairs to fine-tune captioner
  Captioner learns precise descriptions from perfectly-aligned data

The cycle:
  Weak captioner → OK generator
  OK generator → better training pairs → better captioner
  Better captioner → better generator → ...

Catch: The flywheel can amplify errors. Real images in the mix (that 5% ground truth) keep the loop grounded in reality.

Safety measures

T2I safety classifier filters both prompts and generated images. Provenance metadata (C2PA) embedded in outputs. Prompt-level filtering prevents generation of public figures and harmful content.

DALL-E 3 triggered a paradigm shift. Every major lab adopted recaptioning within months. Here’s what it sparked and what comes next.

Failure modes — what DALL-E 3 still gets wrong

The recaptioning wave

DALL-E 3 established recaptioning as standard practice. Every subsequent SOTA model adopted it:

Architecture evolution post-DALL-E 3

U-Net → DiT (Diffusion Transformer)

DALL-E 3 era:
  Text encoder → U-Net with cross-attention → Image
  (Convolutional backbone, attention at select layers)

Post-DALL-E 3:
  Text encoder → Pure transformer (DiT) → Image
  (No convolutions, attention everywhere, scales better)

DiT replaces the U-Net with a pure transformer, scaling more predictably. SD3 and FLUX adopted this. DiT + recaptioning pushed quality beyond DALL-E 3.

DDPM → Flow Matching

DDPM (DALL-E 3):
  Forward: add noise gradually over 1000 steps
  Reverse: predict and remove noise over 1000 steps
  Path: curved, complex noise schedule

Flow Matching (SD3, FLUX):
  Forward: linear interpolation from data to noise
  Reverse: follow straight path back
  Path: straight lines, simpler, faster

Flow matching achieves comparable quality with fewer denoising steps (20–30 vs 50–100), making generation faster.

Single → Triple text encoders

DALL-E 3:     [Single text encoder] → cross-attention → U-Net

SD3:    [CLIP-L] ───┐
        [CLIP-G] ───&boxn;→ concatenate → cross-attention → DiT
        [T5-XXL] ───┘

FLUX:   [CLIP-L] ───┐
        [T5-XXL] ───┘→ concatenate → cross-attention → DiT

SD3 uses three encoders; FLUX uses two (dropping CLIP-G).
Each captures different aspects:
  CLIP-L:  visual concepts, style
  CLIP-G:  scene composition, global semantics (SD3 only)
  T5-XXL:  language understanding, long prompts

Open research frontiers

1. Autoregressive image generation

The hottest shift post-DALL-E 3 is moving away from diffusion entirely. Autoregressive models (Transfusion, Chameleon) generate image tokens left-to-right, like writing text. They naturally unify text + image generation with variable resolution. Both approaches still benefit from recaptioning — the data insight is architecture-agnostic.

2. The captioner becomes the model

DALL-E 3 treated the captioner as a preprocessing step. The frontier is making the captioner and generator the same model — a single unified model that can both understand and generate images (Gemini, GPT-4o, Transfusion). No separate pipeline, no distribution mismatch.

3. Video recaptioning

The same principle applies to video, but captions must describe temporal dynamics: actions, transitions, cause-and-effect, camera motion. A 10-second clip needs a caption covering what happens over time, not just a static description.

4. Compositional generation without cross-attention

5. Evaluation beyond human preference

No single metric captures “is this a good image for this prompt?” The field still relies on human evals for serious comparisons.

The DALL-E 3 legacy

“How do we make image generation better?”
  → Bigger model
  → Better architecture
  → More training compute
  → Better noise schedule

“How do we make image generation better?”
  → Better training data descriptions  ← new top priority
  → Then bigger model
  → Then better architecture

This mirrors the broader data-centric AI movement: for mature architectures, improving data quality yields more than improving the model. DALL-E 3 is the most dramatic demonstration in generative AI.

Lasting contributions (ranked)

1. Synthetic recaptioning as standard practice — now universal across all image/video generation
2. Data quality > model architecture — a clear, reproducible demonstration
3. Caption-generator flywheel — the virtuous cycle of using generators to improve captioners
4. Prompt upsampling via LLM — GPT-4 rewriting at inference, adopted by Midjourney, Ideogram, and others
5. Evaluation framework shift — de-emphasizing FID in favor of human preference and compositional benchmarks

Scorecard