On Evaluation of Embodied Navigation Agents

SPL under the hood

The formula has subtle design choices that matter:

SPL = (1/N) × Σᵢ Sᵢ × (ℓᵢ / max(pᵢ, ℓᵢ))

Sᵢ ∈ {0, 1}
  Binary. Agent either reached goal AND called DONE = 1.
  Otherwise = 0. No partial credit.

ℓᵢ = geodesic shortest path from start to goal
  Computed by the simulator (ground truth).
  This is the OPTIMAL path respecting walls/obstacles.

pᵢ = actual path length agent traveled
  Total distance, including backtracking, dead ends,
  circling. Every wasted meter counts against you.

max(pᵢ, ℓᵢ) — WHY THE MAX?
  Prevents the ratio from exceeding 1.0.
  Edge case: agent clips through a wall (collision)
  and takes a "shortcut" where pᵢ < ℓᵢ.
  Without the max, this cheating would score >1.
  The max clamps it to 1.0 at best.

CRITICAL PROPERTY:
  SPL ∈ [0, 1]
  SPL = 0  → total failure (never reaches goal)
  SPL = 1  → perfect (always reaches goal, optimal paths)

Why not just multiply success rate × average efficiency separately?

Per-episode weighting means an episode where the agent
wanders 10× the optimal path drags down the score MORE
than an episode where it's only 1.5× optimal.

Per-episode weighting prevents a few efficient successes
from masking many wildly inefficient ones.

Geodesic vs. Euclidean — why it changes everything

FLOOR PLAN:
  ┌─────────────────────────┐
  │           WALL           │
  │    ┌──────────────┐      │
  │    │              │      │
  │  A │              │  B   │
  │    │              │      │
  │    └──────┐       │      │
  │           │ door  │      │
  │           └───────┘      │
  └─────────────────────────┘

  EUCLIDEAN distance A→B: ~3 meters (through wall)
  GEODESIC distance A→B: ~12 meters (around wall, through door)

  With Euclidean: agent on the wrong side of a wall
  gets credit for being "close."
  With geodesic: agent must be REACHABLY close.

The DONE action — subtler than it looks

WITH DONE ACTION, the agent must:
  1. Navigate to the goal      (path planning)
  2. Recognize it has arrived   (perception/understanding)
  3. Decide to stop             (confidence/decision-making)

All three are required. Missing any one = failure.

SUCCESS THRESHOLD τ:
  Default: τ = 2 × agent body width
  For body width 0.2m → τ = 0.4m
  For AreaGoal: center of mass inside target area

The four simulator environments

SYNTHETIC (SUNCG, AI2-THOR):
  + Unlimited variations, perfect geometry
  + Interactive objects possible (AI2-THOR)
  - May not reflect real-world complexity
  - "Sim-to-real gap" when deploying to robots

REAL SCANS (Matterport3D, Gibson):
  + Authentic visual complexity and clutter
  + Real lighting, textures, layouts
  - Fixed environments (can't generate new ones)
  - Scanning artifacts (holes, stitching errors)

The exploration–navigation trade-off

Agent gets a "budget" before navigation episodes:

  Budget = 0m    → zero-shot (Regime 1)
  Budget = 500m  → brief exploration
  Budget = 1000m → moderate exploration
  Budget = 2000m → extensive exploration
  Budget = ∞     → full memorization

  SPL
  1.0 ┤
      │                          ●———— ceiling
  0.8 ┤                    ●——
      │               ●——
  0.6 ┤          ●——
      │     ●——
  0.4 ┤●——
      │
  0.2 ┤
      │
  0.0 ┼——┬——┬——┬——┬——┬—
      0   500  1000 1500 2000  ∞
           Exploration Budget (meters)

THIS CURVE is the real evaluation, not a single number.
Different agents may dominate at different budgets.

Agent architecture spectrum

LEVEL 0: PURELY REACTIVE
  Current frame → Deep Network → Action
  No memory. Can't build a mental map.

LEVEL 1: SHORT-TERM VECTORIAL MEMORY (LSTM/GRU)
  Recurrent state carries compressed history.
  Can remember "I tried going left already."
  But fixed-size vector — lossy.

LEVEL 2: RICH INTERNAL REPRESENTATIONS
  Agent builds explicit spatial maps, topologies,
  semantic labels. Can plan paths through its model.

The paper deliberately doesn't recommend an architecture —
it highlights that internal representation IS the core
research question.

SPL’s blind spots — what the standard metric hides

SPL became universal. But universality breeds complacency — and SPL has real limitations the field only discovered by using it for years:

BLIND SPOT 1: ALL FAILURES LOOK THE SAME
  Agent stops 0.1m too early from goal → Sᵢ = 0
  Agent wanders aimlessly for 500m → Sᵢ = 0
  SPL score for both: 0. These are fundamentally different.

BLIND SPOT 2: PATH QUALITY IS ONE-DIMENSIONAL
  Agent takes optimal-LENGTH path but scrapes every wall,
  spins 360° at every intersection, moves in jerky bursts.
  SPL: 1.0 (perfect!). You’d never deploy this robot.

BLIND SPOT 3: NO DYNAMICS
  Agent A reaches goal in 30 seconds.
  Agent B reaches goal in 300 seconds (same path length).
  SPL: identical. Time is invisible.

BLIND SPOT 4: THRESHOLD SENSITIVITY
  Agent consistently stops 0.39m from goal.
  With τ = 0.4m → success rate: 100%
  With τ = 0.35m → success rate: 0%

These blind spots spawned follow-up metrics:

• SoftSPL: Replaces binary Sᵢ with continuous proximity score. Near-misses get partial credit.
• SCT (2021): Success weighted by Completion Time. Penalizes slow agents even with short paths.
• SPL Sweep: Report SPL across varying τ thresholds. Shows robustness to threshold choice.

The Habitat lineage — from paper to platform

  July 2018: This paper published
       ↓
  Dec 2019: Habitat 1.0 released (Facebook AI Research)
       ↓     Built by co-authors Savva, Malik + others
       ↓     Implements EVERY recommendation
       ↓
  2019: First Habitat Challenge — PointGoal focus
       ↓
  2020: PointGoal essentially SOLVED
       ↓     DD-PPO agent: 0.97 SPL on Gibson
       ↓     Near-optimal paths in unseen environments
       ↓     Zero-shot generalization regime
       ↓
  2020+: Challenge shifts to ObjectGoal
       ↓     SPL drops from 0.97 to ~0.3
       ↓     Adding recognition + world knowledge
       ↓     is MASSIVELY harder than pure path planning
       ↓
  2022+: MultiObjectNav, Social Navigation, Rearrangement
       ↓
  2023+: Foundation models enter navigation
         LLM/VLM planners with small RL execution policies

PointGoal solved — ObjectGoal still hard

WHY POINTGOAL WAS "EASY" (in retrospect):
  Agent always KNOWS where the goal is.
  Challenge is purely: find a collision-free path.
  No recognition. No world knowledge. No language.
  Deep RL + massive compute was enough.

WHY OBJECTGOAL IS STILL HARD:
  "Find the refrigerator" requires:
  1. Explore systematically (where to look?)
  2. Recognize the target (is that a refrigerator?)
  3. Use world knowledge (fridges are in kitchens)
  4. Reason about layout (kitchen is near dining room)
  5. Handle ambiguity (which refrigerator?)

  Adding ONE cognitive layer dropped performance by HALF.
  This is the navigation equivalent of "integration density"
  — coordinating multiple cognitive operations is
  fundamentally harder than any single operation.

The sim-to-real gap — the unsolved problem

Agent in simulation:  SPL = 0.95
Same agent on robot:  SPL = 0.30 (if you're lucky)

SOURCES OF THE GAP:

  VISUAL DOMAIN: Clean renders vs. noisy sensor data
  (motion blur, varying lighting, reflections)

  ACTUATION: move_forward(0.25m) → moves EXACTLY 0.25m
  in sim. Real robot: 0.23m, drifts 0.02m right, wheel slips.

  DYNAMICS: Agent is a perfect cylinder in sim.
  Real robot: complex geometry, shifting center of mass.

  ENVIRONMENT: Sim is frozen at scan time.
  Real world: furniture moves, people walk through,
  doors change state, lighting varies hour to hour.

Subtle memorization — deeper than train/test splits

Even with proper train/test splits, memorization leaks:

1. ENVIRONMENT-TYPE MEMORIZATION
   All training kitchens have fridges on south wall →
   agent "learns" to always go south for fridge.
   Works in training distribution. Fails in novel layouts.

2. SIMULATOR ARTIFACT MEMORIZATION
   Agent learns Matterport3D-specific stitching patterns
   as navigation cues. Useless in AI2-THOR or real world.

3. DISTRIBUTION MEMORIZATION
   Training: 80% of goals are 3–10m away.
   Agent calibrates to this range.
   Novel environments with longer paths → degraded SPL.

VLN — the fourth goal type

The paper mentioned natural language goal specification but didn’t standardize it. Peter Anderson (this paper’s first author) simultaneously created Room-to-Room (R2R), which combined natural language instructions with navigation:

"Walk past the dining table, turn left at the hallway,
 go through the second door on your right, and stop
 in front of the bathroom mirror."

This is HARDER because:
- Language is ambiguous ("second door" from which side?)
- Instructions reference landmarks requiring recognition
- Instructions may be wrong or imprecise
- Grounding: mapping words to visual percepts in real-time

Standards documents as field-shapers

ImageNet (2009): standardized image classification eval
  → catalyzed the deep learning revolution

This paper (2018): standardized navigation eval
  → catalyzed embodied AI progress

The meta-lesson: EVALUATION DRIVES RESEARCH DIRECTION.
Whatever you measure is what people optimize.
If the metric is wrong, the field goes sideways.
If the metric is right, the field accelerates.

What the paper got right and wrong (2018 → 2026)

✓ "SPL of 0.5 would represent good performance"
   Understated for PointGoal (hit 0.97), but prescient
   for ObjectGoal (still ~0.4–0.5 in 2026).

✓ "Internal representation is central"
   Became THE research question. The entire "world models"
   movement is about what representations agents build.

✓ "Standardization will catalyze progress"
   Habitat Challenge created a coordination mechanism
   that drove steady improvement year over year.

✓ "Sim-to-real deployment matters"
   Still the hardest problem. Their concern was justified.

✗ DIDN'T ANTICIPATE FOUNDATION MODELS
   The paper assumes agents trained FROM SCRATCH via RL
   in simulation. Nobody in 2018 imagined that a model
   trained on internet text + images could navigate at all.

✗ DIDN'T ANTICIPATE THE LANGUAGE SHIFT
   Natural language became the PRIMARY interface for
   embodied agents by 2024. VLN is now bigger than
   pure ObjectGoal research.

✗ UNDERWEIGHTED MULTI-AGENT SCENARIOS
   No mention of multiple agents, social navigation,
   or collaborative tasks. These are now central.

✗ THE SIMULATOR MONOCULTURE
   Standardizing on 4 environments created a
   "teaching to the test" effect — agents optimized
   for those specific environments rather than
   developing general navigation ability.

Foundation models vs. RL policies — the paradigm shift

THE 2018 PARADIGM (what this paper assumed):
  Train an RL agent FROM SCRATCH in simulation:
  Environment → RGB → CNN → LSTM → Policy → Action
  Millions of episodes. Months of GPU time.

THE 2024+ PARADIGM (what actually happened):
  Foundation model as "brain" + small RL policy for control:

  ┌————————————————————┐
  │    FOUNDATION MODEL         │
  │  (GPT-4V / Gemini / etc.)   │
  │                              │
  │  "I see a hallway with       │
  │   doors. The kitchen is      │
  │   likely ahead-left."        │
  └——————┬—————————————┘
               │ high-level plan
               ▼
  ┌————————————————————┐
  │    LOW-LEVEL RL POLICY       │
  │  (obstacle avoidance,        │
  │   motor commands)            │
  └————————————————————┘

WHY THIS CHANGES EVERYTHING:
  1. ZERO-SHOT GENERALIZATION — model already knows
     "refrigerators are in kitchens" from pretraining
  2. LANGUAGE IS NATIVE — VLN tasks need no special training
  3. COMMON SENSE — impossible for RL agents to learn
  4. SIM-TO-REAL GAP SHRINKS for perception
     (trained on real images, not renders)

THE CATCH: latency (2–5s per decision vs. <100ms needed),
cost, and hallucination risk.

The evaluation crisis beyond SPL

SPL was designed for one agent, one goal, one episode, static environment. Modern embodied AI needs to evaluate far more:

MULTI-STEP TASKS:
  "Go to kitchen, pick up red mug, bring to living room."
  SPL measures: did you reach the kitchen?
  Doesn't measure: did you grab the RIGHT mug?

SOCIAL NAVIGATION:
  "Navigate to exit without making people uncomfortable."
  SPL: did you reach the exit efficiently?
  Doesn't measure: did you violate personal space?

OPEN-ENDED EXPLORATION:
  "Explore this building and report what's here."
  No single goal. No success/failure binary.

CONTINUOUS OPERATION:
  Home robot that navigates ALL DAY.
  How do you measure a month of navigation?

The field needs integrated evaluation that tests navigation
+ manipulation + language + social reasoning together —
not as separate tasks, but as one combined challenge.

The world model connection

The paper’s Recommendation 7 (“internal representation is central”) anticipated today’s world models debate:

2018: "Internal representation" = LSTM hidden state.
      A black-box vector. We don't know what's in it.

2020: Spatial memory maps emerge.
      Agents build top-down 2D maps from egocentric views.

2022: Neural scene representations (NeRF-based).
      Encode 3D geometry + appearance from observations.

2024+: Foundation model "mental models."
       Semantic, not geometric.
       "I've seen a living room and two bedrooms.
        Kitchen is probably beyond the hallway."

THE LIKELY SYNTHESIS:
  Foundation model for HIGH-LEVEL reasoning
  (which room to visit, what strategy to use)
  +
  Geometric map for LOW-LEVEL execution
  (collision avoidance, precise path following)

Neither pure reasoning nor pure spatial computation
is optimal alone. The combination wins.

Connections to perception and generation

The paper’s goal taxonomy maps directly to both media perception and generation:

NAVIGATION GOALS AS A PERCEPTION STACK:
  PointGoal → Spatial perception
    (depth estimation, free-space segmentation)
  ObjectGoal → Object perception + world knowledge
    (detection, recognition, semantic segmentation)
  AreaGoal → Scene perception + layout understanding
    (room classification, spatial layout estimation)

VIDEO GENERATION NEEDS THE SAME CAPABILITIES:
  A video generator that produces coherent sequences
  must internally represent:
  - Consistent 3D geometry across frames
  - Object persistence through occlusion
  - Physics-aware motion
  - Navigable camera paths

  A video generator IS a world model.
  Generation = perception in reverse.

THE LOOP:
  Generation pretraining → better representations
  → better perception → better spatial reasoning
  → better world models → better generation

  It's a loop, not a pipeline.

The embodied AI frontier (2026)

SOLVED:
  ✓ PointGoal in simulation (0.97 SPL)
  ✓ Standard evaluation framework (this paper)
  ✓ High-quality simulators (Habitat, AI2-THOR)

PARTIALLY SOLVED:
  ◑ ObjectGoal (~0.5 SPL, improving)
  ◑ VLN (language-conditioned navigation)
  ◑ Foundation model integration

UNSOLVED:
  ✗ Sim-to-real transfer at scale
  ✗ Manipulation + navigation combined
  ✗ Social navigation (around people)
  ✗ Long-horizon multi-step tasks
  ✗ Continuous real-world operation
  ✗ Real-time foundation model planning

Scorecard