AI Researchers’ Perspectives on Automating AI R&D and Intelligence Explosions

The methodology: semi-structured qualitative interviews

This is a qualitative research study. The goal isn’t to count things or run statistical tests — it’s to surface reasoning patterns and capture why experts hold the views they do.

Sampling

182 researchers invited; 25 agreed (13.7% response rate, normal for elite-expert interviews). Three recruitment channels deliberately to capture different vantage points:

Participant mix: 7 from frontier labs, 4 ex-frontier-lab, 9 academics, 3 industry, 2 nonprofit. The stratification is what lets them compare clusters later.

Interview protocol

14 core questions, 40–60 minutes each, organized into three sections:

• Section A — Conceptualizing ASARA: what it’ll look like, trajectory, intelligence-explosion expectations
• Section B — Organizational dynamics: deployment decisions, internal discussions
• Section C — Risk assessment and governance: red lines, mitigations

Critical detail: when participants didn’t know a concept, the interviewer read a scripted definition. This standardizes the prompt so people react to the same idea.

Inductive coding + AI-assisted classification

The lead author developed codes inductively (grounded-theory style). For the categorical dot plots in Figures 1–4, the authors did something novel: they fed anonymized transcripts to Claude with structured prompts that forced classification into fixed codes. The full prompt is in Appendix B.

This makes the categorical results reproducible — but creates a methodological dependency on Claude that the paper acknowledges only partially.

The four figures — what they actually show

Figures recreated from the paper’s text. Position totals are exact (stated in the paper); per-affiliation dot placements are inferred from qualitative descriptions. Original figures at arxiv.org/html/2603.03338v2.

The “ideation vs execution” framework

The most analytically interesting frame. 15 of 25 participants spontaneously distinguished:

A subset reframed the ideation problem more sharply: it’s not generating ideas that’s hard, it’s validating them. Expert humans with decades of experience struggle to recognize good ideas. And ML models tend to learn the mode of training data, not exceptional cases — exactly the wrong inductive bias for spotting paradigm shifts.

Internal vs. public deployment — the argument map

Why labs will keep ASARA internal (top three codes)

1. Preserving competitive advantage (12 transcripts) — deploying gives competitors capability lifts
2. Limited compute (10) — every GPU serving the API isn’t accelerating internal R&D. One striking quote: $100k of compute used for R&D might be worth $1M in researcher salary equivalent
3. Avoiding diffusion (6) — public deployment enables distillation and reverse-engineering

Why labs will deploy publicly (top three)

1. Financial pressures (10) — labs need to raise $20B+/year
2. Government intervention (7) — regulators may force visibility
3. Business model and culture (6) — Meta’s open-source culture vs OpenAI’s more closed approach

Constraints discussed

Sixteen participants flagged binding constraints. Three came up repeatedly:

• Compute (11 mentions) — split between “engineering challenge, solvable” and “physical limit, real barrier”
• Data (5 mentions) — “data determines the ceiling of self-improvement” (P3)
• Bootstrapping level — the capability threshold required for sustained autonomous improvement. P1: “Can the model evaluate its own answers? Until relatively recently, the models just could not do that.”

Red lines vs transparency — comparing governance approaches

Red lines = bright-line rules that trigger major response when crossed (e.g. IDAIS Beijing: “No AI system should copy or improve itself without explicit human approval”).

Three implementation challenges identified even by supporters:

1. Specification problem — the more precisely you define a threshold, the more it diverges from the abstract risk you’re trying to capture
2. Verification & enforcement — “any kind of AI includes some intelligence and a compiler optimization” makes self-improvement hard to define
3. Timing — too early handicaps beneficial development; too late and it’s useless

Trade-off captured by P8: red lines are “the dumbest possible supervisor but the most trustworthy” — crude but transparent vs. sophisticated but discretionary.

Headline numbers

Since this is a qualitative interview study, L3 goes deep on methodology, the intelligence explosion theoretical literature this paper sits inside, and critical evaluation rather than equations and algorithms.

1. Qualitative methodology — what was done, what’s missing

Sampling architecture

40% of participants came through network/snowball channels including MATS connections (Krueger is a MATS supervisor; lead author is in the program). Snowball sampling is known to produce homophilous networks — even when you ask for disagreement, you get disagreement within your cluster. Likely shape of non-response bias: declines included skeptics who think the framing is uninteresting, systematically reducing the “skeptical academic” representation in the sample. This actually works against the headline schism finding, which is epistemically reassuring.

Inductive coding by a single author

Codes were developed inductively — grounded-theory style, letting concepts emerge. The Limitations section flags that inter-rater reliability was not calculated. The standard remedy is a second researcher independently coding a subset, with Cohen’s kappa or Krippendorff’s alpha. Values above ~0.8 are strong agreement; below 0.6 starts being concerning. This absence matters less for descriptive counts (“20 of 25 said X”) and more for interpretive claims like “a schism emerged.”

AI-assisted coding with Claude

For Figures 1–4, transcripts were fed to Claude with structured prompts forcing classification into fixed codes. Methodological strength: reproducible (anyone can re-run the prompt), removes manual coder bias, auditable via extracted quotes.

Three issues the paper doesn’t engage with:

• Single-LLM coder problem — structurally analogous to a single human coder. No triangulation across model families.
• Prompt-as-framing — forcing participants into one of four buckets is itself an interpretive act; the structure isn’t validated against a held-out human-coded set.
• Conflict-of-interest pattern — Anthropic researchers are in the sample, classified using an Anthropic product. A robustness check would have been appropriate.

2. The intelligence explosion theoretical lineage

Good 1966 — the syllogism

“An ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind.”

Three buried assumptions, each a fault line in later literature:

1. Intelligence is sufficiently coherent (more intelligence → better at AI design)
2. Marginal improvement is substantial, not trivial
3. The design process doesn’t hit external constraints (treats AI design as pure intellectual work)

Yudkowsky — return on cognitive investment

Yudkowsky’s Intelligence Explosion Microeconomics (2013) formalizes Good’s argument with the k-factor:

Mapping onto chain-reaction physics:

k < 1  →  subcritical    (improvements taper off)
k = 1  →  critical       (sustained linear improvement)
k > 1  →  supercritical  (each round produces more than it cost
                            → exponential explosion)

With I(t) the cognitive capability and R(t) the resources invested:

Yudkowsky’s key move: the empirical question isn’t “will AI improve?” (yes) but is k persistently > 1? Most natural processes have k < 1 — diminishing returns dominate.

This formalism also describes the “bootstrapping level” P1 mentioned: the capability threshold below which k < 1 (human time gates progress) and above which k > 1 (system sustains its own improvement).

Bostrom 2014 — the takeoff equation

Fast takeoff: optimization power grows faster than recalcitrance. In the software-only regime, optimization power can grow superlinearly while recalcitrance grows sublinearly. In the hardware regime, recalcitrance grows much faster — you have to fab chips, build datacenters, generate power.

This exactly matches the paper’s ideation/execution distinction: ideation = low-recalcitrance but hard to apply optimization power; execution = higher-recalcitrance but easier to apply optimization power.

Bostrom adds two theses absent from Good/Yudkowsky:

• Orthogonality thesis: capability and goals are independent — you can have a superintelligent paperclip maximizer
• Instrumental convergence: nearly any final goal implies similar subgoals (resource acquisition, self-preservation, cognitive enhancement)

The paper’s “meta risk” framing (ASARA amplifies all other risks) is a downstream consequence of instrumental convergence.

Davidson — the empirical model

Effective compute: E = C × A (physical compute × algorithmic efficiency).

Once AI labor dominates research (R ∝ A), substitution gives:

Regimes:

When β > 0, the ODE produces a finite-time blow-up (in the idealized model). Whether this happens in reality depends on historical algorithmic-progress data once AI dominates research — data we don’t yet have.

Chollet — the principled critique (2018)

1. Intelligence is not scalar — capability is domain-specific. In Yudkowsky’s notation: k > 1 in domain i doesn’t imply k > 1 overall.
2. Feedback loops require reality, not just thought — algorithmic progress requires experiments, which need compute, data, and calendar time.
3. Diminishing returns are the default — almost every system in nature exhibits them. Research productivity per researcher in physics has fallen by orders of magnitude.
4. No general intelligence has ever exhibited recursive self-improvement — smart humans don’t have smarter children at faster rates.

Putting them on the same axes

The key disagreement isn’t whether AI will improve. It’s whether k > 1 (or β > 0) is sustained for AI-on-AI improvement.

The Field et al. participants are basically a poll across this fault line. Frontier-lab researchers cluster near Bostrom/Davidson; academic researchers cluster near Chollet.

3. Ideation vs execution — and the evaluation problem

The sharper frame: research progress is bottlenecked on evaluation, and ASARA-class self-improvement is bottlenecked on evaluating research direction.

Execution = generation + evaluation
         ≈ generation (med)   +  evaluation (easy, fast feedback)
Ideation = generation + evaluation
         ≈ generation (easy)  +  evaluation (hard)

Research taste as a reward model

“Research taste” is the human-research term for what ML calls a reward model:

You don’t just need AI that proposes directions (solved) — you need RewardModelAI matching RewardModelsenior_human well enough that following its rankings produces good research. Two failure modes that echo RLHF:

• Mode collapse on training distribution — reward models trained on what the community valued historically will undervalue paradigm shifts that the community would have rejected at the time
• Goodharting on observable proxies — optimizing citation count, benchmark improvement, conference acceptance produces the proxy, not the underlying thing

The long tail of research impact

~80% of papers       →  cited < 10 times
~15% of papers       →  cited 10–100 times
~5% of papers        →  cited 100–1000 times
~0.1% of papers      →  cited > 10,000 times, define paradigms

Cumulative impact is dominated by the tail. A reward model 95% accurate on the median idea but mode-seeking on the tail produces 95% reasonable-looking work and 5% missed paradigm shifts — functionally similar to “no paradigm shifts.”

Mode-seeking behavior in MLE

MLE minimizes forward KL:

Forward KL is mean-seeking — it penalizes the model for assigning low probability to high-data regions, smearing probability across modes. The colloquial sense (“learning the typical case, missing the tail”) is exactly what MLE does. Both cut against tail-case evaluation accuracy.

RLHF and the Gao et al. 2023 result

Gao, Schulman, Hilton fit functional forms to gold reward vs proxy reward as KL grows. RL version:

Where d = √KL(π ‖ π_init). Gold reward follows an inverted-U: it rises, peaks, then declines. Proxy keeps rising monotonically. The two terms map to:

The iterated RLHF result — critical for ASARA

If you retrain the RM in k stages of distance d/k each:

The new term β · log k is positive but logarithmic in k, not linear. Doubling iterations adds the same as the previous doubling. The α term doesn’t move — regressional Goodhart is unaffected by iteration.

Translation for ASARA: even idealized “AI keeps retraining its own evaluator” produces logarithmic gains, not exponential ones. The mechanism Section 2’s k > 1 regime needs, the math gives you k_effective decaying like 1/log(rounds) — sub-exponential dynamics inside “the explosion.”

What this means for the speed of any recursive loop

Research progress rate = min(generation rate, evaluation rate)

If generation rate → ∞ (AI is fast):
  Bottleneck shifts to evaluation rate
  Evaluation rate is bounded by:
    - quality of the reward model
    - true distribution of idea quality (long-tailed)
    - calibration on tail cases (poor by default)

Asymmetric capability growth pattern:

That asymmetry is already visible in 2025–2026 data. Coding and math benchmarks have shot up; novel-paradigm research from AI has not appeared.

Six improvement vectors

1. Methodological triangulation — multiple human coders, multiple LLM classifiers, held-out validation
2. Larger and more representative samples — supplement depth with breadth via survey instrument
3. Longitudinal tracking — re-interview the same 25 every 6–12 months
4. Operationalizing the observable milestones — concrete metrics with thresholds
5. Empirical grounding for the theoretical disagreement — measure β in the AI-on-AI regime
6. Cross-national and cross-institutional coverage — PRC researchers, open-source / decentralized contexts

What’s happened since

1. Methodological triangulationArea to explore

No follow-on study has reproduced this paper’s interviews with multiple human coders, multiple LLM classifiers, or a held-out validation set. The methodological dependency on a single LLM (Claude) and single human coder remains the largest unaddressed gap. This is the cheapest fix — running the existing 25 transcripts through a Claude-vs-GPT-vs-Gemini comparison could be done in a weekend.

2. Larger and more representative samplesPartial

METR’s May 2026 self-reported productivity survey of 349 technical workers found median 1.4–2× value uplift from AI tools. But it’s about current productivity, not ASARA scenarios — it doesn’t ask about intelligence explosion, governance, or deployment. A semi-structured 200+ researcher study with proper stratification is still wide open.

3. Longitudinal trackingArea to explore

No team is running ASARA-belief surveys on a recurring cadence. Given how quickly the technology is moving — GPT-5 IMO gold August 2025, Claude Mythos May 2026, METR time horizons reaching 16 hours in May 2026 — a snapshot-and-update protocol would be the highest-information-density direction.

4. Operationalizing the observable milestonesSubstantial

METR has built much of this infrastructure:

• Time-horizon metric — duration of human-expert-time tasks AI completes reliably. Doubling every ~7 months.
• RE-Bench (Wijk et al., NeurIPS 2024) — 7 ML research engineering environments, best AI 4× human at 2-hour budgets, humans 2× AI at 32-hour budgets.
• MLE-Bench (OpenAI) — autonomous ML engineering on Kaggle-style tasks.
• MLR-Bench (2025) — open-ended ML research with LLM-judge validated against human reviewers.
• MLRC-Bench (2025) — novel methods on cutting-edge research problems, best agent closes only 9.3% of the human gap — strongest empirical confirmation of the ideation/execution gap.

Open: nobody has formally mapped Field et al.’s qualitative milestone descriptions onto these existing benchmarks.

5. Empirical grounding for the theoretical disagreementSubstantial

Davidson, Halperin, Houlden, and Korinek — “When Does Automating AI Research Produce Explosive Growth? Feedback Loops in Innovation Networks” (NBER Working Paper 35155, 2026) — develops a semi-endogenous growth model with an innovation network and derives a clean analytical condition for superexponential (“explosive”) growth. Two reinforcing channels offset diminishing returns:

1. Technological feedback across research sectors
2. Economic feedback — higher output finances further research

This is the most direct extension of the theoretical lineage — bridges the qualitative k > 1 / β > 0 disagreement into a formal model with testable conditions. Still open: no measurements of β in the AI-on-AI regime, because we don’t yet have sustained AI doing the research at scale.

6. Cross-national and cross-institutional coveragePartial

Substantial reporting on Chinese AI development exists, but no comparable interview study of PRC researchers.

• Carnegie documents Chinese views on AI safety shifting toward greater concern; senior Chinese scientists signed the 2024 IDAIS-Beijing consensus.
• CSET documents that Chinese researchers favor embodied AI as a path to AGI rather than software-only recursive self-improvement — a substantively different model of the path to ASARA.
• ChinaTalk: “Rather than a rapid software-driven intelligence explosion, Chinese thinking converges on something more embodied” — which would invert several Field et al. findings about trajectory shape.

A Field-et-al-style interview study with 20–25 Chinese AI researchers using the same protocol is still missing.

Emergent finding — the governance landscape moved fast

A development post-dating the paper that directly affects the red-lines findings:

The Global Call for AI Red Lines (September 2025) gathered 200+ prominent signatories including Nobel laureates, calling for binding international agreement on AI red lines by end of 2026. Followed by:

• IDAIS-Shanghai (August 2025) — proposed an international coordination body for red line implementation
• India AI Impact Summit (February 2026) — meant to operationalize specific thresholds
• Anthropic-DoD dispute (April 2026) — the DoD requested Anthropic remove contractual red lines on fully autonomous weapons and mass domestic surveillance, providing a real-world test case

Field et al.’s participants in Aug–Sep 2025 were debating red lines as an abstract governance approach. By mid-2026, red lines became an active political fight — making the paper’s documentation of expert preferences a useful historical baseline.

Scorecard