Sound agentic science requires adversarial experiments.

Agents used for data analysis in biology increase the hypothesis space, while the only real way of shrinking the hypothesis space is by running experiments.

ICLR 2026 · Agents in the Wild
Read the paper OpenReview

Software agents improve a program against rapid user feedback. In empirical science, the verifier is nature. Re-running a fixed dataset cannot replace an experiment designed to distinguish competing explanations.

A

Software agent

Fast access to a verifier

A software agent’s verification loop Candidate code is run against program behavior, specifications, tests, and user feedback. Failures drive revision, contracting the space of viable outputs. VERIFIABLE REALITY Program behavior specs · tests · user feedback Candidate code run fail → revise

Further iterations with the agent shrink the hypothesis space.

B

Data analysis agent

No direct access to the verifier

A data analysis agent’s open loop A biological process can be probed only through a new experiment. Without one, the agent loops between observed data and analytic choices, expanding the space of plausible narratives. REALITY Biological process VERIFIER New experiment AGENT-ACCESSIBLE LOOP Analytic choices Observed data new measurement verification gap

Further iterations with the agent expand the hypothesis space.

What follows from the gap

  1. More analysis can mean more narratives. Weighting, exclusions, covariates, outcome definitions, and model families can each be defensible. An agent can search their combinations until a publishable pattern appears.
  2. Statistical support is not a test suite. A low p-value or fluent explanation verifies a result only under one model and its assumptions. It does not verify the scientific claim or its mechanism.
  3. The missing evidence is negative space. The controls, perturbations, and replications that would have broken a claim are often never run. Those absent tests matter more as claims become cheaper to produce.

When the Objective Shapes the Result

We gave two independent agents the same 2017–2018 NHANES data. We instructed one to find a correlation between vitamin D and depression and the other not to find one. Both could vary ordinary epidemiological choices while remaining within defensible practice.

Agent A · association sought Small negative association

−0.045 PHQ-9 points per 10 nmol/L higher vitamin D

95% CI −0.068 to −0.023 · p = 0.0006
Agent B · null sought Effectively no association

+0.0005 PHQ-9 points per 1 nmol/L higher vitamin D

95% CI −0.0050 to +0.0060 · p = 0.855

Mitigation strategy:

For every non-experimental claim produced with agentic help, use the same cheap analytical capacity to propose rival explanations and run targeted checks for confounding, selection, measurement error, and arbitrary analytical choices.

Continue reading