As more of the scientific process is delegated to the agent, it is not enough to evaluate just the results of the analysis, but to understand if we can trust the scientific process through which the agent arrives at the results. To get to the bottom of this question, the source paper tries to systematically analyze components of a good evaluation and how to find out if we’re actually evaluating biological reasoning.
The realism–verifiability trade-off
As the capabilities of agentic systems increased, both the systems and their evaluations moved closer to realistic use cases. That progress introduced another problem: how to balance realism with verifiability. Increasing autonomy and task scope gives agents greater freedom, but that freedom becomes harder to evaluate in practice.
More constrained tasks are easier to verify, yet less representative of research and industrial workflows. An ideal benchmark therefore aims for the highest realism that remains programmatically verifiable, while keeping compute and time requirements low enough to run sufficient evaluations for reliable statistical estimates.
A key part of developing a high-fidelity evaluation suite is determining both what should be assessed and how that assessment should be carried out. Rubric design defines the criteria used to evaluate the quality of a response, while grader design determines how those criteria are applied to produce a score.
What defines the target
Rubric grounding
| Basis | Definition | Examples |
|---|---|---|
| Empirical reference | Derived from an existing empirical reference, such as a published or independently reproduced biological result. | SpatialBench, scBench, EpiBench, VariantBench, SpatialBench-Long, scBench-Long, HeurekaBench |
| Constructed target | Based on a simulated or otherwise constructed target, including synthetic or augmented ground truths that can be verified directly. | CompBioBench, ScienceBoard, GeneBench-Pro |
| Inferred process target | Rewards completion of the analytical steps considered appropriate for the task. | BiomniBench, BioAgent Bench |
How the target is assessed
Grader design
| Design | Definition | Examples |
|---|---|---|
| Deterministic | Produces the same score for the same output. Reproducible, but may reject valid answers outside predefined vocabularies or numerical tolerances. | SpatialBench, CompBioBench, GeneBench-Pro |
| Interpretive | Assesses whether a response satisfies the rubric. Accommodates scientifically defensible variation, but introduces variability across judges and repeated evaluations. | BiomniBench, SpatialBench-Long |
What a benchmark score contains
A score is the value produced when a grader applies the criteria in a rubric to an agent’s outcome, its trajectory, or both. It may be binary, graded, continuous, or comparative, but it is never a direct reading of scientific capability.
The reported value also reflects how the benchmark was constructed, how its target was translated into a task, which path the agent took, and how the grader interpreted the result. The resulting evaluation error for an agent f accumulates across five components:
- Construction bias
- Enters through the benchmark world, task selection, and target answers.
- Specification bias
- Enters when the target becomes a prompt, rubric, threshold, or output schema.
- Judge bias
- Systematic strictness, leniency, or interpretation introduced by the grader.
- Execution variance
- Different trajectories produced by repeated runs of the same agent.
- Judge variance
- Different judgments under otherwise equivalent evaluation conditions.
These choices form a matrix of evaluation strategies. Moving from a constructed target toward an inferred analytical target makes correctness less directly verifiable. Moving from deterministic to interpretive grading gives the judge more flexibility, but introduces greater judge-dependent variance.
| Grader design | Rubric grounding | ||
|---|---|---|---|
| A Constructed target known by design | B Empirical reference published or reproduced | C Inferred process target expected analytical steps | |
| A Deterministic fixed, programmatic rules | |||
| B Interpretive expert or LLM judgment | |||
What failure modes tell us
Single-trajectory failure modes
Each level assumes competence at the one above it. Deeper failures can survive clean code and a completed pipeline.
| Failure category | What fails | Examples from evaluations |
|---|---|---|
| 01 General agentic | Planning, execution, monitoring, or recovery breaks down. | Early stopping, reasoning loops, tool errors, failed error recovery, and broken data-format handling. |
| 02 Data and grounding | The agent selects, checks, represents, or understands biological data incorrectly. | Ignoring corrupted inputs; analyzing an unrelated organism decoy; missing swapped condition labels; using the wrong cell-label field. |
| 03 Analysis design and workflow | A valid workflow or method does not adequately answer the scientific question. | Using end-to-end instead of local alignment for CUT&RUN spike-ins; treating two Bismark rows as independent CpGs; sending paired-end BAMs directly to MACS3. |
| 04 Statistical inference | The estimand, experimental unit, model, parameters, or uncertainty is wrong. | A time-varying Cox model appears reasonable but fails to account for treatment–confounder feedback. |
| 05 Interpretation | Biological meaning, direction, importance, or causal strength is not justified by the analysis. | Favoring familiar CD39 and PD-1 markers over the task definition; replacing a computed result with an expected one; substituting canonical markers for pathway enrichment. |
Future work
Counterfactual reasoning is the cognitive process of imagining alternative, hypothetical scenarios and outcomes for events. Perturbation tests and simulated targets are the first practical steps toward evaluating this capability. By inserting a diagnostic trigger while holding the scientific question fixed, an evaluation can test whether an agent updates its analysis and conclusion appropriately. Future evaluations should move toward counterfactual reinforcement learning for biology, where agents learn how a biological conclusion should change when the evidence changes.