Multi-agent Harness Bench

To test how well harnesses coordinate native subagents, I ran five harnesses on Terminal-Bench 4.0’s vba-userform-port: porting a legacy Excel/VBA application to React, FastAPI and SQLite while preserving its validation, calculations, saves and form behavior.

Time limit: 15 min
Shared CPUs: 8
Shared memory: 16 GiB
Native subagents: Unlimited

Scores for each harness and configured orchestrator → subagent pair.
Harness	Sol 6.1 Max GLM 5.3 Flash High	Opus 5.5 Max Haiku 5.5 High	Astra 6 Max Luna 6 High
Hermes0.20.0	55%Partial credit	96.7%Partial credit	98%Partial credit
OpenCode1.18.1	0%Timed out	0%Response limit	0%Timed out
OMP17.2.10	0%Backend missing	0%Response limit	92.7%Partial credit
Claude Code2.1.280	96.7%Partial credit	90.7%Partial credit	0%Delegation failed
Codex0.137.0	91.3%Partial credit	96.7%Partial credit	89.3%Partial credit

Harnesses ranked by mean score across all three configured model pairs, including zero-score runs.
Harness	Mean score

Orchestrator → subagent pairs ranked by mean score across all five harnesses, including zero-score runs.
Model combination	Mean score

0–100% creditOrchestrator → subagents

Scoring · 28 checks, 10 subtasks

API & SQLite contract1 check
CRUD & persistence3 checks
Work-order validation3 checks
Atomic saves2 checks
VBA calculations & tax2 checks
Approval & billing guards5 checks
Form defaults & cascades3 checks
Dirty-field SLA lifecycle2 checks
UI editing, saves & totals5 checks
UI validation & status2 checks

Score = Σ (10% × passed checks / subtask checks)

Read the versioned rubric ↗