Harness Bench — Dionizije Fa

The longer-term goal is to reconstruct the task × model × harness success matrix without running every cell. I will test whether success has enough low-rank structure for SVD to identify a small set of Terminal-Bench tasks that predicts the full matrix.

If that works, the reduced task panel becomes a fast screen for many more model–harness combinations which leaves space to reserve full runs for combinations that look promising.

Harness Bench architecture in four layers. On the always-on main device, Dagster schedules work through Celery. The queue distributes rollouts across interchangeable server workers. Each worker runs Verifiers v1 over a Harbor task with a pinned harness connected to the model API. Dagster stores run metadata in Postgres. Rollout workers store logs in MinIO. A separate control Celery worker receives completed step values and serializes benchmark results into SQLite.

Explore 78 Terminal-Bench tasks across DeepSeek V4 Flash, GLM 5.3 Flash and Muse Spark 1.3 Contributor, by model, by harness or as individual combinations.

Loading results…

General

Models

Harnesses