← The ledger
Receipt 003Evals & benchmarks

12 tasks, 7 domains, 3 execution modes — with a human baseline

The falsifiable questionDoes orchestrated multi-agent execution actually beat a single agent — and a human — on real initiative-level work, measured the same way every week?

ARunnable artifactearned
BMeaningful baselineearned
MMeasured resultearned
RReproducible methodearned
FDocumented failureearned
EExternal validationwithheld

External validation withheld until judges or reproductions come from outside the project.

Baseline

Two baselines, run through the identical harness: a single agent with the same tools, and a human doing the same task. Most agent demos compare against nothing.

Method

A public benchmark repo (useorgx/autonomous-initiative-benchmark): a 15-task catalog across three tiers, 12 tasks live across 7 domains, each run in 3 execution modes. Bundle validator and scorecard recompute keep results honest; methodology is published, with the literature it builds on cited.

Measured result

59 result bundles committed with numeric per-task scores. A verified gpt-5-nano run completed 12/12 tasks through the bundle validator. Instrumented-worlds eval: pass^k = 1.0 on 4 of 5 worlds, including an 8/8 on the six-trap arithmetic world.

Where it broke

Catalog coverage was 12/15 at the June gap analysis — three tier-3 tasks were specified but not yet runnable. And 1 of 5 instrumented worlds did not hold pass^k = 1.0. Both gaps are documented in the repo, not smoothed over.

Next test

Independent judges scoring runs they did not build — then the E column gets earned.