Derived from the released ledger at build time · complete-desk-comparison-2026-09-16
Findings
Seven observations from 1,122 attempts, each with its number, its computation, and its caveat. None of these is a claim about models in general. They are what the two configured systems did on this workload, and they are the phenomena Business Bench exists to measure.
Definitions
A task t has a set of required checks Ct. An attempt passes iff every c ∈ Ct passes. With k = 3 repetitions per task:
- pass@1: mean attempt pass rate. Equal to the headline pass rate.
- pass@k: fraction of tasks with at least one passing repetition. What a demo shows.
- pass^k: fraction of tasks where all k repetitions pass (the τ-bench consistency metric). What an owner experiences.
- check rate c̄: passed required checks over all required checks, pooled across attempts.
- conjunctive gap: c̄ − pass@1. Credit a partial-credit grader would award that a conjunctive grader does not.
- near miss: a failed attempt with exactly one failed required check.
1. The conjunctive gap is 7 to 9 points
| System | Required checks | Check rate c̄ | pass@1 | Gap (pp) |
|---|---|---|---|---|
| Proto / DeepSeek V4.1 Flash | 3801 | 97.29% | 90.37% | 6.92 |
| Codex / GPT-5.6 sol | 3801 | 93.61% | 84.31% | 9.29 |
Both systems satisfy well over nine in ten required checks. Scored per check, the two would look nearly saturated and nearly identical. Scored per task, 9.3 and 6.9 points disappear. The gap is the quantity a rubric or checkpoint grader awards as partial credit, and it is precisely the work an owner would have to redo. Caveat: the gap depends on how many checks a task declares; tasks with more checks have more ways to fail one. The per-category table in finding 4 shows it is not an artifact of one task family.
2. Most failures are one check away
| System | Failed attempts | Failed exactly 1 check | Share | Mean share of checks passed in a failure | Single failed check, by type |
|---|---|---|---|---|---|
| Proto / DeepSeek V4.1 Flash | 54 | 34 | 63.0% | 72.3% | custom 18csv_values_match 4text_sentence_matches 4text_not_contains 4 |
| Codex / GPT-5.6 sol | 88 | 39 | 44.3% | 64.4% | custom 12text_sentence_matches 9csv_values_match 6xlsx_value_present 5 |
| Failed required checks per failed attempt | Proto | Codex |
|---|---|---|
| 1 | 34 | 39 |
| 2 | 14 | 17 |
| 3 | 0 | 5 |
| 4 | 1 | 5 |
| 5 | 2 | 7 |
| 6 | 0 | 6 |
| 7 | 0 | 8 |
| 8 | 0 | 1 |
| 9 | 3 | 0 |
A failed attempt is usually not a wrong approach. It is a deliverable that is almost right: the address line that kept one unabbreviated suffix, the memo that stated the figure but not the exception, the workbook with one hardcoded cell. The dominant single failure is a task-specific grader, which encodes the business rule the ask buried in an email. This is the strongest argument for treating self-verification as a capability in its own right (v2 axis 6): the systems had the information to catch these and did not check. Caveat: a single failed check can also be a strict check; the false-negative audit on the roadmap is what separates the two readings.
3. pass@3 and pass^3 are 20 points apart
| System | pass@1 | pass@3 | pass^3 | Spread (pp) | Tasks with mixed outcomes / 187 |
|---|---|---|---|---|---|
| Proto / DeepSeek V4.1 Flash | 90.4% | 98.4% | 78.6% | 19.8 | 37 |
| Codex / GPT-5.6 sol | 84.3% | 93.0% | 75.4% | 17.6 | 33 |
Ask "can the system do this task?" and the answer is yes for 98% of tasks. Ask "does it do it every time?" and the answer drops to 79%. Roughly one task in five has a different outcome depending on the run. For a monthly process that is the number that matters, and it is invisible to any leaderboard that reports pass@1 or best-of-n. Three repetitions estimate pass^3 coarsely; v2 moves to five, and the task-clustered bootstrap on the results page gives the descriptive interval for the headline difference.
4. The systems diverge on text, not numbers
| Category | Proto check rate | Proto pass@1 | Codex check rate | Codex pass@1 | pass@1 difference (pp) |
|---|---|---|---|---|---|
| Spreadsheet | 97.9% | 93.5% | 91.6% | 88.6% | 4.9 |
| Bookkeeping | 96.8% | 91.7% | 96.3% | 90.7% | 0.9 |
| Reports | 98.1% | 90.0% | 85.0% | 62.2% | 27.8 |
| Reformatting | 99.0% | 94.9% | 100.0% | 100.0% | -5.1 |
| Extraction | 95.3% | 89.7% | 97.6% | 88.5% | 1.3 |
| Drafting | 97.7% | 86.0% | 93.3% | 64.9% | 21.1 |
| Tooling | 93.0% | 70.4% | 99.2% | 96.3% | -25.9 |
On reformatting, extraction, bookkeeping, and spreadsheets the two systems are within a few points of each other and both are near their check ceilings. The headline difference is concentrated in reports and drafting, where deliverables are prose graded by sentence-level rules, and reversed in tooling. Two readings are consistent with this table: one system is better at source-grounded writing, or the sentence-level checks are stricter on one system's style. The frozen scorer's equivalence graders exist because the second reading was true for seven tasks. Until the false-negative audit runs, the honest statement is that the difference lives in the categories where grading text is hardest.
5. Cost per passing attempt differs by an order of magnitude
| System | Estimated cost, total USD | Passing attempts | USD per pass | Median wall time | Input tokens |
|---|---|---|---|---|---|
| Proto / DeepSeek V4.1 Flash | 22.06 | 507 | 0.0435 | 2.1 min | 824M |
| Codex / GPT-5.6 sol | 219.13 | 473 | 0.4633 | 2.3 min | 166M |
The cheaper system does five times the token work and lands a passing deliverable for a tenth of the price. Three caveats carry the weight here: the price table is API list price applied to a subscription route, captured usage omits errored requests, and a flash-tier model on a pinned first-party endpoint is a different product from a frontier model behind a CLI. Cost per pass is the right unit for a buyer; it is not evidence about model quality.
6. The scorer moved the gap more than the models did
Under the runner's original grader the lead was 16 attempts. Under the frozen scorer, which adds equivalence graders for seven tasks and a stricter base grader, it is 34: the scorer raised Proto by 60 and Codex by 43. A benchmark whose verdict depends this much on scorer revisions must publish both verdicts and freeze the scorer by hash, which is what this release does. It must also have the scorer reviewed by someone who did not build the leading system, which this release has not yet done. Details on the results page and the self-audit.
7. The clerical band is close to saturated
116 of 187 tasks were passed by both systems in every repetition. 15 tasks had at least one system at zero for three, and one task defeated both. For the two systems measured, the current task set separates them on roughly a fifth of its tasks. That is enough to rank, not enough to characterise. It is why v2 defines a difficulty axis with analyst and controller bands from the same generators, and why axes 4, 6, and 7 (adversarial inputs, self-verification, process discovery) target behaviours neither system has been asked to show.
Every figure above is recomputed from results/latest/attempts.jsonl when the site builds; the test suite asserts the derived rates agree with summary.json. Reproduce with python bench/export_campaign.py --verify and the loader in site/src/lib/data.ts.