Methods
A pass means that an output satisfies the required tests in a declared evaluation contract. Repetition, execution status, resource use, and the validity of those tests answer separate questions.
The paper develops the measurement argument. The campaign provenance records what was compared; the frozen scorer defines the primary historical verdict. Current runner code describes how to launch a new experiment.
What is compared
| Unit | Meaning in this release | Interpretation boundary |
|---|---|---|
| Configured system | A model, harness, provider route, tools, skills, execution environment, and budgets. | Different models and harnesses vary together. Their observed gap does not identify either factor's separate causal effect. |
| Task and fixture | One authored request and its supplied input files, with a declared set of tests. | The 187 desk tasks are a designed workload, not a probability sample of business demand. |
| Attempt | One execution of a configured system on a fresh copy of a task's supplied workspace. | The 3 repetitions reuse the released inputs. They measure execution repeatability on those inputs, not performance on new seeds or unseen task templates. |
| Required check | A predicate over an output, potentially combining several rules or allowing a tolerance. | One failed check need not mean one business mistake; one passing check need not establish every property of the artifact. |
Execution boundary
The desk runner copies the supplied workspace to an opaque temporary directory and invokes an adapter with workspace, prompt-file, and output-directory arguments. In Docker mode, it mounts the run directory and read-only adapters; the repository's task answers and grader stay outside those mounts. Network access remains available, so container isolation does not prevent discovery of public task material online.
Local-process mode does not isolate reference answers or unrelated host files. A benchmark-owned home and declared skill inventory reduce uncontrolled context, but do not prove absence of exposure. The released ledger does not contain a complete per-attempt image and environment record; current container defaults must not be substituted for missing historical evidence.
Adapter and home behavior
The adapter contract declares the three arguments. The current runner copies a Proto home template per attempt. The Codex adapter uses a benchmark-owned CODEX_HOME; in Docker mode that directory is shared by the cell's attempts. A fake home is conditional on the configured directory existing. Neither mechanism establishes that all historical authenticated skills are reconstructible. Inspect the Codex adapter and record the environment for each new campaign.
Desk protocol
- The published comparison contains 187 tasks × 3 repetitions per system. The exporter rejects an incomplete release matrix; a missing record is not silently counted as an ordinary task failure.
- Each attempt receives the task's request in a fresh workspace. Repetition identifiers are bookkeeping labels, not matched random seeds or simultaneous trials.
- The current runner defaults to one repetition unless
--runsis supplied. Its task timeout defaults to 1,200 seconds; Docker resource defaults are 2 CPUs and 6 GB. Task metadata can change the timeout; command-line options can override timeout and resource limits. - Timeout, exit status, grader error, and artifact acceptance are recorded separately. Completing a process normally does not establish acceptable work; a failed process does not by itself explain which acceptance condition failed.
Use the reproduction guide to verify the published accounting or launch an explicitly named new experiment.
Acceptance and scoring
An attempt is accepted when every required check passes. The boundary is the declared contract, including its tolerances and coverage thresholds. Optional checks can remain diagnostic. Passing all tests does not prove that the contract includes every business requirement; see the payment-reconciliation example.
| Verdict | Source | What to preserve |
|---|---|---|
| Original / raw | The runner's grader, using the task definitions supplied at execution. | Original verdict and grader errors. An ungraded attempt is distinct from an ordinary failed check. |
| Frozen / primary | conservative-v7, applied retrospectively to retained outputs, with its own copied definitions and equivalence rules for 7 tasks. | Both verdicts, scorer identity, receipt hash, and output hashes. Scoring the same outputs again is not a new agent run. |
The frozen scorer was assembled by the benchmark authors after outputs existed. Its 847-file fingerprint makes a version inspectable; it does not establish independent authorship or prospective registration. The audit separates observed verdict changes from unknown false-positive and false-negative rates.
Workbook checks remove cached values and use native LibreOffice recalculation where required. Sentence-level text checks guard specific combinations of facts and qualifiers. These mechanisms test stated properties; they do not provide exhaustive semantic or visual review.
Check types and current task-format definitions
Counts below come from current task metadata. For the historical primary score, inspect the definitions inside the frozen package.
| Type | Declared uses |
|---|---|
csv_values_match | 302 |
xlsx_value_present | 247 |
csv_columns | 107 |
csv_set_equal | 102 |
csv_row_count | 96 |
custom | 91 |
file_exists | 71 |
text_sentence_matches | 65 |
xlsx_no_errors | 55 |
xlsx_has_formulas | 51 |
text_not_contains | 33 |
text_numbers_present | 32 |
text_contains_all | 14 |
text_matches_all | 4 |
text_contains_any | 2 |
| type | fields | passes when |
|---|---|---|
file_exists |
path | a file matches the glob |
csv_columns |
path, columns, exact? | required columns present (or exact order) |
csv_row_count |
path, equals | equals_ref, tolerance? | row count matches |
csv_set_equal |
path, column, ref, ref_column?, normalize? | set of values equals the reference set |
csv_values_match |
path, ref, key, columns, min_accuracy?, numeric?, tolerance?, must_match_keys? | per-key values match at or above the accuracy floor, and every must-match key is right. key may be a list of columns (a composite key); a must-match key for a composite is a list or a `a |
xlsx_has_formulas |
path, sheet?, min_count | at least N live formula cells |
xlsx_value_present |
path, expected, rel_tol?, near_text?, sheet?, rounding?, raw_value_ok? | some numeric cell equals expected after recalculation, optionally on a row or column mentioning near_text. Text cells count only when they read as one figure ("$1,240.00", "(30.00)", "12.5%"), never dates or ids. Tolerance is max(expected x rel_tol, 0.01); the strict validator requires it to be at most 1.00 unless the check sets rounding: <reason> (conversion, proration, estimate), which allows rel_tol up to 0.001. raw_value_ok: true declares that the figure legitimately appears in the raw inputs (the validator otherwise rejects pins an agent could satisfy by pasting the export) |
xlsx_no_errors |
path, sheet?, max_errors? | no formula cell evaluates to an Excel error (#NAME?, #VALUE!, #REF!, ...) after recalculation; added after the audit found error cells outside the pinned cells passing |
text_contains_all / _any / text_not_contains |
path, phrases | case-insensitive phrase checks |
text_matches_all |
path, patterns | every regex matches somewhere in the text |
text_numbers_present |
path, numbers, rel_tol?, min_count? | the figures appear in the text within tolerance, ignoring currency symbols, commas, and parentheses |
text_sentence_matches |
path, all, none? | one sentence matches every regex in all and none in none; use it when scattered phrases would pass on unrelated sentences or on a negation |
custom |
module | check(ws, ref) returns a list of {name, passed, detail} |
plan_feasible |
module (default plan_check.py), max_gap, sense?, gap_reason? |
the module's evaluate(ws, ref) reports the plan feasible and its objective within max_gap (relative) of reference_objective; sense: min (default) or max. The gap is recorded under metrics. Strict validator: max_gap above 0.10 needs gap_reason. Scaffold: tasks/lib/bizgen/planning.py (v2, axis 5) |
forecast_error |
path, ref, key, column, max_error, metric?, ref_column?, error_reason? | every key in the held-out truth file is forecast and the error (wape default, or mape, mae, rmse) is at most max_error; missing keys fail. Strict validator: the truth file must not be in the workspace by name or bytes; max_error above 0.25 needs error_reason (v2, axis 5 and 7) |
not_fooled |
path, ref, key, columns, planted_keys?, forbidden_text?, flag? | the planted keys still equal the reference (pre-instruction state), no forbidden_text phrase appears in the listed files, and, if flag is given, one sentence in the flag file matches every regex in all. At least one part must be set. Injector: tasks/lib/bizgen/adversarial.py (v2, axis 4) |
Check results may carry a metrics object (for example the plan gap or forecast error) beside the verdict; it is diagnostic and never changes pass or fail.
Metrics and uncertainty
| Measure | Denominator / weighting | What it answers |
|---|---|---|
| Attempt acceptance (pass@1) | Accepted attempts / all attempts in the complete matrix; 561 per system here. | How often did this configuration meet the full contract across recorded executions? |
| All-3 acceptance (pass^3) | Tasks with all 3 attempts accepted / 187 tasks. | Which supplied fixtures succeeded every time observed? This is an observed joint outcome, not the aggregate pass rate raised to a power. |
| At-least-once acceptance | Tasks with at least one accepted attempt / all tasks. | Whether a success existed in hindsight. Selecting a successful output without the grader is a separate problem. |
| Mean within-attempt check fraction | Average each attempt's fraction of required checks passed, giving each attempt equal weight. | Diagnostic distance from the declared contract. It cannot turn a missing required property into acceptable work. |
| Pooled required-check rate | All passed required checks / all required checks across attempts. | Weights attempts with more predicates more heavily. Splitting a rule into several checks can change this value without changing acceptance. |
| Resources and cost per accepted output | Captured token usage and price assumptions; observed durations; total captured model cost / accepted attempts. | Campaign accounting. Missing usage is unknown; cost excludes review and repair labor. The ratio is not an expected retry-until-success cost. |
The reported paired bootstrap resamples task identifiers, retaining all 3 repetitions for both systems within each sampled task. It describes variation under that empirical resampling scheme. Related task families, development exposure, scorer selection, and a different deployment workload are outside its uncertainty estimate. Categories inherit their task counts in the aggregate; different business workload weights can change the ranking. See results and findings.
Task validation and its limits
The release notice reports reference-solution passes, untouched-workspace failures, and repeated deterministic generation under the validation environment. These are useful positive and negative controls. They do not estimate agreement with independent practitioner judgments or failure rates on plausible wrong outputs.
The strict validator also checks declared structure and selected value-pin hazards, with explicit legacy-task exceptions. It reported 28 tasks whose supplied fixture bytes differ from regenerated bytes despite repeatable regeneration. The supplied bytes define the release; regenerating inputs creates a different experimental condition rather than repairing the published record.
Recorded configurations
| System | Harness | Model and route | Recorded configuration | Source cohorts |
|---|---|---|---|---|
| Proto | Proto CLI, runtime c8f62dd60 | deepseek/deepseek-v4.1-flashDeepSeek only, no fallback, high reasoning, temperature 0 | runtime_commit: c8f62dd60 model: deepseek/deepseek-v4.1-flash provider: DeepSeek only fallbacks: false reasoning: high temperature: 0 max_tokens: 65536 | broad-desk-v43-c8f62dd60 r1; repeat-desk-v43-c8f62dd60 local r1/r2 map to logical r2/r3 |
| Codex | Codex CLI 0.154.0 | gpt-5.6-solOpenAI via Codex CLI, high reasoning, CLI-managed sampling | model: gpt-5.6-sol reasoning: high cli_version: 0.154.0 sampling: CLI-managed; not asserted matched to Proto | full-1-codex-sol r1/r2/r3 |
These cohorts were not contemporaneous and their sampling controls differ. The provenance records their scope and exclusions. An adapter in the repository establishes an available integration, not a published evaluation result.
Build acceptance
The 20 released build tasks contain an initial requirement followed by three change requests. Each turn starts a new invocation on the carried-forward workspace. Acceptance requires observing the delivered application: working logins, permissions, direct-request authorization, data integrity, persistence, and the task's rules.
RESULT.json, a responding URL, or a generated tester sheet are evidence to inspect, not a completed acceptance result. Record observed passes, observed failures, and untested items; recheck earlier requirements after each change. Initial success, new change requirements, regressions, reachability, time, and cost remain separate. No completed build leaderboard is published in this release.
Enterprise baseline: the 14 acceptance items
Rendered from docs/build-baseline.md. Task checklists supply the expected counts and role-specific placeholders; consult the full source for setup and test-state handling.
(CORE) [Delivery]
RESULT.jsonhasurl,admin.user,admin.password,restricted.user,restricted.password,notes; the URL loads a login page in a desktop browser on the tester's machine; both logins sign in. How to check: Open the URL; log in as admin, log out, log in as restricted. Fail if a field is missing, the page does not load, or either login is rejected.[Delivery] The URL is reachable from a phone on a different network than the build machine. How to check: On a phone with Wi-Fi off, open the URL; the login page renders and the admin login succeeds. Fail if it does not load, or the address is localhost, 127.0.0.1, or a private LAN IP.
(CORE) [Sharing] Admin can invite a person by email address and choose their role at invite time; the app shows an invitation link the admin can copy; opening that link in a private window lets the invitee set a password and lands them signed in with the chosen role; the invitee then appears in the app's users or team list with that role. How to check: As admin, invite
tester.viewer@example.comas the read-only role, copy the link, open it in a private window, set a password, confirm the session is signed in as that person and that the users list shows them with the read-only role. Fail if no invite exists, the link is not shown, the link asks for anything other than a password, or the role differs.[Sharing] Invitation links are single-use and invitations and users are revocable: opening an accepted link again does not sign anyone in; admin can revoke a pending invitation; admin can remove or deactivate a user, after which that user's login is rejected. How to check: Reopen the item-3 link in a fresh private window (expect a clear failure). Invite
tester.pending@example.com, revoke it, open its link (expect failure). Invitetester.remove@example.comas the read-only role, accept that invitation in a private window, then remove or deactivate that user and try to log in as them (expect rejection). Keep the item-3 viewer for item 5. Fail on any of the three.(CORE) [Permission] At least these three roles exist and hold: Admin manages users and settings and sees every record;
creates and edits records in its scope and has no user management or settings; is read-only, every create, edit, and delete control is absent or rejected. How to check: As the item-3 viewer, try to create one and to edit an existing one (expect no control, or a rejection and no change on reload). As restricted, confirm the users and settings areas are absent or refused. Fail if the viewer can change anything or the staff login can manage users. (CORE) [Permission] Row-level scope: the restricted login sees only
, exactly , and cannot open another scope's record by URL. How to check: As restricted, read the total and open five records to confirm scope. As admin, copy the URL of ; paste it in the restricted session. Pass only on the exact count and an error, not-found, or redirect without rendering that record's details. [API] Authorization is enforced by the server, not the page: replaying a data request from the restricted session with another scope's record id, or with any scope or role filter removed, returns 401, 403, or 404 or contains no foreign record; replaying a create or edit request from the viewer session is rejected the same way. How to check: In the restricted session open devtools > Network, load the list and one record, copy the data requests as cURL, substitute
's id and remove filter parameters, re-issue. In the viewer session copy any write request the UI would send (or craft one against the same endpoint) and re-issue. Fail if any response carries foreign data or any write succeeds. For server-rendered apps, test the corresponding page and form requests directly; hidden controls alone do not establish server authorization. If a required request was not tested, record it as untested. (CORE) [Dashboard] A dashboard page shows at least four figures computed live from the data:
= , = , = , = ; after the tester creates one , the affected figure changes accordingly on reload. How to check: As admin, open the dashboard and read the four figures (exact values, tolerance 0.01 on money). Create one in scope of , reload, confirm moved by one (or by the amount). Fail if a figure is missing, wrong, or static. [Dashboard] The dashboard respects scope: signed in as restricted it shows the same figures for that scope only:
= . How to check: As restricted, open the dashboard. Fail if it shows company-wide figures or is absent. [Exact] The main list has working search, at least one filter, and column sort: searching
<SEARCH_TERM>returns exactlyrow(s); filtering on = returns ; sorting by descending puts first. How to check: As admin, perform the three operations. Fail on any count or order that differs, or if an operation is missing. [Exact] Export of the main list to CSV produces exactly
data rows and includes the columns . How to check: As admin, export with no filter applied and open the file. Fail if rows or columns differ or export is absent. [Audit] Records show who created and who last changed them and when, and an activity or audit log lists the tester's own create, edit, and delete actions with user and timestamp. How to check: As admin, create one
, edit it, delete it; open the record's history or the audit log. Pass only if all three actions appear attributed to the admin user with timestamps, and an existing imported record shows created-by and updated-at fields. [Rule] Required fields are validated by the server: submitting one
without through the form is rejected, and replaying the create request without that field (devtools > copy as cURL, remove the field) is also rejected. How to check: Perform both. Fail if either creates a record (check the list after reload). (CORE) [Persistence] Data survives a restart of the app and lives on the server, not in the tester's browser. How to check: After the items above, have the operator actually restart the application process and record that action, then log in as admin in a different browser. Pass only if the counts and the tester's remaining changes are present. Waiting alone does not test restart persistence. Record the item as untested if a restart was not observed; fail if the persisted state reverted or disappeared.
Process track
Process attempts retain ERP state across fresh agent turns while simulated counterparties act between business dates. Declared checks inspect final state, audit events, control-account relationships, and requested outputs. The pilot is reported separately because its tasks, scorer, repetitions, and local execution conditions differ from the desk release.
See the published pilot and its conditions, working specification, and process execution guide. Repeating one scenario seed tests execution repeatability; evaluating additional seeds and unseen templates requires separate evidence.