Protocol, grader, metrics
Methods
A condensed account of how an attempt is executed and judged. The normative text is the specification; where this page and the paper differ, the paper wins.
The unit of comparison is the configured system
A model does not act alone. Its harness decides how it reads files, invokes tools, manages context, recovers from errors, and delivers artifacts. Each experimental cell is therefore identified by harness, model configuration, provider route, tools, and substrate, and results describe the system that ran. An end-to-end gap between two cells does not, by itself, isolate a harness or a model effect; that needs matched controls.
Agent boundary
The runner copies only the task workspace into an opaque temporary directory, writes the ask to a prompt file, and calls a harness adapter with three arguments: workspace, prompt file, output directory. In Docker mode the agent sees /run/ws and a per-run copy of its home; the task tree, reference answers, checks, and traps stay on the host. Containers run as the host user with CPU, memory, and process limits. Network access is available for model calls and permitted tooling, so this is not an offline benchmark and not a defence against finding the published tasks online.
The adapter contract is one shell script per cell, with the model pinned inside it: harnesses/contract.md.
Desk protocol
- Fresh workspace per repetition; three repetitions per task; single turn.
- Time budget from the task, 1,200 seconds by default, with 2 CPUs and 6 GB per attempt.
- A complete arm has exactly one record per task and repetition. Missing records are a runner error, not a silent gap.
- Execution status (exit code, timeout) is recorded separately from the artifact verdict.
Desk grading
Two verdicts exist for every released attempt. The raw verdict is what the runner's grader recorded at run time. The frozen verdict, the primary score, comes from the immutable conservative-v7 package under scoring/frozen-v7: its own copy of the base grader and task definitions, plus equivalence graders for 7 tasks whose deliverables have more than one valid layout. Each frozen verdict is bound by a receipt to the original result hash, the output-file hashes, and the scorer manifest (847 hashed files). Run it on any workspace with python scoring/frozen-v7/scorer.py TASK_ID WORKSPACE.
An attempt passes when every required check passes. Checks are declared in task.yaml and evaluated by bench/grade.py; tasks with rules the built-in types cannot express add a check.py. Workbooks are graded after stripping cached values and recalculating natively in LibreOffice, so a formula that only looks right in the file's cache does not pass. Text checks work at sentence level where scattered phrase matches could pass on the wrong sentence or on a negation.
Check types in use
| Type | Uses across 187 tasks |
|---|---|
csv_values_match | 302 |
xlsx_value_present | 247 |
csv_columns | 107 |
csv_set_equal | 102 |
csv_row_count | 96 |
custom | 91 |
file_exists | 71 |
text_sentence_matches | 65 |
xlsx_no_errors | 55 |
xlsx_has_formulas | 51 |
text_not_contains | 33 |
text_numbers_present | 32 |
text_contains_all | 14 |
text_matches_all | 4 |
text_contains_any | 2 |
Definitions, from docs/task-format.md
| type | fields | passes when |
|---|---|---|
file_exists |
path | a file matches the glob |
csv_columns |
path, columns, exact? | required columns present (or exact order) |
csv_row_count |
path, equals | equals_ref, tolerance? | row count matches |
csv_set_equal |
path, column, ref, ref_column?, normalize? | set of values equals the reference set |
csv_values_match |
path, ref, key, columns, min_accuracy?, numeric?, tolerance?, must_match_keys? | per-key values match at or above the accuracy floor, and every must-match key is right. key may be a list of columns (a composite key); a must-match key for a composite is a list or a `a |
xlsx_has_formulas |
path, sheet?, min_count | at least N live formula cells |
xlsx_value_present |
path, expected, rel_tol?, near_text?, sheet?, rounding?, raw_value_ok? | some numeric cell equals expected after recalculation, optionally on a row or column mentioning near_text. Text cells count only when they read as one figure ("$1,240.00", "(30.00)", "12.5%"), never dates or ids. Tolerance is max(expected x rel_tol, 0.01); the strict validator requires it to be at most 1.00 unless the check sets rounding: <reason> (conversion, proration, estimate), which allows rel_tol up to 0.001. raw_value_ok: true declares that the figure legitimately appears in the raw inputs (the validator otherwise rejects pins an agent could satisfy by pasting the export) |
xlsx_no_errors |
path, sheet?, max_errors? | no formula cell evaluates to an Excel error (#NAME?, #VALUE!, #REF!, ...) after recalculation; added after the audit found error cells outside the pinned cells passing |
text_contains_all / _any / text_not_contains |
path, phrases | case-insensitive phrase checks |
text_matches_all |
path, patterns | every regex matches somewhere in the text |
text_numbers_present |
path, numbers, rel_tol?, min_count? | the figures appear in the text within tolerance, ignoring currency symbols, commas, and parentheses |
text_sentence_matches |
path, all, none? | one sentence matches every regex in all and none in none; use it when scattered phrases would pass on unrelated sentences or on a negation |
custom |
module | check(ws, ref) returns a list of {name, passed, detail} |
Task validation
Before release every desk task passed a strict validator: the reference solution passes its checks, the untouched workspace fails them, and repeated generation is byte-deterministic. Value pins that an agent could satisfy by pasting a figure from the inputs are rejected unless the task declares why the figure legitimately appears there. Twenty-eight tasks' supplied fixtures differ from freshly regenerated bytes; the supplied bytes are the release inputs and the difference is documented in docs/validation.md rather than patched away.
Build protocol
The initial requirement is followed by up to three change requests. Each turn is a new one-shot invocation on the carried-forward workspace. Only the owner's text of a change is sent. The agent must leave RESULT.json with a reachable URL, an admin login, and the named restricted login, and the restricted identity must stay consistent across changes.
Enterprise baseline, items 1 to 14
Every build checklist opens with these, with placeholders filled from the task's reference counts. A tester works the list with the URL and RESULT.json and nothing else.
(CORE) [Delivery]
RESULT.jsonhasurl,admin.user,admin.password,restricted.user,restricted.password,notes; the URL loads a login page in a desktop browser on the tester's machine; both logins sign in. How to check: Open the URL; log in as admin, log out, log in as restricted. Fail if a field is missing, the page does not load, or either login is rejected.[Delivery] The URL is reachable from a phone on a different network than the build machine. How to check: On a phone with Wi-Fi off, open the URL; the login page renders and the admin login succeeds. Fail if it does not load, or the address is localhost, 127.0.0.1, or a private LAN IP.
(CORE) [Sharing] Admin can invite a person by email address and choose their role at invite time; the app shows an invitation link the admin can copy; opening that link in a private window lets the invitee set a password and lands them signed in with the chosen role; the invitee then appears in the app's users or team list with that role. How to check: As admin, invite
tester.viewer@example.comas the read-only role, copy the link, open it in a private window, set a password, confirm the session is signed in as that person and that the users list shows them with the read-only role. Fail if no invite exists, the link is not shown, the link asks for anything other than a password, or the role differs.[Sharing] Invitation links are single-use and invitations and users are revocable: opening an accepted link again does not sign anyone in; admin can revoke a pending invitation; admin can remove or deactivate a user, after which that user's login is rejected. How to check: Reopen the item-3 link in a fresh private window (expect a clear failure). Invite
tester.pending@example.com, revoke it, open its link (expect failure). Invitetester.remove@example.comas the read-only role, accept that invitation in a private window, then remove or deactivate that user and try to log in as them (expect rejection). Keep the item-3 viewer for item 5. Fail on any of the three.(CORE) [Permission] At least these three roles exist and hold: Admin manages users and settings and sees every record;
creates and edits records in its scope and has no user management or settings; is read-only, every create, edit, and delete control is absent or rejected. How to check: As the item-3 viewer, try to create one and to edit an existing one (expect no control, or a rejection and no change on reload). As restricted, confirm the users and settings areas are absent or refused. Fail if the viewer can change anything or the staff login can manage users. (CORE) [Permission] Row-level scope: the restricted login sees only
, exactly , and cannot open another scope's record by URL. How to check: As restricted, read the total and open five records to confirm scope. As admin, copy the URL of ; paste it in the restricted session. Pass only on the exact count and an error, not-found, or redirect without rendering that record's details. [API] Authorization is enforced by the server, not the page: replaying a data request from the restricted session with another scope's record id, or with any scope or role filter removed, returns 401, 403, or 404 or contains no foreign record; replaying a create or edit request from the viewer session is rejected the same way. How to check: In the restricted session open devtools > Network, load the list and one record, copy the data requests as cURL, substitute
's id and remove filter parameters, re-issue. In the viewer session copy any write request the UI would send (or craft one against the same endpoint) and re-issue. Fail if any response carries foreign data or any write succeeds. Server-rendered apps with no separate data requests pass this item when items 5 and 6 pass. (CORE) [Dashboard] A dashboard page shows at least four figures computed live from the data:
= , = , = , = ; after the tester creates one , the affected figure changes accordingly on reload. How to check: As admin, open the dashboard and read the four figures (exact values, tolerance 0.01 on money). Create one in scope of , reload, confirm moved by one (or by the amount). Fail if a figure is missing, wrong, or static. [Dashboard] The dashboard respects scope: signed in as restricted it shows the same figures for that scope only:
= . How to check: As restricted, open the dashboard. Fail if it shows company-wide figures or is absent. [Exact] The main list has working search, at least one filter, and column sort: searching
<SEARCH_TERM>returns exactlyrow(s); filtering on = returns ; sorting by descending puts first. How to check: As admin, perform the three operations. Fail on any count or order that differs, or if an operation is missing. [Exact] Export of the main list to CSV produces exactly
data rows and includes the columns . How to check: As admin, export with no filter applied and open the file. Fail if rows or columns differ or export is absent. [Audit] Records show who created and who last changed them and when, and an activity or audit log lists the tester's own create, edit, and delete actions with user and timestamp. How to check: As admin, create one
, edit it, delete it; open the record's history or the audit log. Pass only if all three actions appear attributed to the admin user with timestamps, and an existing imported record shows created-by and updated-at fields. [Rule] Required fields are validated by the server: submitting one
without through the form is rejected, and replaying the create request without that field (devtools > copy as cURL, remove the field) is also rejected. How to check: Perform both. Fail if either creates a record (check the list after reload). (CORE) [Persistence] Data survives a restart of the app and lives on the server, not in the tester's browser. How to check: After the items above, ask the operator to restart the app (or wait 10 minutes), then log in as admin in a different browser. Pass only if the counts and the tester's remaining changes are present. Fail if anything reverted to import values.
Application-specific items follow. Score the initial checklist, each change's new items, and regressions separately; there is no combined desk and build score.
Metrics
- Attempt pass rate: passing artifacts over every scheduled attempt in a complete arm (561 per arm here).
- All-three success: tasks passing in all three repetitions, over 187. Observed consistency, not a confidence interval.
- Resources: median wall time, recorded token usage with cached input priced separately, estimated cost from the bundled price table, with missing observations stated.
- Build measures: accepted baseline items, core delivery criteria, new change requirements, regressions, time and cost across the sequence.
Cells in the complete desk comparison
| Cell | Harness | Model and route | Recorded configuration | Source cohorts |
|---|---|---|---|---|
proto-deepseek | Proto CLI, runtime c8f62dd60 | deepseek/deepseek-v4.1-flash DeepSeek only, no fallback, high reasoning, temperature 0 | runtime_commit: c8f62dd60model: deepseek/deepseek-v4.1-flashprovider: DeepSeek onlyfallbacks: falsereasoning: hightemperature: 0max_tokens: 65536 | broad-desk-v43-c8f62dd60 r1; repeat-desk-v43-c8f62dd60 local r1/r2 map to logical r2/r3 |
codex-sol | Codex CLI 0.154.0 | gpt-5.6-sol OpenAI via Codex CLI, high reasoning, CLI-managed sampling | model: gpt-5.6-solreasoning: highcli_version: 0.154.0sampling: CLI-managed; not asserted matched to Proto | full-1-codex-sol r1/r2/r3 |
Proto cells use builtin skills only and a per-run copy of a bare home, so memory and skills never leak between attempts. The Codex cell uses a benchmark-owned CODEX_HOME and a fake HOME that hides personal skills. The two cohorts were not run at the same time and their sampling controls differ; the comparison is between configured systems, not between models or harnesses in isolation.
Further adapters ship without published results in this release: Proto on GLM-5.3 Flash and Qwen 3.8 Flash (recorded in an earlier snapshot in the repository history under the original grader), Proto on GPT-5.6 tiers through a subscription route, and Proto with access to the ERP.AI application platform. A platform-backed cell is a declared experimental condition, not an invisible advantage, and will be reported as its own labeled campaign.