File tasks·results from 16 September 2026

Results, 16 September 2026

Two AI agent setups each did all 187 file tasks 3 times, and every answer was checked with the same rules. Proto (built by ERP.AI) used DeepSeek V4.1 Flash; Codex (OpenAI) used GPT-5.6 sol. Because the models differ, this compares complete setups, not the agents alone.

Keep in mind

The two setups differ in model, agent and run dates, so this does not show which part made the difference. The final grading rules were fixed after the answers existed, by ERP.AI, which builds Proto; the original grades are kept alongside, and the grading changes section lists every answer whose verdict changed.

Newer results

Desk comparison, 28 September 2026: Proto and Codex on the same model, plus Proto on DeepSeek, each doing every task three times (Proto / DeepSeek V4.1 Flash 502 of 561, Proto / GPT-6 sol 495 of 561, Codex / GPT-6 sol 487 of 561).

Frozen and raw scores

System	Frozen pass rate	Frozen / 561	Raw / 561	All 3 / 187	Median min	p90 min	Est. model cost / attempt (USD)
Proto / DeepSeek V4.1 Flash Proto CLI, runtime c8f62dd60 · DeepSeek only, no fallback, high reasoning, temperature 0	90.37%	507	447	147	2.08	7.59	0.0393
Codex / GPT-5.6 sol Codex CLI 0.154.0 · OpenAI via Codex CLI, high reasoning, CLI-managed sampling	84.31%	473	431	141	2.25	6.72	0.3906

The lead is 6.06 percentage points under the frozen scorer. A paired, task-clustered bootstrap (20,000 resamples, seed 20260916, each task's 3 repetitions kept together) gives a descriptive 95% interval of +1.25 to +11.05 points. Task identifiers, rather than task families, are resampled. The interval does not account for related families, scorer selection, development exposure, or a different workload.

Proto / DeepSeek V4.1 FlashProto CLI, runtime c8f62dd60

90.4%[87.3%, 93.2%]

Codex / GPT-5.6 solCodex CLI 0.154.0

84.3%[79.9%, 88.6%]

Figure 1. Attempt pass rate (pass@1, filled), share of tasks passed in all 3 repetitions (pass^3, open) and in at least one (pass@3, square). Whiskers: 95% percentile intervals from 20,000 bootstrap resamples of the 187 tasks, repetitions kept together; the value column gives the pass@1 interval. All-three acceptance describes repeated execution of fixed inputs; at-least-one acceptance assumes hindsight selection using the grader. Neither is a deployment success probability. See repeated outcomes.

By repetition

System	Rep 1 frozen	Rep 2 frozen	Rep 3 frozen	Rep 1 raw	Rep 2 raw	Rep 3 raw
Proto / DeepSeek V4.1 Flash	171 / 187	167 / 187	169 / 187	147 / 187	148 / 187	152 / 187
Codex / GPT-5.6 sol	152 / 187	160 / 187	161 / 187	138 / 187	145 / 187	148 / 187

The same supplied inputs are used in each repetition. Repetition identifiers are bookkeeping labels, not matched random seeds or simultaneous trials. Variation here measures execution repeatability; new fixture instances and held-out templates require separate experiments.

By category

Spreadsheet41 tasks

+4.9 ppProto − Codex

Bookkeeping36 tasks

+0.9 ppProto − Codex

Reports30 tasks

+27.8 ppProto − Codex

Reformatting26 tasks

−5.1 ppProto − Codex

Extraction26 tasks

+1.3 ppProto − Codex

Drafting19 tasks

+21.1 ppProto − Codex

Tooling9 tasks

−25.9 ppProto − Codex

Figure 2. Frozen attempt pass rate per category (3 repetitions of each task). The value column is the difference between the two systems in percentage points. Categories hold 9 to 41 tasks, and one attempt changes a category rate by 3.70 points in the smallest category versus 0.81 in the largest. A task changing all 3 repetitions has 3 times that effect.

Counts by category

Category	Attempts	Proto	Codex
Spreadsheet	123	115 / 123	109 / 123
Bookkeeping	108	99 / 108	98 / 108
Reports	90	81 / 90	56 / 90
Reformatting	78	74 / 78	78 / 78
Extraction	78	70 / 78	69 / 78
Drafting	57	49 / 57	37 / 57
Tooling	27	19 / 27	26 / 27

How category weights change the comparison

The published mix weights categories by task counts. Equal category weights change the Proto-minus-Codex gap from 6.06 to 3.55 percentage points. The control below changes weights without rerunning an agent or changing a verdict.

Change category weights to inspect the ranking's dependence on workload composition. Weights are relative: 2 gives a category twice the weight of 1. The released mix gives each task equal weight.

Category weights

Category	Share	Proto observed	Codex observed
Spreadsheet	21.9%	93.50%	88.62%
Bookkeeping	19.3%	91.67%	90.74%
Reports	16.0%	90.00%	62.22%
Reformatting	13.9%	94.87%	100.00%
Extraction	13.9%	89.74%	88.46%
Drafting	10.2%	85.96%	64.91%
Tooling	4.8%	70.37%	96.30%

Weighted observed acceptance: Proto + DeepSeek 90.37%; Codex + Sol 84.31%. Difference, Proto minus Codex: +6.06 pp.

This calculation holds each observed category rate fixed and changes only its weight. It supplies no new-task prediction, confidence interval, failure-severity weighting, or estimate of business demand.

What the frozen scorer changed

The runner recorded original verdicts at execution time. The shared conservative-v7 package then evaluated retained outputs; its source is released at scoring/frozen-v7. It includes native workbook recalculation and declared equivalence rules for 7 tasks. These are evaluator changes on the same artifacts, not improvements from rerunning or repairing an agent's work.

System	Pass → pass	Fail → pass	Ungraded → pass	Pass → fail	Fail → fail	Ungraded → fail	Net pass change
Proto / DeepSeek V4.1 Flash	447	56	4	0	54	0	+60
Codex / GPT-5.6 sol	430	40	3	1	87	0	+42

The original gap is 2.85 points and the frozen gap is 6.06: a 3.21-point change in the measured comparison. Ungraded attempts remain in the full original denominator without being relabeled as ordinary agent failures. Each transition row accounts for all 561 attempts.

The scorer was authored after outputs existed. A shared rule makes the primary scores comparable under that rule; its hash makes later changes detectable. Neither establishes that revisions were independent of the outputs or that every revised judgment is correct. The public ledger supports arithmetic verification, while independent artifact adjudication requires access to retained outputs that remain private.

Equivalence tasks and scorer identity

price-list-page, event-schedule-page, returns-analysis, rent-roll-build, minutes-from-transcript, refund-apology-letter, sop-from-thread.

Scorer manifest: b4720db00f2a55461ca70767a4baf8dd2d25aa9d301e8fe22ea13d21ccca358f. Every frozen receipt in this release points to this manifest. Verdicts are recorded as raw_passed and passed; raw_grader_error_count distinguishes ungraded originals.

Execution accounting

System	Timed out	Nonzero exit	Passing, abnormal exit	Raw grader errors	Frozen grader errors	Missing usage
Proto / DeepSeek V4.1 Flash	3	3	0	4	0	0
Codex / GPT-5.6 sol	0	0	0	3	0	0

Every frozen pass also completed normally. The desk budget was 1,200 seconds per attempt.

Most-failed required check types

System	Failed check types among frozen failures (count)
Proto / DeepSeek V4.1 Flash	`custom` 24`csv_values_match` 23`xlsx_value_present` 19
Codex / GPT-5.6 sol	`xlsx_value_present` 141`xlsx_no_errors` 31`custom` 28

Resource use

System	Input tokens	Cached input	Uncached input	Output tokens	Est. cost, sum USD	Summed task time (h)	Price basis, USD / 1M (in / cached / out)
Proto / DeepSeek V4.1 Flash	823.6M	775.7M	47.9M	20.9M	22.06	31.5	0.15 / 0.003 / 0.6
Codex / GPT-5.6 sol	166.0M	150.3M	15.6M	4.8M	219.13	30.9	4 / 0.4 / 20

Costs are estimates from captured usage and the list prices in bench/prices.json. Codex ran through a subscription and is priced at API list price. Proto has lower estimated model spending, more input and output tokens, and a worse p90 duration. These are different resource dimensions. Summed durations measure occupied task time, not elapsed campaign time under concurrency.

Costs exclude infrastructure, human review, repair, and uncaptured usage. Proto records 16,954 completed model responses and nine request starts without completed responses; Codex exposes 561 task-level usage aggregates rather than comparable request counts. See spending per accepted attempt for the campaign accounting ratio.

Per-task matrix

3 squares per cell, one per repetition. Green is a frozen pass; a green square with a hollow centre passed under the frozen scorer but was originally failed or ungraded; grey is a fail; a hollow grey centre is a timeout. One task passed nowhere; 116 passed everywhere.

pass pass, frozen scorer only fail fail, timed out

Task	Category	Frozen / 6	Raw / 6
Clean the catalog mailing list for the mail house address-standardize	Reformatting	6	6
Build the Amazon inventory upload from the product sheet amazon-inventory-loader	Reformatting	6	6
Payables aging by vendor for the line of credit review ap-aging	Bookkeeping	2	2
Spring no-show rates by tutor under the 24-hour policy appointment-no-shows	Spreadsheet	6	2
Receivables aging at month end for the bank ar-aging-report	Reports	4	4
August bank reconciliation for the checking account bank-reconciliation	Bookkeeping	5	5
Key the July and August bank statements into the transactions file bank-statement-pdf	Extraction	6	6
Material cost per product from the bill of materials bom-cost-rollup	Spreadsheet	4	4
First-half budget vs actual, line by line budget-vs-actual	Spreadsheet	6	6
CRM contacts from the scanned expo business cards business-cards-scanned	Extraction	5	5
Cost per lead by channel for the quarter campaign-cpl-report	Reports	6	6
Code August company-card charges to the ledger by item type card-expense-coding	Bookkeeping	6	6
August transactions from the company card statements card-statements-pdf	Extraction	6	6
August cash drawer variance by day cash-drawer-variance	Bookkeeping	6	6
Money in and out by month for the bank meeting cash-flow-monthly	Reports	4	3
Normalize the catalog sizes for the new POS catalog-normalize-units	Spreadsheet	6	0
Side-by-side table of the anniversary dinner catering quotes catering-quotes-table	Extraction	6	6
Chargeback tracker with evidence deadlines chargeback-tracker	Bookkeeping	4	4
Restate this year's GL lines on the new chart of accounts chart-of-accounts-remap	Reformatting	6	6
Fall to spring enrollment, adds and drops by course class-roster-changes	Spreadsheet	6	6
Q3 hours per client project, billable and not client-hours-by-project	Spreadsheet	6	6
Visits per provider and the no-show rate for Q1 clinic-visits-summary	Reports	3	3
Weekly hours audit from time-clock punches clock-punch-audit	Spreadsheet	6	6
August sales commissions with refund recoveries commission-calculation	Spreadsheet	5	5
Commission clawbacks on refunded deals since June commission-clawbacks	Bookkeeping	6	6
First-half complaints by category and month complaints-summary	Reports	6	0
Pull key terms from our service contracts into a register contract-key-terms	Extraction	6	6
Renewal summary for the Wi-Fi services contract contract-renewal-summary	Drafting	5	4
2025 1099-NEC recipients into the e-file service's import file contractor-1099-list	Reformatting	6	6
Turn September returns into credit notes and apply them credit-notes-apply	Bookkeeping	6	6
Clean a legacy customer export for HubSpot import customer-dedupe	Reformatting	6	6
Lifetime value per customer from the shop export customer-value-summary	Spreadsheet	6	6
Applicant list from the operations coordinator resumes cv-batch-extraction	Extraction	6	6
2026 deferred revenue schedule for the annual plans deferred-revenue-schedule	Bookkeeping	4	3
On-time delivery by hub, and why the late ones were late delivery-performance	Reports	4	4
2026 depreciation schedule from the fixed asset register depreciation-schedule	Bookkeeping	6	6
Q2 expenses by department against budget dept-expense-report	Reports	4	3
FY2026 fundraising figures for the board donor-annual-figures	Reports	2	2
Per-donor giving totals for 2024 and 2025 donor-totals-by-year	Spreadsheet	6	6
Find bills that were entered twice before the check run duplicate-invoice-finder	Spreadsheet	6	6
Find supplier payments we made twice duplicate-payments	Bookkeeping	6	6
Monthly electricity by building from meter reads energy-usage-sites	Reports	4	2
Announcement for the wine club harvest dinner event-announcement	Drafting	6	6
Door list for the author evening event-attendee-merge	Spreadsheet	5	5
Wrap-up numbers for the summit event-postmortem	Reports	6	3
Conference schedule page for the lobby screen event-schedule-page	Tooling	5	4
Load the concert sign-ups into Eventbrite eventbrite-attendee-import	Reformatting	6	6
Categorize card transactions per policy, with thread overrides expense-categorize	Bookkeeping	5	5
Audit August expense reports against the travel policy expense-report-audit	Bookkeeping	6	6
Website FAQ from the summer support tickets faq-from-tickets	Drafting	6	0
First-half FIFO cost of goods sold fifo-cogs	Bookkeeping	4	4
Cost per mile per vehicle for the first half fleet-cost-per-mile	Reports	6	6
Monthly miles per van for the insurer fleet-mileage-monthly	Spreadsheet	6	6
Boat campaign progress page with top donors fundraiser-progress-page	Tooling	5	5
Fall appeal letter for the Hatch Farm campaign fundraising-appeal	Drafting	6	6
Realized FX gain or loss on the euro invoices fx-invoice-gains	Bookkeeping	4	4
Gift card liability per card at year end gift-card-liability	Bookkeeping	6	6
Move the office phone list into Google Contacts google-contacts-import	Reformatting	6	6
Final weighted grades for the bookkeeping course gradebook-weighted	Spreadsheet	5	5
Log this year's grant award letters into the tracker grant-award-letters	Extraction	6	1
Grant budget tracking for the board packet grant-budget-tracking	Bookkeeping	6	6
Build the Gusto employee upload from the HR roster gusto-employee-import	Reformatting	6	6
May helpdesk tickets by category and SLA breaches helpdesk-tickets-report	Reports	3	3
Import the roofing pipeline into HubSpot deals hubspot-deals-import	Reformatting	6	6
Formal incident report for the Line 3 injury incident-report	Drafting	5	5
Half-year safety incidents by site and type incident-summary	Reports	5	5
Open findings from this season's inspection reports inspection-findings	Extraction	6	6
Log vendor insurance certificates and flag lapsed policies insurance-certificates	Extraction	6	6
Reconcile the physical count against the system stock inventory-count-reconcile	Spreadsheet	4	0
Shrinkage by department from the Sunday floor count inventory-shrinkage	Bookkeeping	6	3
Stock page for the workshop tablet inventory-tracker-page	Tooling	6	6
Stock valued at cost by department for the accountant inventory-valuation	Spreadsheet	6	6
Q2 investor update letter from the KPI tracker investor-update	Drafting	3	2
Write the front desk coordinator job posting job-posting-from-notes	Drafting	6	2
August operations scorecard page kpi-scorecard-page	Tooling	4	1
Q2 leads and won revenue by marketing source lead-source-rollup	Spreadsheet	6	6
Abstract the plaza leases for the lender lease-abstracts	Extraction	5	5
Check the gym loan's interest and principal split loan-schedule-check	Bookkeeping	6	6
Apply today's lockbox checks to open invoices lockbox-apply	Bookkeeping	6	6
Fall attendance grid for the district grant long-to-wide-attendance	Reformatting	6	6
Summer signups into one Mailchimp audience import mailchimp-audience-import	Reformatting	6	6
Water shutoff notices for the riser valve work maintenance-notices	Drafting	6	5
October renewal letters and the lapsed list membership-renewals	Spreadsheet	6	6
Cost out the fall menu, plate by plate menu-cost-sheet	Spreadsheet	6	6
August menu items ranked by sales and margin menu-item-performance	Reports	5	5
Fall menu page with allergens for the QR code menu-page-allergens	Tooling	6	6
August mileage reimbursement for the field staff mileage-reimbursement	Bookkeeping	6	0
Minutes from the leadership meeting recording minutes-from-transcript	Drafting	3	0
August accrual entry for the theatre month-end-accruals	Bookkeeping	6	6
Monthly revenue report by region from a messy sales export monthly-report	Reports	5	5
Monthly revenue report by region from a messy sales export monthly-report-v9	Reports	4	4
MRR by month and churn for the first half mrr-report	Reports	2	2
First-half sales in dollars across three currencies multi-currency-sales	Spreadsheet	6	6
Split the tracker notes into Pipedrive activities notes-to-activities	Reformatting	4	4
Q1 occupancy per cabin per month occupancy-report	Reports	6	0
Welcome email for the new project engineer onboarding-welcome-email	Drafting	5	0
What is still coming on our purchase orders open-purchase-orders	Spreadsheet	6	6
This week's orders page for the packing bench orders-status-page	Tooling	5	5
Write this month's past-due reminders overdue-reminders	Drafting	3	3
Match this week's packing slips to the open POs packing-slips-vs-po	Extraction	6	6
Key the August pay stubs for the CPA pay-stubs-extraction	Extraction	6	6
Match bank payments to open invoices payments-match	Bookkeeping	6	6
Match bank payments to open invoices, with card fees, short wires, and a wrong reference payments-match-v2	Bookkeeping	6	6
Journal entry for the 4 September payroll payroll-journal-entry	Bookkeeping	6	6
2025 payroll by quarter for the accountant payroll-quarterly-summary	Reports	6	6
Federal payroll tax deposits and due dates since the provider stopped payroll-tax-deposits	Bookkeeping	6	0
Match the scanned field trip slips to the class roster permission-forms-scanned	Extraction	4	4
Reconcile the front desk petty cash and code the top-up petty-cash-reconcile	Bookkeeping	6	6
Convert the CRM contact phones to the new phone system's import file phones-to-e164	Reformatting	6	6
Catch-up prorations for February to April plan changes plan-change-proration	Bookkeeping	6	6
Memo to the crews on the 2026 policy changes policy-update-memo	Drafting	4	0
Load the fall menu into the new POS with modifiers pos-menu-import	Reformatting	6	6
2026 prepaid expense amortization schedule prepaid-amortization	Bookkeeping	6	6
Draft the standing order price increase notice price-increase-notice	Drafting	2	1
Fall wholesale price list page for shop buyers price-list-page	Tooling	4	4
Apply the supplier's price letter to our master price list price-list-update	Spreadsheet	6	6
First-half purchasing savings by category procurement-savings	Reports	6	6
August yield and scrap by production line production-yield	Reports	3	3
West Gallery budget burn for the capital committee project-budget-burn	Bookkeeping	6	6
Margin by job, with the loss-makers flagged project-margin	Reports	6	1
Kitchen job task board for the crew wall project-status-board	Tooling	5	5
How the August promo codes did promo-code-analysis	Spreadsheet	5	5
Log this year's property tax bills property-tax-bills	Extraction	6	6
Write up the warehouse lighting proposal proposal-from-notes	Drafting	5	5
Key this week's customer POs into order entry purchase-orders-pdf	Extraction	6	6
Q3 wholesale sales by product, with the odd things called out quarterly-sales-report	Reports	5	4
CRM accounts into a QuickBooks Online customer import quickbooks-customer-import	Reformatting	6	6
Compare three laptop quotes (USD/EUR, hidden fees) and recommend one quote-comparison	Extraction	6	6
Expense table from scanned trip receipts receipts-scanned	Extraction	6	6
Reply to a customer about a cracked skillet refund-apology-letter	Drafting	2	1
Reconcile August refunds to the card processor refund-reconciliation	Bookkeeping	5	5
Q2 sales by territory from the monthly exports regional-sales-monthly	Reports	5	5
October rent roll for the bank rent-roll-build	Spreadsheet	5	4
Days out per machine per month for the insurer rental-utilization	Spreadsheet	6	6
This week's reorder list for the bike shop reorder-list	Spreadsheet	6	6
Customer cohort retention table for the first half retention-cohorts	Reports	4	3
Q2 return rate by product and reason returns-analysis	Reports	4	3
Request for quotation for the rolling bench parts rfq-email	Drafting	6	6
Rep by month sales table from the invoice export sales-pivot-by-rep	Spreadsheet	4	4
Sales tax on each September order sales-tax-by-address	Spreadsheet	6	6
Second-quarter sales tax due by local jurisdiction sales-tax-liability	Bookkeeping	6	6
Turn the Q3 leads sheet into a Salesforce lead import salesforce-lead-import	Reformatting	6	6
Schedule change notices for next week's revised rota schedule-change-notices	Drafting	6	4
Fall class schedule as a calendar file for the website schedule-to-ics	Reformatting	6	6
Allocate August overhead to departments shared-cost-allocation	Bookkeeping	6	6
How short the published week is against minimum coverage shift-coverage-gaps	Spreadsheet	6	6
Shipment weights and costs from last week's manifests shipment-manifests	Extraction	6	6
Work out what the carrier should charge per order shipping-rate-lookup	Spreadsheet	5	5
Fall catalog into Shopify's product import shopify-product-import	Reformatting	5	5
Move the shop inventory onto the new SKUs sku-crosswalk-migration	Spreadsheet	6	6
Turn the refunds email thread into a counter procedure sop-from-thread	Drafting	4	3
Load the stock count into Square's item import square-inventory-import	Reformatting	6	6
One staff directory from HR and IT staff-directory-merge	Spreadsheet	6	6
September billable utilization per person staff-utilization	Spreadsheet	2	2
Build the Stripe customer import from the old billing export stripe-customer-import	Reformatting	6	6
Active subscribers and cancellations by month subscription-status-monthly	Spreadsheet	3	3
Key this week's superbills for charge entry superbill-codes	Extraction	6	6
Metric supplier price list into our inch-and-pound catalog supplier-catalog-imperial	Spreadsheet	6	6
Dispute letter for a supplier invoice that does not match the PO supplier-dispute-letter	Drafting	3	3
Key August supplier bills into the AP import file supplier-invoices-to-csv	Extraction	2	2
Member survey scores per question survey-tally	Spreadsheet	6	6
Find double-booked tables in the reservations table-double-bookings	Spreadsheet	6	0
Staff directory page for the backstage screen team-directory-page	Tooling	5	5
August resident balances with late fees tenant-ledger-balances	Bookkeeping	6	6
Third-quarter ledger from the landlord statements tenant-statements	Extraction	0	0
Consolidate the crew timesheets for payroll timesheet-consolidate	Spreadsheet	6	6
Daily hours from the scanned caregiver time sheets timesheets-scanned	Extraction	6	6
Split last week's tip pool by shift tip-pooling	Bookkeeping	5	5
Spring tuition collected and outstanding by program tuition-collections	Reports	4	5
Monthly usage and charges from the utility bills utility-bills-extraction	Extraction	6	6
Flatten the nested fall catalog into the inventory platform's variant import variants-flatten	Reformatting	6	6
2025 reportable totals per vendor, cards and corporations out vendor-1099-totals	Bookkeeping	3	3
Merge the purchasing and accounting vendor lists vendor-contact-merge	Spreadsheet	6	6
Load the fall supplier price sheets for the ordering guide vendor-price-sheets	Extraction	4	4
Year-to-date spend by vendor and category vendor-spend-categories	Spreadsheet	3	3
Tie out the Keystone supply statement to our books vendor-statement-reconcile	Bookkeeping	6	6
Log new vendors' tax forms with masked TINs vendor-tax-forms	Extraction	5	5
Key the September venue invoices into job costing venue-invoices	Extraction	6	6
Volunteer hours per program for the county volunteer-hours	Reports	6	6
Equipment warranties running out in the next 90 days warranty-expiry-list	Spreadsheet	6	6
Spring website traffic by month, crawler taken out web-traffic-summary	Reports	6	6
One-page weekly KPIs from five system exports weekly-kpi-dashboard	Reports	3	3
Reshape the by-month sales report for the planning tool upload wide-to-long-sales	Reformatting	6	6
Turn August web store orders into journal entries woocommerce-orders-to-journal	Reformatting	6	6
August job invoices into Xero's sales invoice import xero-sales-invoices-import	Reformatting	5	5
September checking transactions into YNAB ynab-transactions-import	Reformatting	6	6
Import client contacts as Zendesk users zendesk-users-import	Reformatting	6	6

Provenance

Campaign	`complete-desk-comparison-2026-09-16`
Ledger SHA-256	559ba499d014e1bbf7a17afa132b504cdb2fd2740e6a7f319fec29b40560a242
Scorer manifest SHA-256	b4720db00f2a55461ca70767a4baf8dd2d25aa9d301e8fe22ea13d21ccca358f
Original scorecard SHA-256	04705007d04df365d8ba2b2debbeac109c47e3b6ab55ba1193f4b421e2f64485
Configuration: Proto	runtime_commit: c8f62dd60model: deepseek/deepseek-v4.1-flashprovider: DeepSeek onlyfallbacks: falsereasoning: hightemperature: 0max_tokens: 65536
Configuration: Codex	model: gpt-5.6-solreasoning: highcli_version: 0.154.0sampling: CLI-managed; not asserted matched to Proto
Source cohorts: Proto	broad-desk-v43-c8f62dd60 r1; repeat-desk-v43-c8f62dd60 local r1/r2 map to logical r2/r3
Source cohorts: Codex	full-1-codex-sol r1/r2/r3
Evidence verification	Original result hashes, receipt scorer identity and frozen verdicts checked for every attempt during export; frozen scorer package file hashes verified.
Scorer note	Immutable manifest retains original assembly-status text; subsequent 1341-attempt replay reported zero differences. Original bytes are preserved.
Cost note	Captured-usage API-equivalent estimate from the frozen pricing table, not subscription or reconciled provider billing. Errored/interrupted requests without captured usage have unknown additional cost, not proven zero cost.
Limitations	Development task set; different models and non-contemporaneous cohorts. Not a causal harness-only or unseen-generalization comparison. Raw artifacts remain private; hashes identify but do not reconstruct them.
Build track	No completed fully acceptance-validated build campaign published.
Excluded	Later Proto-vs-Proto experiments, pilots, tuning history, unrelated old arms, raw traces and credentials.

Verify the released data with python bench/export_campaign.py --verify, which checks matrix completeness, score arithmetic, the ledger hash, and every file in the frozen scorer package against results/latest/attempts.jsonl. It does not re-grade the original artifacts, which remain private; their hashes are in the ledger.