Desk track · complete-desk-comparison-2026-09-16 · snapshot 2026-09-16

Results

All 187 desk tasks, 3 repetitions per system, one shared frozen scorer. Every scheduled attempt is in the denominator. The numbers below are recomputed at build time from the released attempt ledger; raw verdicts from the runner's original grader are shown beside the frozen ones throughout.

What this table is. An end-to-end comparison of two configured systems on a development task set. They use different models and sampling controls, and their cohorts were not run at the same time. The Codex cohort is the same three repetitions recorded in the earlier four-system snapshot; the Proto cohort is a later runtime revision. Neither difference is a harness or model effect on its own. See scorer and provenance.

Frozen and raw scores

System	Frozen pass rate	Frozen / 561	Raw / 561	All 3 / 187	Median min	p90 min	Mean cost (USD)
Proto / DeepSeek V4.1 Flash Proto CLI, runtime c8f62dd60 · DeepSeek only, no fallback, high reasoning, temperature 0	90.37%	507	447	147	2.08	7.59	0.0393
Codex / GPT-5.6 sol Codex CLI 0.154.0 · OpenAI via Codex CLI, high reasoning, CLI-managed sampling	84.31%	473	431	141	2.25	6.72	0.3906

The lead is 6.06 percentage points under the frozen scorer. A paired, task-clustered bootstrap (20,000 resamples, seed 20260916, each task's 3 repetitions kept together) gives a descriptive 95% interval of +1.25 to +11.05 points. That is uncertainty over these task families, not evidence about unseen tasks.

By repetition

System	Rep 1 frozen	Rep 2 frozen	Rep 3 frozen	Rep 1 raw	Rep 2 raw	Rep 3 raw
Proto / DeepSeek V4.1 Flash	171 / 187	167 / 187	169 / 187	147 / 187	148 / 187	152 / 187
Codex / GPT-5.6 sol	152 / 187	160 / 187	161 / 187	138 / 187	145 / 187	148 / 187

An aggregate above 90% is not 90% in every repetition; the per-repetition rows are the reliability claim, not the total.

By category

Frozen passes out of tasks × 3 per category. Bars share one scale.

Category	Attempts	Proto / DeepSeek V4.1 Flash	Codex / GPT-5.6 sol
Spreadsheet	123	115 / 123	109 / 123
Bookkeeping	108	99 / 108	98 / 108
Reports	90	81 / 90	56 / 90
Reformatting	78	74 / 78	78 / 78
Extraction	78	70 / 78	69 / 78
Drafting	57	49 / 57	37 / 57
Tooling	27	19 / 27	26 / 27

What the frozen scorer changed

Every attempt was scored twice: once by the runner's grader at run time (the raw verdict) and once by the frozen conservative-v7 package released under scoring/frozen-v7 (the primary verdict). The frozen package carries its own copy of the task definitions and base grader, plus task-specific equivalence graders for 7 tasks that accept alternative valid representations (a rate written as a fraction or a percentage, a schedule laid out as a table or as cards). Its manifest digest is b4720db00f2a55461ca70767a4baf8dd2d25aa9d301e8fe22ea13d21ccca358f and every receipt in the ledger points to it.

System	Raw passes	Frozen passes	Fail → pass	Pass → fail	Of which on equivalence tasks	Fail → pass by category
Proto / DeepSeek V4.1 Flash	447	507	+60	−0	8	Drafting 21Reports 15Spreadsheet 10Bookkeeping 9Tooling 3Extraction 2
Codex / GPT-5.6 sol	431	473	+43	−1	0	Reports 12Spreadsheet 11Drafting 9Bookkeeping 7Extraction 3Tooling 1

The scorer raised both systems. It raised the leader more, so the gap under the frozen scorer is larger than under the original grader. The equivalence tasks are price-list-page, event-schedule-page, returns-analysis, rent-roll-build, minutes-from-transcript, refund-apology-letter, sop-from-thread. Both verdicts are in attempts.jsonl as raw_passed and passed; the original grader had 4 (Proto) and 3 (Codex) grader-error attempts, the frozen scorer none.

Execution accounting

System	Timed out	Nonzero exit	Passing, abnormal exit	Raw grader errors	Frozen grader errors	Missing usage
Proto / DeepSeek V4.1 Flash	3	3	0	4	0	0
Codex / GPT-5.6 sol	0	0	0	3	0	0

Every frozen pass also completed normally. The desk budget was 1,200 seconds per attempt.

Most-failed required check types

System	Failed check types among frozen failures (count)
Proto / DeepSeek V4.1 Flash	`custom` 24`csv_values_match` 23`xlsx_value_present` 19
Codex / GPT-5.6 sol	`xlsx_value_present` 141`xlsx_no_errors` 31`custom` 28

Resource use

System	Input tokens	Cached input	Uncached input	Output tokens	Est. cost, sum USD	Summed task time (h)	Price basis, USD / 1M (in / cached / out)
Proto / DeepSeek V4.1 Flash	823.6M	775.7M	47.9M	20.9M	22.06	31.5	0.15 / 0.003 / 0.6
Codex / GPT-5.6 sol	166.0M	150.3M	15.6M	4.8M	219.13	30.9	4 / 0.4 / 20

Costs are captured-usage estimates under bench/prices.json, not invoices. Codex ran through a subscription and is priced at API list price. The cheaper system did far more token work; lower cost is not lower resource use. Summed task time is not elapsed campaign time.

Per-task matrix

3 squares per cell, one per repetition. Green is a frozen pass; a green square with a hollow centre passed under the frozen scorer but failed under the original grader; grey is a fail; a hollow grey centre is a timeout. One task passed nowhere; 116 passed everywhere.

pass pass, frozen scorer only fail fail, timed out

Task	Category	Frozen / 6	Raw / 6
Clean the catalog mailing list for the mail house address-standardize	Reformatting	6	6
Build the Amazon inventory upload from the product sheet amazon-inventory-loader	Reformatting	6	6
Payables aging by vendor for the line of credit review ap-aging	Bookkeeping	2	2
Spring no-show rates by tutor under the 24-hour policy appointment-no-shows	Spreadsheet	6	2
Receivables aging at month end for the bank ar-aging-report	Reports	4	4
August bank reconciliation for the checking account bank-reconciliation	Bookkeeping	5	5
Key the July and August bank statements into the transactions file bank-statement-pdf	Extraction	6	6
Material cost per product from the bill of materials bom-cost-rollup	Spreadsheet	4	4
First-half budget vs actual, line by line budget-vs-actual	Spreadsheet	6	6
CRM contacts from the scanned expo business cards business-cards-scanned	Extraction	5	5
Cost per lead by channel for the quarter campaign-cpl-report	Reports	6	6
Code August company-card charges to the ledger by item type card-expense-coding	Bookkeeping	6	6
August transactions from the company card statements card-statements-pdf	Extraction	6	6
August cash drawer variance by day cash-drawer-variance	Bookkeeping	6	6
Money in and out by month for the bank meeting cash-flow-monthly	Reports	4	3
Normalize the catalog sizes for the new POS catalog-normalize-units	Spreadsheet	6	0
Side-by-side table of the anniversary dinner catering quotes catering-quotes-table	Extraction	6	6
Chargeback tracker with evidence deadlines chargeback-tracker	Bookkeeping	4	4
Restate this year's GL lines on the new chart of accounts chart-of-accounts-remap	Reformatting	6	6
Fall to spring enrollment, adds and drops by course class-roster-changes	Spreadsheet	6	6
Q3 hours per client project, billable and not client-hours-by-project	Spreadsheet	6	6
Visits per provider and the no-show rate for Q1 clinic-visits-summary	Reports	3	3
Weekly hours audit from time-clock punches clock-punch-audit	Spreadsheet	6	6
August sales commissions with refund recoveries commission-calculation	Spreadsheet	5	5
Commission clawbacks on refunded deals since June commission-clawbacks	Bookkeeping	6	6
First-half complaints by category and month complaints-summary	Reports	6	0
Pull key terms from our service contracts into a register contract-key-terms	Extraction	6	6
Renewal summary for the Wi-Fi services contract contract-renewal-summary	Drafting	5	4
2025 1099-NEC recipients into the e-file service's import file contractor-1099-list	Reformatting	6	6
Turn September returns into credit notes and apply them credit-notes-apply	Bookkeeping	6	6
Clean a legacy customer export for HubSpot import customer-dedupe	Reformatting	6	6
Lifetime value per customer from the shop export customer-value-summary	Spreadsheet	6	6
Applicant list from the operations coordinator resumes cv-batch-extraction	Extraction	6	6
2026 deferred revenue schedule for the annual plans deferred-revenue-schedule	Bookkeeping	4	3
On-time delivery by hub, and why the late ones were late delivery-performance	Reports	4	4
2026 depreciation schedule from the fixed asset register depreciation-schedule	Bookkeeping	6	6
Q2 expenses by department against budget dept-expense-report	Reports	4	3
FY2026 fundraising figures for the board donor-annual-figures	Reports	2	2
Per-donor giving totals for 2024 and 2025 donor-totals-by-year	Spreadsheet	6	6
Find bills that were entered twice before the check run duplicate-invoice-finder	Spreadsheet	6	6
Find supplier payments we made twice duplicate-payments	Bookkeeping	6	6
Monthly electricity by building from meter reads energy-usage-sites	Reports	4	2
Announcement for the wine club harvest dinner event-announcement	Drafting	6	6
Door list for the author evening event-attendee-merge	Spreadsheet	5	5
Wrap-up numbers for the summit event-postmortem	Reports	6	3
Conference schedule page for the lobby screen event-schedule-page	Tooling	5	4
Load the concert sign-ups into Eventbrite eventbrite-attendee-import	Reformatting	6	6
Categorize card transactions per policy, with thread overrides expense-categorize	Bookkeeping	5	5
Audit August expense reports against the travel policy expense-report-audit	Bookkeeping	6	6
Website FAQ from the summer support tickets faq-from-tickets	Drafting	6	0
First-half FIFO cost of goods sold fifo-cogs	Bookkeeping	4	4
Cost per mile per vehicle for the first half fleet-cost-per-mile	Reports	6	6
Monthly miles per van for the insurer fleet-mileage-monthly	Spreadsheet	6	6
Boat campaign progress page with top donors fundraiser-progress-page	Tooling	5	5
Fall appeal letter for the Hatch Farm campaign fundraising-appeal	Drafting	6	6
Realized FX gain or loss on the euro invoices fx-invoice-gains	Bookkeeping	4	4
Gift card liability per card at year end gift-card-liability	Bookkeeping	6	6
Move the office phone list into Google Contacts google-contacts-import	Reformatting	6	6
Final weighted grades for the bookkeeping course gradebook-weighted	Spreadsheet	5	5
Log this year's grant award letters into the tracker grant-award-letters	Extraction	6	1
Grant budget tracking for the board packet grant-budget-tracking	Bookkeeping	6	6
Build the Gusto employee upload from the HR roster gusto-employee-import	Reformatting	6	6
May helpdesk tickets by category and SLA breaches helpdesk-tickets-report	Reports	3	3
Import the roofing pipeline into HubSpot deals hubspot-deals-import	Reformatting	6	6
Formal incident report for the Line 3 injury incident-report	Drafting	5	5
Half-year safety incidents by site and type incident-summary	Reports	5	5
Open findings from this season's inspection reports inspection-findings	Extraction	6	6
Log vendor insurance certificates and flag lapsed policies insurance-certificates	Extraction	6	6
Reconcile the physical count against the system stock inventory-count-reconcile	Spreadsheet	4	0
Shrinkage by department from the Sunday floor count inventory-shrinkage	Bookkeeping	6	3
Stock page for the workshop tablet inventory-tracker-page	Tooling	6	6
Stock valued at cost by department for the accountant inventory-valuation	Spreadsheet	6	6
Q2 investor update letter from the KPI tracker investor-update	Drafting	3	2
Write the front desk coordinator job posting job-posting-from-notes	Drafting	6	2
August operations scorecard page kpi-scorecard-page	Tooling	4	1
Q2 leads and won revenue by marketing source lead-source-rollup	Spreadsheet	6	6
Abstract the plaza leases for the lender lease-abstracts	Extraction	5	5
Check the gym loan's interest and principal split loan-schedule-check	Bookkeeping	6	6
Apply today's lockbox checks to open invoices lockbox-apply	Bookkeeping	6	6
Fall attendance grid for the district grant long-to-wide-attendance	Reformatting	6	6
Summer signups into one Mailchimp audience import mailchimp-audience-import	Reformatting	6	6
Water shutoff notices for the riser valve work maintenance-notices	Drafting	6	5
October renewal letters and the lapsed list membership-renewals	Spreadsheet	6	6
Cost out the fall menu, plate by plate menu-cost-sheet	Spreadsheet	6	6
August menu items ranked by sales and margin menu-item-performance	Reports	5	5
Fall menu page with allergens for the QR code menu-page-allergens	Tooling	6	6
August mileage reimbursement for the field staff mileage-reimbursement	Bookkeeping	6	0
Minutes from the leadership meeting recording minutes-from-transcript	Drafting	3	0
August accrual entry for the theatre month-end-accruals	Bookkeeping	6	6
Monthly revenue report by region from a messy sales export monthly-report	Reports	5	5
Monthly revenue report by region from a messy sales export monthly-report-v9	Reports	4	4
MRR by month and churn for the first half mrr-report	Reports	2	2
First-half sales in dollars across three currencies multi-currency-sales	Spreadsheet	6	6
Split the tracker notes into Pipedrive activities notes-to-activities	Reformatting	4	4
Q1 occupancy per cabin per month occupancy-report	Reports	6	0
Welcome email for the new project engineer onboarding-welcome-email	Drafting	5	0
What is still coming on our purchase orders open-purchase-orders	Spreadsheet	6	6
This week's orders page for the packing bench orders-status-page	Tooling	5	5
Write this month's past-due reminders overdue-reminders	Drafting	3	3
Match this week's packing slips to the open POs packing-slips-vs-po	Extraction	6	6
Key the August pay stubs for the CPA pay-stubs-extraction	Extraction	6	6
Match bank payments to open invoices payments-match	Bookkeeping	6	6
Match bank payments to open invoices, with card fees, short wires, and a wrong reference payments-match-v2	Bookkeeping	6	6
Journal entry for the 4 September payroll payroll-journal-entry	Bookkeeping	6	6
2025 payroll by quarter for the accountant payroll-quarterly-summary	Reports	6	6
Federal payroll tax deposits and due dates since the provider stopped payroll-tax-deposits	Bookkeeping	6	0
Match the scanned field trip slips to the class roster permission-forms-scanned	Extraction	4	4
Reconcile the front desk petty cash and code the top-up petty-cash-reconcile	Bookkeeping	6	6
Convert the CRM contact phones to the new phone system's import file phones-to-e164	Reformatting	6	6
Catch-up prorations for February to April plan changes plan-change-proration	Bookkeeping	6	6
Memo to the crews on the 2026 policy changes policy-update-memo	Drafting	4	0
Load the fall menu into the new POS with modifiers pos-menu-import	Reformatting	6	6
2026 prepaid expense amortization schedule prepaid-amortization	Bookkeeping	6	6
Draft the standing order price increase notice price-increase-notice	Drafting	2	1
Fall wholesale price list page for shop buyers price-list-page	Tooling	4	4
Apply the supplier's price letter to our master price list price-list-update	Spreadsheet	6	6
First-half purchasing savings by category procurement-savings	Reports	6	6
August yield and scrap by production line production-yield	Reports	3	3
West Gallery budget burn for the capital committee project-budget-burn	Bookkeeping	6	6
Margin by job, with the loss-makers flagged project-margin	Reports	6	1
Kitchen job task board for the crew wall project-status-board	Tooling	5	5
How the August promo codes did promo-code-analysis	Spreadsheet	5	5
Log this year's property tax bills property-tax-bills	Extraction	6	6
Write up the warehouse lighting proposal proposal-from-notes	Drafting	5	5
Key this week's customer POs into order entry purchase-orders-pdf	Extraction	6	6
Q3 wholesale sales by product, with the odd things called out quarterly-sales-report	Reports	5	4
CRM accounts into a QuickBooks Online customer import quickbooks-customer-import	Reformatting	6	6
Compare three laptop quotes (USD/EUR, hidden fees) and recommend one quote-comparison	Extraction	6	6
Expense table from scanned trip receipts receipts-scanned	Extraction	6	6
Reply to a customer about a cracked skillet refund-apology-letter	Drafting	2	1
Reconcile August refunds to the card processor refund-reconciliation	Bookkeeping	5	5
Q2 sales by territory from the monthly exports regional-sales-monthly	Reports	5	5
October rent roll for the bank rent-roll-build	Spreadsheet	5	4
Days out per machine per month for the insurer rental-utilization	Spreadsheet	6	6
This week's reorder list for the bike shop reorder-list	Spreadsheet	6	6
Customer cohort retention table for the first half retention-cohorts	Reports	4	3
Q2 return rate by product and reason returns-analysis	Reports	4	3
Request for quotation for the rolling bench parts rfq-email	Drafting	6	6
Rep by month sales table from the invoice export sales-pivot-by-rep	Spreadsheet	4	4
Sales tax on each September order sales-tax-by-address	Spreadsheet	6	6
Second-quarter sales tax due by local jurisdiction sales-tax-liability	Bookkeeping	6	6
Turn the Q3 leads sheet into a Salesforce lead import salesforce-lead-import	Reformatting	6	6
Schedule change notices for next week's revised rota schedule-change-notices	Drafting	6	4
Fall class schedule as a calendar file for the website schedule-to-ics	Reformatting	6	6
Allocate August overhead to departments shared-cost-allocation	Bookkeeping	6	6
How short the published week is against minimum coverage shift-coverage-gaps	Spreadsheet	6	6
Shipment weights and costs from last week's manifests shipment-manifests	Extraction	6	6
Work out what the carrier should charge per order shipping-rate-lookup	Spreadsheet	5	5
Fall catalog into Shopify's product import shopify-product-import	Reformatting	5	5
Move the shop inventory onto the new SKUs sku-crosswalk-migration	Spreadsheet	6	6
Turn the refunds email thread into a counter procedure sop-from-thread	Drafting	4	3
Load the stock count into Square's item import square-inventory-import	Reformatting	6	6
One staff directory from HR and IT staff-directory-merge	Spreadsheet	6	6
September billable utilization per person staff-utilization	Spreadsheet	2	2
Build the Stripe customer import from the old billing export stripe-customer-import	Reformatting	6	6
Active subscribers and cancellations by month subscription-status-monthly	Spreadsheet	3	3
Key this week's superbills for charge entry superbill-codes	Extraction	6	6
Metric supplier price list into our inch-and-pound catalog supplier-catalog-imperial	Spreadsheet	6	6
Dispute letter for a supplier invoice that does not match the PO supplier-dispute-letter	Drafting	3	3
Key August supplier bills into the AP import file supplier-invoices-to-csv	Extraction	2	2
Member survey scores per question survey-tally	Spreadsheet	6	6
Find double-booked tables in the reservations table-double-bookings	Spreadsheet	6	0
Staff directory page for the backstage screen team-directory-page	Tooling	5	5
August resident balances with late fees tenant-ledger-balances	Bookkeeping	6	6
Third-quarter ledger from the landlord statements tenant-statements	Extraction	0	0
Consolidate the crew timesheets for payroll timesheet-consolidate	Spreadsheet	6	6
Daily hours from the scanned caregiver time sheets timesheets-scanned	Extraction	6	6
Split last week's tip pool by shift tip-pooling	Bookkeeping	5	5
Spring tuition collected and outstanding by program tuition-collections	Reports	4	5
Monthly usage and charges from the utility bills utility-bills-extraction	Extraction	6	6
Flatten the nested fall catalog into the inventory platform's variant import variants-flatten	Reformatting	6	6
2025 reportable totals per vendor, cards and corporations out vendor-1099-totals	Bookkeeping	3	3
Merge the purchasing and accounting vendor lists vendor-contact-merge	Spreadsheet	6	6
Load the fall supplier price sheets for the ordering guide vendor-price-sheets	Extraction	4	4
Year-to-date spend by vendor and category vendor-spend-categories	Spreadsheet	3	3
Tie out the Keystone supply statement to our books vendor-statement-reconcile	Bookkeeping	6	6
Log new vendors' tax forms with masked TINs vendor-tax-forms	Extraction	5	5
Key the September venue invoices into job costing venue-invoices	Extraction	6	6
Volunteer hours per program for the county volunteer-hours	Reports	6	6
Equipment warranties running out in the next 90 days warranty-expiry-list	Spreadsheet	6	6
Spring website traffic by month, crawler taken out web-traffic-summary	Reports	6	6
One-page weekly KPIs from five system exports weekly-kpi-dashboard	Reports	3	3
Reshape the by-month sales report for the planning tool upload wide-to-long-sales	Reformatting	6	6
Turn August web store orders into journal entries woocommerce-orders-to-journal	Reformatting	6	6
August job invoices into Xero's sales invoice import xero-sales-invoices-import	Reformatting	5	5
September checking transactions into YNAB ynab-transactions-import	Reformatting	6	6
Import client contacts as Zendesk users zendesk-users-import	Reformatting	6	6

Provenance

Campaign	`complete-desk-comparison-2026-09-16`
Ledger SHA-256	559ba499d014e1bbf7a17afa132b504cdb2fd2740e6a7f319fec29b40560a242
Scorer manifest SHA-256	b4720db00f2a55461ca70767a4baf8dd2d25aa9d301e8fe22ea13d21ccca358f
Original scorecard SHA-256	04705007d04df365d8ba2b2debbeac109c47e3b6ab55ba1193f4b421e2f64485
Configuration: Proto	runtime_commit: c8f62dd60model: deepseek/deepseek-v4.1-flashprovider: DeepSeek onlyfallbacks: falsereasoning: hightemperature: 0max_tokens: 65536
Configuration: Codex	model: gpt-5.6-solreasoning: highcli_version: 0.154.0sampling: CLI-managed; not asserted matched to Proto
Source cohorts: Proto	broad-desk-v43-c8f62dd60 r1; repeat-desk-v43-c8f62dd60 local r1/r2 map to logical r2/r3
Source cohorts: Codex	full-1-codex-sol r1/r2/r3
Evidence verification	Original result hashes, receipt scorer identity and frozen verdicts checked for every attempt during export; frozen scorer package file hashes verified.
Scorer note	Immutable manifest retains original assembly-status text; subsequent 1341-attempt replay reported zero differences. Original bytes are preserved.
Cost note	Captured-usage API-equivalent estimate from the frozen pricing table, not subscription or reconciled provider billing. Errored/interrupted requests without captured usage have unknown additional cost, not proven zero cost.
Limitations	Development task set; different models and non-contemporaneous cohorts. Not a causal harness-only or unseen-generalization comparison. Raw artifacts remain private; hashes identify but do not reconstruct them.
Build track	No completed fully acceptance-validated build campaign published.
Excluded	Later Proto-vs-Proto experiments, pilots, tuning history, unrelated old arms, raw traces and credentials.

Recompute this page with python bench/export_campaign.py --verify, which checks matrix completeness, score arithmetic, the ledger hash, and every file in the frozen scorer package against results/latest/attempts.jsonl. It does not re-grade the original artifacts, which remain private; their hashes are in the ledger.