Results·28 September 2026, 05:29–10:48 UTC

Desk comparison, 28 September 2026

Three AI agent setups each did all 187 file tasks three times, and every answer was checked with the same rules: 1,683 answers in total, none left out.

Keep in mind

Proto is built by ERP.AI, which also runs this benchmark; Codex is OpenAI's coding agent. The two setups on GPT-6 sol share the model, its settings and the account, so their difference comes from the agent. The tasks were available while Proto was developed, and the three rounds did not all run under the same machine load, so compare task times with care. These results are separate from the 16 September results.

Frozen and raw scores

System	Frozen pass rate	Frozen / 561	By repetition	All 3 / 187	Raw / 561	Timed out	Median min	p90 min
Proto / DeepSeek V4.1 Flash Proto CLI, unreleased build · DeepSeek API direct, high reasoning, temperature 0.7	89.5%	502	171 · 168 · 163	147	444	1	1.30	3.12
Proto / GPT-6 sol Proto CLI, unreleased build · ChatGPT subscription through the Codex sign-in, high reasoning	88.2%	495	165 · 167 · 163	153	443	1	1.91	3.63
Codex / GPT-6 sol Codex CLI 0.158.0-alpha.2.1 · ChatGPT subscription, high reasoning, CLI-managed sampling	86.8%	487	162 · 162 · 163	144	440	0	1.48	3.98

Proto / GPT-6 sol minus Codex / GPT-6 sol is +1.43 percentage points under the frozen scorer. A paired, task-clustered bootstrap (20,000 resamples, seed 20260928) gives a descriptive 95% interval of −1.78 to +4.81 points. The two share model gpt-6-sol and reasoning high. The interval describes variation across these tasks; it excludes related task families, evaluator selection and deployment shift.

The same supplied inputs are used in each repetition, so variation here measures execution repeatability, not performance on new business cases. Task times depend on what else was running (see Provenance).

By category

Category	Tasks	Proto DeepSeek	Proto GPT-6	Codex
Spreadsheet	41	116 / 123	120 / 123	115 / 123
Bookkeeping	36	94 / 108	95 / 108	92 / 108
Reports	30	78 / 90	67 / 90	68 / 90
Reformatting	26	73 / 78	75 / 78	77 / 78
Extraction	26	73 / 78	71 / 78	72 / 78
Drafting	19	51 / 57	44 / 57	40 / 57
Tooling	9	17 / 27	23 / 27	23 / 27

Categories hold between 9 and 41 tasks, so one attempt moves a category rate by several points.

What the frozen scorer changed

The runner recorded original verdicts at execution time. The same frozen package used for the release, scoring/frozen-v7, then scored each retained output. These are evaluator changes on the same artifacts, not reruns or repairs.

System	Pass → pass	Fail → pass	Ungraded → pass	Pass → fail	Fail → fail	Ungraded → fail	Net pass change
Proto / DeepSeek V4.1 Flash	444	55	3	0	58	1	+58
Proto / GPT-6 sol	442	50	3	1	65	0	+52
Codex / GPT-6 sol	439	45	3	1	73	0	+47

System	Nonzero exit	Passing, abnormal exit	Raw grader errors
Proto / DeepSeek V4.1 Flash	1	1	4
Proto / GPT-6 sol	1	0	3
Codex / GPT-6 sol	0	0	3

Resource use

System	Input tokens	Cached input	Uncached input	Output tokens	Summed task time (h)
Proto / DeepSeek V4.1 Flash	300.1M	275.3M	24.8M	10.9M	16.6
Proto / GPT-6 sol	172.7M	153.4M	19.3M	2.8M	21.4
Codex / GPT-6 sol	92.8M	81.6M	11.2M	3.0M	19.6

No cost estimates are published for this campaign. The frozen pricing table has no gpt-6-sol or direct DeepSeek V4.1 Flash rate. Codex CLI results had been priced at gpt-5.6-sol rates because of a default model label, so those figures are omitted rather than published mispriced. Both gpt-6-sol cells used a ChatGPT subscription, which is not billed per token. Token usage for the Proto cells comes from Proto's provider traces and for Codex from the CLI's reported turn usage. Codex CLI results carried the usage extractor's default model label gpt-5.6-sol; the CLI's own session context records gpt-6-sol at high effort, so that key is renamed in usage.by_model. Summed durations measure occupied task time, not elapsed campaign time under concurrency.

Per-task matrix

One square per repetition. Green is a frozen pass; a green square with a hollow centre passed under the frozen scorer but was originally failed or ungraded; grey is a fail; a hollow grey centre is a timeout. 0 tasks passed for no system; 114 passed for every system.

pass pass, frozen scorer only fail fail, timed out

Task	Category	Frozen / 9	Raw / 9
Clean the catalog mailing list for the mail house address-standardize	Reformatting	9	9
Build the Amazon inventory upload from the product sheet amazon-inventory-loader	Reformatting	9	9
Payables aging by vendor for the line of credit review ap-aging	Bookkeeping	6	6
Spring no-show rates by tutor under the 24-hour policy appointment-no-shows	Spreadsheet	9	2
Receivables aging at month end for the bank ar-aging-report	Reports	8	8
August bank reconciliation for the checking account bank-reconciliation	Bookkeeping	7	7
Key the July and August bank statements into the transactions file bank-statement-pdf	Extraction	9	9
Material cost per product from the bill of materials bom-cost-rollup	Spreadsheet	9	9
First-half budget vs actual, line by line budget-vs-actual	Spreadsheet	9	9
CRM contacts from the scanned expo business cards business-cards-scanned	Extraction	9	9
Cost per lead by channel for the quarter campaign-cpl-report	Reports	9	9
Code August company-card charges to the ledger by item type card-expense-coding	Bookkeeping	8	8
August transactions from the company card statements card-statements-pdf	Extraction	9	9
August cash drawer variance by day cash-drawer-variance	Bookkeeping	9	9
Money in and out by month for the bank meeting cash-flow-monthly	Reports	8	8
Normalize the catalog sizes for the new POS catalog-normalize-units	Spreadsheet	9	6
Side-by-side table of the anniversary dinner catering quotes catering-quotes-table	Extraction	8	8
Chargeback tracker with evidence deadlines chargeback-tracker	Bookkeeping	9	9
Restate this year's GL lines on the new chart of accounts chart-of-accounts-remap	Reformatting	9	9
Fall to spring enrollment, adds and drops by course class-roster-changes	Spreadsheet	9	9
Q3 hours per client project, billable and not client-hours-by-project	Spreadsheet	6	6
Visits per provider and the no-show rate for Q1 clinic-visits-summary	Reports	7	7
Weekly hours audit from time-clock punches clock-punch-audit	Spreadsheet	9	9
August sales commissions with refund recoveries commission-calculation	Spreadsheet	8	8
Commission clawbacks on refunded deals since June commission-clawbacks	Bookkeeping	9	9
First-half complaints by category and month complaints-summary	Reports	9	0
Pull key terms from our service contracts into a register contract-key-terms	Extraction	7	7
Renewal summary for the Wi-Fi services contract contract-renewal-summary	Drafting	7	4
2025 1099-NEC recipients into the e-file service's import file contractor-1099-list	Reformatting	9	9
Turn September returns into credit notes and apply them credit-notes-apply	Bookkeeping	8	8
Clean a legacy customer export for HubSpot import customer-dedupe	Reformatting	8	8
Lifetime value per customer from the shop export customer-value-summary	Spreadsheet	9	9
Applicant list from the operations coordinator resumes cv-batch-extraction	Extraction	9	9
2026 deferred revenue schedule for the annual plans deferred-revenue-schedule	Bookkeeping	4	4
On-time delivery by hub, and why the late ones were late delivery-performance	Reports	9	2
2026 depreciation schedule from the fixed asset register depreciation-schedule	Bookkeeping	9	9
Q2 expenses by department against budget dept-expense-report	Reports	8	8
FY2026 fundraising figures for the board donor-annual-figures	Reports	3	3
Per-donor giving totals for 2024 and 2025 donor-totals-by-year	Spreadsheet	8	8
Find bills that were entered twice before the check run duplicate-invoice-finder	Spreadsheet	9	9
Find supplier payments we made twice duplicate-payments	Bookkeeping	9	9
Monthly electricity by building from meter reads energy-usage-sites	Reports	6	1
Announcement for the wine club harvest dinner event-announcement	Drafting	9	9
Door list for the author evening event-attendee-merge	Spreadsheet	9	9
Wrap-up numbers for the summit event-postmortem	Reports	7	0
Conference schedule page for the lobby screen event-schedule-page	Tooling	9	7
Load the concert sign-ups into Eventbrite eventbrite-attendee-import	Reformatting	9	9
Categorize card transactions per policy, with thread overrides expense-categorize	Bookkeeping	2	2
Audit August expense reports against the travel policy expense-report-audit	Bookkeeping	9	9
Website FAQ from the summer support tickets faq-from-tickets	Drafting	8	1
First-half FIFO cost of goods sold fifo-cogs	Bookkeeping	8	8
Cost per mile per vehicle for the first half fleet-cost-per-mile	Reports	9	9
Monthly miles per van for the insurer fleet-mileage-monthly	Spreadsheet	9	9
Boat campaign progress page with top donors fundraiser-progress-page	Tooling	7	7
Fall appeal letter for the Hatch Farm campaign fundraising-appeal	Drafting	9	8
Realized FX gain or loss on the euro invoices fx-invoice-gains	Bookkeeping	8	8
Gift card liability per card at year end gift-card-liability	Bookkeeping	9	9
Move the office phone list into Google Contacts google-contacts-import	Reformatting	9	9
Final weighted grades for the bookkeeping course gradebook-weighted	Spreadsheet	7	7
Log this year's grant award letters into the tracker grant-award-letters	Extraction	9	6
Grant budget tracking for the board packet grant-budget-tracking	Bookkeeping	8	8
Build the Gusto employee upload from the HR roster gusto-employee-import	Reformatting	9	9
May helpdesk tickets by category and SLA breaches helpdesk-tickets-report	Reports	3	3
Import the roofing pipeline into HubSpot deals hubspot-deals-import	Reformatting	9	9
Formal incident report for the Line 3 injury incident-report	Drafting	8	8
Half-year safety incidents by site and type incident-summary	Reports	6	6
Open findings from this season's inspection reports inspection-findings	Extraction	9	9
Log vendor insurance certificates and flag lapsed policies insurance-certificates	Extraction	9	9
Reconcile the physical count against the system stock inventory-count-reconcile	Spreadsheet	6	0
Shrinkage by department from the Sunday floor count inventory-shrinkage	Bookkeeping	9	5
Stock page for the workshop tablet inventory-tracker-page	Tooling	7	7
Stock valued at cost by department for the accountant inventory-valuation	Spreadsheet	7	7
Q2 investor update letter from the KPI tracker investor-update	Drafting	6	6
Write the front desk coordinator job posting job-posting-from-notes	Drafting	9	6
August operations scorecard page kpi-scorecard-page	Tooling	8	2
Q2 leads and won revenue by marketing source lead-source-rollup	Spreadsheet	9	9
Abstract the plaza leases for the lender lease-abstracts	Extraction	9	9
Check the gym loan's interest and principal split loan-schedule-check	Bookkeeping	8	8
Apply today's lockbox checks to open invoices lockbox-apply	Bookkeeping	9	9
Fall attendance grid for the district grant long-to-wide-attendance	Reformatting	9	9
Summer signups into one Mailchimp audience import mailchimp-audience-import	Reformatting	8	8
Water shutoff notices for the riser valve work maintenance-notices	Drafting	9	8
October renewal letters and the lapsed list membership-renewals	Spreadsheet	9	9
Cost out the fall menu, plate by plate menu-cost-sheet	Spreadsheet	9	9
August menu items ranked by sales and margin menu-item-performance	Reports	7	7
Fall menu page with allergens for the QR code menu-page-allergens	Tooling	9	9
August mileage reimbursement for the field staff mileage-reimbursement	Bookkeeping	8	0
Minutes from the leadership meeting recording minutes-from-transcript	Drafting	2	0
August accrual entry for the theatre month-end-accruals	Bookkeeping	9	9
Monthly revenue report by region from a messy sales export monthly-report	Reports	4	4
Monthly revenue report by region from a messy sales export monthly-report-v9	Reports	5	4
MRR by month and churn for the first half mrr-report	Reports	4	4
First-half sales in dollars across three currencies multi-currency-sales	Spreadsheet	9	9
Split the tracker notes into Pipedrive activities notes-to-activities	Reformatting	9	9
Q1 occupancy per cabin per month occupancy-report	Reports	9	0
Welcome email for the new project engineer onboarding-welcome-email	Drafting	9	2
What is still coming on our purchase orders open-purchase-orders	Spreadsheet	9	9
This week's orders page for the packing bench orders-status-page	Tooling	6	6
Write this month's past-due reminders overdue-reminders	Drafting	3	3
Match this week's packing slips to the open POs packing-slips-vs-po	Extraction	9	9
Key the August pay stubs for the CPA pay-stubs-extraction	Extraction	9	9
Match bank payments to open invoices payments-match	Bookkeeping	9	9
Match bank payments to open invoices, with card fees, short wires, and a wrong reference payments-match-v2	Bookkeeping	4	4
Journal entry for the 4 September payroll payroll-journal-entry	Bookkeeping	9	9
2025 payroll by quarter for the accountant payroll-quarterly-summary	Reports	9	9
Federal payroll tax deposits and due dates since the provider stopped payroll-tax-deposits	Bookkeeping	8	1
Match the scanned field trip slips to the class roster permission-forms-scanned	Extraction	8	8
Reconcile the front desk petty cash and code the top-up petty-cash-reconcile	Bookkeeping	9	9
Convert the CRM contact phones to the new phone system's import file phones-to-e164	Reformatting	9	9
Catch-up prorations for February to April plan changes plan-change-proration	Bookkeeping	9	9
Memo to the crews on the 2026 policy changes policy-update-memo	Drafting	7	2
Load the fall menu into the new POS with modifiers pos-menu-import	Reformatting	9	9
2026 prepaid expense amortization schedule prepaid-amortization	Bookkeeping	7	7
Draft the standing order price increase notice price-increase-notice	Drafting	5	4
Fall wholesale price list page for shop buyers price-list-page	Tooling	8	8
Apply the supplier's price letter to our master price list price-list-update	Spreadsheet	9	9
First-half purchasing savings by category procurement-savings	Reports	9	9
August yield and scrap by production line production-yield	Reports	9	8
West Gallery budget burn for the capital committee project-budget-burn	Bookkeeping	8	8
Margin by job, with the loss-makers flagged project-margin	Reports	9	1
Kitchen job task board for the crew wall project-status-board	Tooling	8	8
How the August promo codes did promo-code-analysis	Spreadsheet	9	9
Log this year's property tax bills property-tax-bills	Extraction	9	9
Write up the warehouse lighting proposal proposal-from-notes	Drafting	9	9
Key this week's customer POs into order entry purchase-orders-pdf	Extraction	9	9
Q3 wholesale sales by product, with the odd things called out quarterly-sales-report	Reports	8	3
CRM accounts into a QuickBooks Online customer import quickbooks-customer-import	Reformatting	7	7
Compare three laptop quotes (USD/EUR, hidden fees) and recommend one quote-comparison	Extraction	8	8
Expense table from scanned trip receipts receipts-scanned	Extraction	9	9
Reply to a customer about a cracked skillet refund-apology-letter	Drafting	5	2
Reconcile August refunds to the card processor refund-reconciliation	Bookkeeping	9	9
Q2 sales by territory from the monthly exports regional-sales-monthly	Reports	5	5
October rent roll for the bank rent-roll-build	Spreadsheet	8	8
Days out per machine per month for the insurer rental-utilization	Spreadsheet	9	9
This week's reorder list for the bike shop reorder-list	Spreadsheet	9	9
Customer cohort retention table for the first half retention-cohorts	Reports	9	0
Q2 return rate by product and reason returns-analysis	Reports	9	8
Request for quotation for the rolling bench parts rfq-email	Drafting	9	9
Rep by month sales table from the invoice export sales-pivot-by-rep	Spreadsheet	6	6
Sales tax on each September order sales-tax-by-address	Spreadsheet	9	9
Second-quarter sales tax due by local jurisdiction sales-tax-liability	Bookkeeping	9	9
Turn the Q3 leads sheet into a Salesforce lead import salesforce-lead-import	Reformatting	8	8
Schedule change notices for next week's revised rota schedule-change-notices	Drafting	9	5
Fall class schedule as a calendar file for the website schedule-to-ics	Reformatting	9	9
Allocate August overhead to departments shared-cost-allocation	Bookkeeping	9	9
How short the published week is against minimum coverage shift-coverage-gaps	Spreadsheet	9	9
Shipment weights and costs from last week's manifests shipment-manifests	Extraction	9	9
Work out what the carrier should charge per order shipping-rate-lookup	Spreadsheet	9	9
Fall catalog into Shopify's product import shopify-product-import	Reformatting	8	8
Move the shop inventory onto the new SKUs sku-crosswalk-migration	Spreadsheet	9	9
Turn the refunds email thread into a counter procedure sop-from-thread	Drafting	3	2
Load the stock count into Square's item import square-inventory-import	Reformatting	6	6
One staff directory from HR and IT staff-directory-merge	Spreadsheet	9	9
September billable utilization per person staff-utilization	Spreadsheet	9	9
Build the Stripe customer import from the old billing export stripe-customer-import	Reformatting	9	9
Active subscribers and cancellations by month subscription-status-monthly	Spreadsheet	9	9
Key this week's superbills for charge entry superbill-codes	Extraction	9	9
Metric supplier price list into our inch-and-pound catalog supplier-catalog-imperial	Spreadsheet	9	9
Dispute letter for a supplier invoice that does not match the PO supplier-dispute-letter	Drafting	9	9
Key August supplier bills into the AP import file supplier-invoices-to-csv	Extraction	7	7
Member survey scores per question survey-tally	Spreadsheet	9	9
Find double-booked tables in the reservations table-double-bookings	Spreadsheet	9	0
Staff directory page for the backstage screen team-directory-page	Tooling	1	1
August resident balances with late fees tenant-ledger-balances	Bookkeeping	7	7
Third-quarter ledger from the landlord statements tenant-statements	Extraction	4	4
Consolidate the crew timesheets for payroll timesheet-consolidate	Spreadsheet	9	9
Daily hours from the scanned caregiver time sheets timesheets-scanned	Extraction	9	9
Split last week's tip pool by shift tip-pooling	Bookkeeping	9	9
Spring tuition collected and outstanding by program tuition-collections	Reports	5	3
Monthly usage and charges from the utility bills utility-bills-extraction	Extraction	9	9
Flatten the nested fall catalog into the inventory platform's variant import variants-flatten	Reformatting	9	9
2025 reportable totals per vendor, cards and corporations out vendor-1099-totals	Bookkeeping	1	1
Merge the purchasing and accounting vendor lists vendor-contact-merge	Spreadsheet	9	9
Load the fall supplier price sheets for the ordering guide vendor-price-sheets	Extraction	3	3
Year-to-date spend by vendor and category vendor-spend-categories	Spreadsheet	7	7
Tie out the Keystone supply statement to our books vendor-statement-reconcile	Bookkeeping	9	9
Log new vendors' tax forms with masked TINs vendor-tax-forms	Extraction	9	9
Key the September venue invoices into job costing venue-invoices	Extraction	9	9
Volunteer hours per program for the county volunteer-hours	Reports	9	9
Equipment warranties running out in the next 90 days warranty-expiry-list	Spreadsheet	9	9
Spring website traffic by month, crawler taken out web-traffic-summary	Reports	9	9
One-page weekly KPIs from five system exports weekly-kpi-dashboard	Reports	2	2
Reshape the by-month sales report for the planning tool upload wide-to-long-sales	Reformatting	9	9
Turn August web store orders into journal entries woocommerce-orders-to-journal	Reformatting	9	9
August job invoices into Xero's sales invoice import xero-sales-invoices-import	Reformatting	9	9
September checking transactions into YNAB ynab-transactions-import	Reformatting	9	9
Import client contacts as Zendesk users zendesk-users-import	Reformatting	9	9

Provenance

Campaign	`complete-desk-comparison-2026-09-28`, three repetitions, executed 2026-09-28T05:29:42Z to 2026-09-29T10:48:25Z
Ledger SHA-256	76b6ecc00c541d0100f945899d9d768ff3c31a9b98eedb546f9d83ba1d0a7ebd
Scorer manifest SHA-256	b4720db00f2a55461ca70767a4baf8dd2d25aa9d301e8fe22ea13d21ccca358f
Configuration: Proto / GPT-6 sol	system: Proto CLI, headless one-shot run (-y), Uninterrupted modebuild: Proto runtime bundle built from Proto commit 3bbb33f8e plus unreleased changes: the open prompt-cache change erphq/proto#601 and working-tree changes later committed as 9cdfcb545..1103377dc. Proto's source is private; bundle_sha256 identifies the buildbundle_sha256: 046142d9363378c3384a4980a07dca71246a903ffe12ae2468c59ac34b4294afmodel: gpt-6-solprovider: ChatGPT subscription through Codex OAuth, the same account as codex-sol6reasoning: highsampling: provider defaultadapter: harnesses/proto-sol6-sub.sh (calls harnesses/proto-sub.sh)home: llmProvider codex, selectedModel openai-codex/gpt-6-sol; no ERP.AI sign-inconcurrency: 4 for every repetition
Configuration: Codex / GPT-6 sol	system: Codex CLI execcli_version: 0.158.0-alpha.2.1model: gpt-6-solprovider: ChatGPT subscription, the same account as proto-sol6-subreasoning: highsampling: CLI-managedadapter: harnesses/codex-sol6.shhome: minimal CODEX_HOME: sign-in plus a config with apps and plugins offconcurrency: 4 for every repetition
Configuration: Proto / DeepSeek V4.1 Flash	system: Proto CLI, headless one-shot run (-y)build: the same Proto bundle as proto-sol6-subbundle_sha256: 046142d9363378c3384a4980a07dca71246a903ffe12ae2468c59ac34b4294afmodel: DeepSeek V4.1 Flash (API model deepseek-flash)provider: DeepSeek API, directreasoning: hightemperature: 0.7max_tokens: 65536adapter: harnesses/proto-deepseek-direct.sh (its header comment predates this campaign; the build is identified by bundle_sha256)home: custom provider at api.deepseek.com; the home holds an ERP.AI sign-in bootstrap, but no ERP.AI actions were offered in its requestsconcurrency: 8 for every repetition
Conditions by repetition	Repetition 1 ran on 28 September 05:29-07:33 UTC with all three systems side by side. Repetitions 2 and 3 of Codex and of Proto on DeepSeek ran side by side from 28 September 20:57 to 29 September 00:45 UTC. Repetitions 2 and 3 of Proto on gpt-6-sol ran alone on 29 September. All used the same images, adapters and homes.
Evidence verification	Every attempt: the receipt names this run, its original_result_sha256 equals the hash of the original result.json, and it was scored by the frozen conservative-v7 package.
Excluded	Proto + gpt-6-sol attempts that made no model request because the provider rejected the sign-in (68) are not in the ledger; each affected task was rerun to completion. Also excluded: development runs on task subsets, smoke runs, pilots, raw traces and credentials.

Verify the published data with python bench/export_desk_campaign.py --verify. It checks the complete task × system × repetition matrix, the summary arithmetic, the paired bootstrap, the ledger hash and every file in the frozen scorer package against the attempt ledger, summary and provenance. It does not regrade the original artifacts, which remain private; their hashes are in the ledger.