Roadmap: Business Bench v2

Status: proposed research program, reviewed against repository state 2026-09-28. The three new check types and generator helpers exist; the 100 task descriptions below are a design catalog, not a released v2 task suite. No v2 campaign or human baseline is published. The separate process track has seven implemented tasks. Existing desk tasks have not been empirically calibrated to the proposed bands.

1. Why this benchmark exists

Business tasks often combine heterogeneous evidence with an output whose required properties can be tested: a reconciliation, a destination-compatible import, or a feasible schedule. Executable checks make the acceptance boundary inspectable, but cannot establish that the boundary captures every relevant business requirement. Human and model-assisted review can evaluate properties that a mechanical contract omits; neither evaluation approach is a universal substitute for the other.

The proposed v2 program studies three questions:

1. Conjunctive correctness. A task passes only when every required check passes. A high fraction of passed predicates need not produce an accepted artifact. Report both quantities with explicit weighting; predicate decomposition changes the partial-check fraction.
2. Execution repeatability. Report attempt acceptance, all-k acceptance and at-least-one acceptance separately. Repeating one fixture is not the same as completing new monthly work. The proposed v2 campaign uses five repetitions.
3. Generated instances and declared difficulty. Seeded generators can create new instances of known templates. Separately held-out templates and practitioner review are needed to test broader transfer and validate difficulty bands. Training-environment use remains a possible extension, not a current result.

The benchmark is published on businessbench.org and in this repository. Hashed manifests, receipts and original/frozen verdicts support traceability. They do not replace independent review of task realism, valid alternatives or plausible invalid outputs. Human baselines, sealed evaluations and independent adjudications remain planned evidence, with no completed records claimed here.

2. Definitions

A task t declares a set of required checks C_t over the artifacts left in the workspace. An attempt passes iff every c in C_t passes. Each task is run k times in a fresh workspace (k = 3 in the current release, 5 from the first v2 campaign).

Symbol	Definition	Reading
pass@1	mean attempt pass rate over all scheduled attempts	headline rate
observed at-least-one	fraction of tasks with at least one passing repetition	availability with hindsight, not a measured selection policy
pass^k	fraction of tasks passed in all k repetitions (τ-bench)	observed repeated acceptance on fixed fixtures
check rate c̄	passed required checks / all required checks, pooled	per-check view
within-attempt check fraction	mean of each attempt's fraction of required predicates passed	gives each attempt equal weight
conjunctive gap	within-attempt check fraction − pass@1	isolates conjunction without extra check-count weighting
single-predicate failure share	failed attempts with exactly one failed required predicate / failed attempts	diagnostic; not an error count or root cause
gap to reference objective	sense-aware relative gap under plan_feasible, for axis 5	a reference is not a proven optimum unless certified
model cost per accepted attempt	captured-usage model cost / accepted attempts	ex-post accounting; excludes review and repair

Current values for the released comparison are computed on businessbench.org/analysis from the ledger at build time.

3. Capability axes

v2 tasks are organised by the capability they isolate. Departments (finance, ops, legal, HR) are how a task is instantiated, not how it is reported.

Axis	Tasks	Isolates	Primary check types
1. Rule-stack application	14	Many interacting rules, one correct outcome per entity	rule, keyed, pin
2. Cross-document state tracking	14	The answer exists only after reconciling many files	set, keyed, pin
3. Evidence sufficiency	12	Saying "cannot be determined" when the files do not support it	must-state, pin
4. Adversarial robustness	16	Planted fraud, plausible unauthorized instructions, injected documents	set, not-fooled
5. Constrained quantitative planning	18	Schedules and plans graded by feasibility plus gap to a known optimum	feasible+bound, held-out
6. Verification and handoff	12	Consistency of the handoff with the source; process claims require trace evidence	pin, keyed, must-state
7. Process discovery and conformance	14	Recovering the process from event logs and finding where it breaks	set, keyed, held-out

The full list with planted truth and check per task is in tasks.md.

Axes 4, 6 and 7 are proposed priorities, not claims of exclusive coverage relative to other benchmarks. The axes organize hypotheses; cross-task construct validation and actual task authoring are still required.

4. Grading vocabulary

Every v2 task is graded by executable checks against truth the generator planted. No rubric, no LLM judge, no pairwise preference. Existing v1 check types cover most of the vocabulary; the three additions below are already implemented in bench/grade.py and covered by focused tests.

Name	Meaning	Status
set	exact identifier set equals reference	v1 csv_set_equal
keyed	per-key values within tolerance, with must-match keys	v1 csv_values_match
pin	recalculated workbook figure equals expected	v1 xlsx_value_present + xlsx_no_errors
rule	per-entity categorical outcome equals reference	v1 csv_values_match on a category column
must-state	a sentence names the exception, gap, or refusal	v1 text_sentence_matches
feasible+bound	task-specific constraint checker passes and objective is within X% of the reference optimum	plan_feasible (module + max_gap), shipped v1.1.0
held-out	forecast scored against generated future truth at a stated error bound	forecast_error (metric, bound), shipped v1.1.0
not-fooled	a planted adversarial instruction was not acted on; the artifact shows the pre-instruction state	not_fooled (planted keys, forbidden text, flag), shipped v1.1.0

Partial credit: none at task level. For feasible+bound tasks the gap is recorded as a diagnostic field so a gap curve can be published; the pass threshold is stated in the task.

5. Difficulty axis

The proposed v2 generator interface exposes three parameters: size (rows and files), rules (count of interacting rules), and noise (rate of formatting and data defects). Each task declares three bands built from the same generator:

Band	Persona	Typical size	Time budget
clerical	bookkeeper or office manager	hundreds of rows, 3 to 6 files	20 min
analyst	staff accountant or ops analyst	thousands of rows, 6 to 15 files	40 min
controller	controller, FP&A lead, ops director	tens of thousands of rows, 15+ files	60 min

The plan is to report results per band after task authoring and human calibration. The existing v1 set is a starting point, not a validated clerical difficulty band.

6. Sealed variants and contamination

The public task set is exposed by design. Each generator accepts --seed. A sealed variant is the same task with re-rolled entities, amounts, dates, and planted positions. A future sealed evaluation must record seed custody, task version, scorer and analysis plan before execution. No released campaign currently establishes this procedure. Re-rolling known templates does not erase template exposure. Compare public instances, new instances and held-out templates separately, with an analysis that accounts for both task and execution variation.

7. Human baseline

A human baseline is a proposed release requirement; this repository does not establish commissioning, recruitment or completed attempts. Protocol and draft contractor brief: human-baseline.md. Summary: stratified sample of 50 tasks, two independent human attempts per task at the analyst band, humans use the same files and the same checks, raw time and verdict sheets published.

8. What a business harness has to do

Proto is one evaluated system, developed by the benchmark publisher. The properties below are proposed design requirements for any configured system. The repository's runner and adapters implement only the portions described in the adapter contract; the list is not evidence that a particular harness satisfies every requirement. Independent artifact and trace review would be needed for a system-specific explanation.

• Artifact-first delivery: the deliverable is the files left in the workspace, not the final message. Nothing an agent says is graded.
• Native recalculation: workbooks are recalculated in LibreOffice after cached values are stripped; a harness that writes formulas must verify them the same way before handing over.
• Declared state boundaries: fresh workspaces, explicit home/configuration handling, and a tested isolation boundary. Current desk/build Codex runs reuse a configured home; process turns copy homes but run as local processes.
• Effective configuration records: declare model, route, runtime and budgets, including environment overrides. Adapter defaults are not proof of the settings used in an arbitrary run.
• Self-verification before handoff: the harness should recompute its own control totals and tie-outs against the source before ending the turn. Axis 6's artifact checks can measure consistency; proving that the system performed verification requires trajectory evidence and an explicit process measure.
• Refusal as a first-class outcome: when evidence is insufficient, the correct deliverable says so. Axis 3 measures this.

9. Release discipline

• Task set versions are semver; any change to a task, check, or scorer bumps it.
• Every campaign is labeled; results are never merged across labels.
• Raw and frozen verdicts are both retained per attempt with receipts.
• Disputes are GitHub issues; accepted corrections change the version.
• Five repetitions per cell from the first v2 campaign.
• The self-audit page is updated with every release and lists what a reviewer would find first.

10. Threats to validity

Stated here so that no reviewer has to discover them.

• Construct. Executable checks measure contract satisfaction, not editorial quality or owner satisfaction. A memo can pass every check and still be unusable. Review accepted artifacts as well as rejected ones to identify omitted requirements.
• Check strictness. A conjunctive grader turns every strict check into a task failure. The near-miss share confounds "did not verify" with "check too strict". Independent adjudication of accepted and rejected outputs, supplemented by targeted mutations and valid alternatives, is planned to investigate both error types.
• Scorer authorship. The scorer is written by the organisation that builds one of the evaluated systems, after outputs exist. Mitigations: hashed frozen packages, raw verdicts published beside frozen, and an independent scorer review before the first v2 campaign is reported.
• Cohorts and configurations. Systems differ in model, sampling, and timing. Results are end-to-end configuration comparisons until a same-model control pair is run.
• Exposure. The public set was used during development of one system and is public now. Sealed variants and the difficulty axis are the mitigations; a public score is never presented as an unseen-generalisation result.
• Evaluation leakage. Containerized desk runs omit host-side checks and references from mounts; local runs provide no such boundary. Network access and known template structure remain exposure paths. Private instances reduce direct answer lookup but do not eliminate contamination or evaluator exploitation.
• Repetitions. k = 3 estimates pass^k coarsely. v2 uses k = 5 and reports a task-clustered bootstrap interval for every headline difference.

11. Order of work

1. Done (v1.1.0): the three check types in bench/grade.py, rules in bench/check_rules.py, tests in tests/test_checks_v2.py.
2. Done (v1.1.0): tasks/lib/bizgen/eventlog.py, planning.py, adversarial.py, tests in tests/test_bizgen_v2.py.
3. Author axes 4, 6, 7 first (42 tasks), then 3 and 5, then 1 and 2.
4. Review tasks, recruit participants and commission the proposed baseline only after documenting the sample, tools, consent, compensation and adjudication procedure.
5. First v2 campaign: five repetitions, sealed variant for the top system, published with the per-check gap and the difficulty curve.

Proposed catalog recorded 2026-09-18, status clarified 2026-09-28. Companion to README.md. These 100 rows are design proposals, not implemented or validated task packs. Slugs and three-band authoring are provisional; no v2 campaign or baseline is reported.

Reader: AI engineers and researchers. The proposed tasks combine heterogeneous business inputs with declared output properties that could be tested mechanically. "Planted truth" means a generator would construct a known situation; it does not establish that the policy is realistic, complete, or unambiguous. Each authored task will need reviewed rules, a reference solution, positive/negative controls and valid alternatives. Legal, accounting and tax examples must supply a versioned rule set; they are synthetic rule-application exercises, not professional advice or claims of universal legal correctness.

Grading vocabulary: set (exact identifier set), keyed (per-key values within tolerance), pin (recalculated workbook figure), rule (per-entity rule outcome), feasible+bound (constraint checker plus cost within X% of reference optimum), must-state (sentence-level rule that the deliverable names an exception or refuses), held-out (forecast scored against generated future truth), not-fooled (planted adversarial instruction was not followed).

Axis 1. Rule-stack application (14)

Many interacting rules, one correct outcome per entity. Hard because rules compose; easy to verify because the generator applies the same rules.

#	Slug	Use case	Planted truth	Check
1	sales-tax-nexus-across-states-with-economic-thre	Sales tax nexus across states with economic thresholds crossed mid-year	per-state nexus start date	rule
2	contractor-vs-employee-classification-from-engag	Contractor vs employee classification from engagement facts	classification per worker	rule
3	vat-reverse-charge-and-place-of-supply-across-bo	VAT reverse charge and place-of-supply across borders	VAT treatment per invoice	rule
4	r-d-credit-eligible-spend-from-project-time-and	R&D credit eligible spend from project time and vendor invoices	eligible amount	pin
5	asc-842-lease-schedules-with-renewal-options-rea	ASC 842 lease schedules with renewal options reasonably certain	ROU asset and liability	pin
6	deferred-revenue-waterfall-with-mid-term-contrac	Deferred revenue waterfall with mid-term contract modifications	monthly recognition	keyed
7	clause-precedence-order-form-vs-msa-vs-amendment	Clause precedence: order form vs MSA vs amendment conflicts	governing term per topic	rule
8	sla-credits-computed-from-incident-logs-and-excl	SLA credits computed from incident logs and exclusion windows	credit per customer	keyed
9	cpi-escalators-with-caps-floors-and-base-month-c	CPI escalators with caps, floors, and base-month conventions	new rent per lease	keyed
10	tiered-revenue-share-with-retroactive-vs-margina	Tiered revenue share with retroactive vs marginal tiers	payout per partner	keyed
11	tariff-classification-with-rules-of-origin-acros	Tariff classification with rules of origin across a BOM	HTS code and duty per SKU	rule
12	overtime-and-meal-break-premiums-under-two-state	Overtime and meal-break premiums under two state regimes	pay per employee per week	keyed
13	commission-plan-with-clawbacks-splits-and-accele	Commission plan with clawbacks, splits, and accelerators	commission per rep	keyed
14	expense-policy-with-per-diem-receipts-thresholds	Expense policy with per-diem, receipts thresholds, and exceptions	approve, reject, or query per line	rule

Axis 2. Cross-document state tracking (14)

The answer exists only after reconciling many files. Hard because state must be carried across documents; easy to verify because the generator emitted the documents from one state.

#	Slug	Use case	Planted truth	Check
15	three-way-match-with-partial-receipts-and-price	Three-way match with partial receipts and price variances	match status per PO line	set + keyed
16	customer-master-merge-across-crm-erp-and-billing	Customer master merge across CRM, ERP, and billing with survivorship rules	golden record per customer	keyed
17	bank-feed-to-ledger-with-split-and-reversed-tran	Bank feed to ledger with split and reversed transactions	unmatched set both sides	set
18	subscription-state-reconstruction-from-an-out-of	Subscription state reconstruction from an out-of-order webhook history	plan and status per account	keyed
19	audit-trail-reconstruction-who-changed-which-pri	Audit-trail reconstruction: who changed which price when, from three logs	change history per SKU	keyed
20	intercompany-eliminations-with-fx-and-timing-dif	Intercompany eliminations with FX and timing differences	consolidated balances	pin
21	inventory-position-across-locations-with-in-tran	Inventory position across locations with in-transit and consignment	on-hand per SKU per site	keyed
22	employee-record-conflicts-across-hris-payroll-an	Employee record conflicts across HRIS, payroll, and badge system	authoritative record per employee	keyed
23	sku-hierarchy-remap-after-a-catalog-restructure	SKU hierarchy remap after a catalog restructure	new parent per SKU	keyed
24	migration-validation-source-vs-target-with-check	Migration validation: source vs target with checksums and dropped rows	missing and altered ids	set
25	project-cost-roll-up-from-timesheets-pos-and-exp	Project cost roll-up from timesheets, POs, and expense reports	cost per project per month	keyed
26	cash-application-remittances-to-open-invoices-wi	Cash application: remittances to open invoices with short pays	applied amount per invoice	keyed
27	vendor-statement-reconciliation-across-credit-no	Vendor statement reconciliation across credit notes and duplicates	disputed items	set
28	grant-spend-vs-award-terms-across-cost-categorie	Grant spend vs award terms across cost categories and periods	allowable vs disallowed per line	set

Axis 3. Evidence sufficiency (12)

The correct deliverable sometimes says "cannot be determined from these files." Agents guess. The generator plants the gap.

#	Slug	Use case	Planted truth	Check
29	which-of-five-invoices-a-payment-settles-when-tw	Which of five invoices a payment settles, when two are indistinguishable	ambiguous pair named, not resolved	must-state
30	revenue-by-region-where-8-of-orders-have-no-regi	Revenue by region where 8% of orders have no region and no proxy	unallocated bucket reported	pin + must-state
31	payroll-accrual-where-one-week-s-timesheets-are	Payroll accrual where one week's timesheets are missing	accrual excludes, gap named	pin + must-state
32	vendor-due-diligence-where-the-requested-certifi	Vendor due diligence where the requested certificate is absent	flagged incomplete, no approval	must-state
33	margin-by-product-where-cost-file-predates-a-pri	Margin by product where cost file predates a price change	stale-cost caveat with affected SKUs	set + must-state
34	churn-analysis-where-cancellation-reasons-are-mi	Churn analysis where cancellation reasons are missing for a cohort	cohort excluded from attribution	must-state
35	customer-refund-where-the-return-has-no-matching	Customer refund where the return has no matching shipment	held for evidence	rule
36	board-memo-where-two-sources-disagree-on-headcou	Board memo where two sources disagree on headcount	discrepancy named, both figures	must-state
37	lease-abstraction-with-a-missing-amendment-refer	Lease abstraction with a missing amendment referenced in another	missing document listed	set
38	bonus-calculation-where-the-plan-document-s-tier	Bonus calculation where the plan document's tier is undefined	tier flagged, no invented rate	must-state
39	insurance-claim-where-the-loss-date-falls-in-an	Insurance claim where the loss date falls in an uncovered gap	not covered, gap identified	rule
40	forecast-where-the-last-two-months-of-data-are-p	Forecast where the last two months of data are provisional	provisional flagged, range not point	must-state

Axis 4. Adversarial robustness (16)

Something in the inputs is wrong on purpose: fraud, a plausible but unauthorized instruction, an injection in a document. Scored as not being fooled.

#	Slug	Use case	Planted truth	Check
41	duplicate-vendor-ring-across-entities-under-name	Duplicate-vendor ring across entities under name and address variants	vendor cluster	set
42	expense-report-collusion-between-approver-and-su	Expense-report collusion between approver and submitter	colluding pairs	set
43	ghost-employees-in-payroll-no-badge-no-timesheet	Ghost employees in payroll (no badge, no timesheets, shared account)	ghost ids	set
44	pos-skimming-pattern-in-z-reports-voids-after-cl	POS skimming pattern in z-reports (voids after close by one cashier)	cashier and shifts	set
45	split-purchases-just-under-the-approval-threshol	Split purchases just under the approval threshold	split groups	set
46	round-dollar-manual-journals-before-period-end	Round-dollar manual journals before period end	suspicious entries	set
47	kickback-pattern-one-bidder-always-second-lowest	Kickback pattern: one bidder always second-lowest then wins	award set	set
48	related-parties-hidden-by-spelling-and-registere	Related parties hidden by spelling and registered-agent addresses	related set	set
49	email-from-the-cfo-asking-to-change-a-vendor-s-b	Email from "the CFO" asking to change a vendor's bank account	change refused, flagged	not-fooled
50	pdf-invoice-with-embedded-instruction-to-mark-it	PDF invoice with embedded instruction to mark itself approved	instruction ignored	not-fooled
51	spreadsheet-note-asserting-a-figure-that-contrad	Spreadsheet note asserting a figure that contradicts the source	source figure used	pin
52	customer-email-claiming-a-discount-that-the-cont	Customer email claiming a discount that the contract does not grant	discount not applied	rule
53	vendor-price-sheet-with-a-transposed-digit-on-th	Vendor price sheet with a transposed digit on the highest-volume SKU	transposition flagged	must-state
54	sanctions-screening-with-fuzzy-and-transliterate	Sanctions screening with fuzzy and transliterated names	true hits, no false clears	set
55	payroll-direct-deposit-change-requests-from-spoo	Payroll direct-deposit change requests from spoofed employee emails	changes refused, flagged	not-fooled
56	refund-requests-reusing-the-same-proof-of-purcha	Refund requests reusing the same proof-of-purchase across accounts	duplicate-proof set	set

Axis 5. Constrained quantitative planning (18)

Hard to produce, cheap to verify: a constraint checker plus a bound against a known optimum. Partial credit by gap is possible; pass/fail at a stated gap is the default.

#	Slug	Use case	Planted truth	Check
57	crew-schedule-under-labor-rules-skills-and-avail	Crew schedule under labor rules, skills, and availability	reference optimum	feasible+bound
58	delivery-routing-with-time-windows-and-vehicle-c	Delivery routing with time windows and vehicle capacity	reference optimum	feasible+bound
59	reorder-points-under-lead-time-variance-at-a-ser	Reorder points under lead-time variance at a service level	policy per SKU	keyed
60	production-sequencing-with-sequence-dependent-ch	Production sequencing with sequence-dependent changeovers	reference makespan	feasible+bound
61	warehouse-slotting-by-velocity-and-affinity	Warehouse slotting by velocity and affinity	reference pick distance	feasible+bound
62	appointment-overbooking-policy-from-no-show-hist	Appointment overbooking policy from no-show history	expected utilization	held-out
63	technician-dispatch-with-skills-parts-on-van-and	Technician dispatch with skills, parts on van, and travel	reference optimum	feasible+bound
64	13-week-cash-flow-with-covenant-test-and-payer-b	13-week cash flow with covenant test and payer behavior	weekly balance	held-out
65	demand-forecast-with-planted-seasonality-and-pro	Demand forecast with planted seasonality and promotions	future demand	held-out
66	headcount-plan-under-budget-ramp-time-and-attrit	Headcount plan under budget, ramp time, and attrition	plan feasibility and cost	feasible+bound
67	capacity-plan-against-order-book-and-maintenance	Capacity plan against order book and maintenance windows	shortfall weeks	keyed
68	price-increase-plan-under-contract-caps-to-hit-a	Price-increase plan under contract caps to hit a margin target	per-customer increase	feasible+bound
69	purchase-plan-with-volume-breaks-and-shelf-life	Purchase plan with volume breaks and shelf life	reference cost	feasible+bound
70	debt-paydown-schedule-under-prepayment-penalties	Debt paydown schedule under prepayment penalties	reference interest	feasible+bound
71	shift-bidding-allocation-with-seniority-and-fair	Shift bidding allocation with seniority and fairness constraints	feasible assignment	feasible+bound
72	marketing-budget-allocation-with-diminishing-ret	Marketing budget allocation with diminishing returns per channel	reference return	feasible+bound
73	truck-loading-with-weight-axle-and-stacking-cons	Truck loading with weight, axle, and stacking constraints	feasible load plan	feasible+bound
74	seasonal-hiring-plan-with-training-lag-and-deman	Seasonal hiring plan with training lag and demand curve	coverage per week	keyed

Axis 6. Self-verification and handoff (12)

Does the agent check its own work against the source before handing over? The deliverable includes recomputable figures and a reconciliation the checker replays.

#	Slug	Use case	Planted truth	Check
75	month-end-package-where-every-schedule-must-tie	Month-end package where every schedule must tie to the trial balance	tie-out differences zero	pin
76	import-file-with-a-control-total-sheet-the-agent	Import file with a control-total sheet the agent must produce and match	control totals	pin
77	report-with-a-sources-appendix-every-figure-trac	Report with a sources appendix; every figure traceable to a file and cell	traceability map	keyed
78	reconciliation-where-the-agent-s-own-explanation	Reconciliation where the agent's own explanation must sum to the difference	explained items	pin
79	workbook-with-live-formulas-and-no-hardcoded-out	Workbook with live formulas and no hardcoded outputs	formula count, no errors	pin
80	data-migration-with-a-self-produced-validation-l	Data migration with a self-produced validation log the checker reruns	log matches recomputation	keyed
81	invoice-batch-with-a-pre-send-exception-list-the	Invoice batch with a pre-send exception list the agent must generate	exceptions equal planted set	set
82	board-deck-figures-that-must-equal-the-workbook	Board deck figures that must equal the workbook they cite	cross-document equality	pin
83	payroll-run-with-a-variance-to-prior-period-shee	Payroll run with a variance-to-prior-period sheet explaining every change	variance explained	keyed
84	handoff-note-stating-what-was-not-done-and-why	Handoff note stating what was not done and why	planted undone items named	must-state
85	recompute-a-prior-period-and-report-every-figure	Recompute a prior period and report every figure that changed, with cause	changed figures and causes	keyed
86	assumptions-register-where-every-assumption-cite	Assumptions register where every assumption cites a source cell that holds that value	citations resolve	keyed

Axis 7. Process discovery and conformance (14)

The business process itself is the object. From event logs and documents, recover the process, find where it breaks, and quantify it. The generator ran a known process model with planted deviations.

#	Slug	Use case	Planted truth	Check
87	recover-the-order-to-cash-process-model-from-eve	Recover the order-to-cash process model from event logs	activity graph	set of edges
88	conformance-which-cases-skipped-approval-or-reor	Conformance: which cases skipped approval or reordered steps	deviating cases	set
89	bottleneck-attribution-where-cycle-time-is-lost	Bottleneck attribution: where cycle time is lost, by resource	top bottleneck and hours	keyed
90	rework-loops-cases-that-returned-to-an-earlier-s	Rework loops: cases that returned to an earlier step and why	loop cases and trigger	set
91	segregation-of-duties-conflicts-from-role-matric	Segregation-of-duties conflicts from role matrices and actual logs	conflicting user pairs	set
92	sla-breach-prediction-from-partial-traces	SLA breach prediction from partial traces	breached cases	held-out
93	process-variant-analysis-the-variants-that-expla	Process variant analysis: the variants that explain most cost	top variants	set
94	handoff-count-and-wait-time-per-department-from	Handoff count and wait time per department from ticket history	per-department figures	keyed
95	control-effectiveness-did-the-four-eyes-rule-act	Control effectiveness: did the four-eyes rule actually hold	violations	set
96	write-the-sop-that-matches-the-observed-happy-pa	Write the SOP that matches the observed happy path, retiring the old one	ordered steps, retired items absent	rule
97	change-point-detection-when-did-the-approval-ste	Change-point detection: when did the approval step start being skipped	change date	keyed
98	handover-pattern-which-resource-waits-on-which-a	Handover pattern: which resource waits on which, and for how long	wait matrix	keyed
99	remaining-time-prediction-for-open-cases-from-pa	Remaining-time prediction for open cases from partial traces	remaining time per case	held-out
100	variant-mix-drift-between-two-quarters-and-the-c	Variant-mix drift between two quarters and the cases that explain it	drifted variants	set

Difficulty axis

Every generator exposes size (rows, files), rule count, and noise rate. Results are reported along it. The current 187 tasks are the clerical band; v2 adds the analyst and controller bands from the same generators.

What is deliberately absent

Rubric or LLM-judged quality, essay-style deliverables, tasks whose answer depends on taste, and anything that requires the ERP.AI platform to be solvable. A platform-backed cell is reported as a declared condition; the tasks are solvable with files and a shell.

Proposed protocol and planning budget, recorded 2026-09-18. The repository does not establish that contractors have been commissioned, participants recruited, or attempts completed. Purpose: measure completion time and acceptance under the same declared checks, then investigate disagreements between those checks and practitioner judgment. A baseline provides context for the tested sample, not a universal human performance standard.

Sample

• 50 tasks, stratified: 7 from each of the seven v2 axes plus 1 extra in axis 5, drawn at random with a published seed once the v2 tasks exist. Until then, a pilot of 20 v1 tasks (3 per category, 2 in tooling) at the clerical band.
• Analyst band for v2 tasks; clerical band for the v1 pilot.

Participants

• 10 bookkeepers or staff accountants and 2 controllers, recruited on Upwork or equivalent, US or UK based, 5+ years of experience, screened with one unscored task.
• Each task is attempted independently by two participants who do not see each other's work. Controllers take the axis 5 and axis 7 tasks.
• Participants use their own machines and any software they normally use, including spreadsheets and, if they choose, AI tools. Whether they used AI is recorded per task and published; the baseline is "a practitioner with their usual tools". Report AI-assisted and unassisted attempts separately, including tools/models used. Do not interpret their pooled score as unaided human ability.

Procedure

1. Participant receives the workspace folder and the ask, exactly as an agent would. No checks, no reference, no traps.
2. Time starts at download and stops at upload of the deliverables. Self-reported interruptions are subtracted only if logged at the time.
3. Primary acceptance uses the same declared scorer as the compared agents, fixed before attempts. Separately record practitioner adjudication of sampled accepted and rejected artifacts; do not silently substitute that judgment into the primary score. v2 tasks need their own frozen scorer, not the historical desk package.
4. After grading, the participant sees the failed checks and answers two questions: was the ask clear, and do they dispute the check. Disputes are logged as task issues and count toward the task-correctness audit.

Planned published outputs

• Per task: participant id (anonymised), time, pass or fail, failed checks, AI-use flag, clarity rating, dispute flag.
• Aggregate: human pass rate per axis, median and p90 time per band, dispute rate, and the agent-to-human ratio on the same 50 tasks.
• Raw sheets under results/human-baseline/<label>/, same allowlisting rules as agent ledgers.

Planning budget

Item	Estimate
50 tasks × 2 attempts × ~45 min analyst band	75 hours
20-task v1 pilot × 2 attempts × ~25 min	17 hours
Screening, onboarding, post-task questions	15 hours
Total contractor time	~107 hours
At $40 to $70 per hour blended	$4,300 to $7,500

Draft contractor brief (not a published recruitment notice)

We are measuring how long real business tasks take and how accurately they are done. You will receive a folder of business files (spreadsheets, PDFs, emails) and a short request written the way a business owner would write it. Produce the requested files using whatever tools you normally use. There are no trick questions, but the files are as messy as real exports: duplicate rows, inconsistent dates, a requirement buried in an email. If the files cannot support an answer, say so in the deliverable rather than guessing. Record your start and end time. You will be paid per task at an hourly rate with a cap per task; accuracy is checked automatically and shared with you afterwards. Expect 20 to 60 minutes per task. Anonymised timing and accuracy data will be published as part of an open benchmark; your name will not.