Skip to content
Business BenchA benchmark for AI agents on business work

Reproduction and evidence·repository guide

Reproduce

Start by checking the released accounting without calling a model. Running a new agent and regrading a historical output are separate experiments, with different requirements and evidence.

Choose the evidence you need

GoalNeededWhat success establishes
Verify the public releaseRepository and Python dependencies; no model access.Published ledgers, summaries, and declared fingerprints agree.
Run a new configured systemAuthorized model access, an adapter, declared tools/settings, and an evaluation host.A new observation under the configuration and conditions you recorded.
Regrade an outputThat output workspace, the chosen scorer, and native recalculation where required.The chosen evaluator's verdict on those bytes.
Replay the exact historical campaignHistorical outputs and a complete original environment.This is not supported by the public clone alone; those materials are not fully distributed.

1. Verify public accounting without a model

Use Python 3.11 or 3.12. These commands install dependencies and inspect the public data; neither verifier calls an agent or regenerates task inputs.

git clone https://github.com/erphq/business-bench.git
cd business-bench
python3.12 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python bench/export_campaign.py --verify
python bench/export_desk_campaign.py --verify
python bench/export_process_campaign.py --verify

The desk verifier recomputes the summary from the complete ledger and checks the recorded ledger and frozen-scorer fingerprints. The later desk campaigns under results/desk are checked the same way, including each declared paired bootstrap. The process verifier checks each published campaign's declared matrix, ledger hash, and recomputed summary. Neither command regrades the private original artifacts or establishes that the tests measure every business requirement.

Validate the task packages and runner behavior

The full instructions include unit tests, build-task validation, and strict desk validation with native workbook recalculation. These use reference solutions and deterministic test adapters, without spending model credits. Their positive and negative controls check specified behavior; they are not a human acceptance study. Consult the release validation notice for fixture differences and exceptions.

Keep the supplied release inputs intact. A deterministic generator rerun can differ from the published bytes; version regenerated inputs as a new experimental condition.

2. Run a new desk experiment

Follow the Codex setup and run commands or the optional Proto adapter instructions. These runs use your authorized model access and may incur charges. Linux and Docker are the documented route for containerized desk evaluation; local-process mode does not isolate benchmark answers from the agent.

  1. Declare the repository/task versions, model and provider route, harness version, image digest, skills, sampling controls, budgets, task list, and repetitions before launch.
  2. Use a dedicated benchmark home, perform a one-task smoke run, then use a fresh label for the full campaign. The runner rejects existing labels and incomplete result matrices.
  3. Keep execution status, original grades, output artifacts, usage, and any later frozen-scorer grades separately. Record unavailable fields as unknown.

The published desk comparison uses three repetitions of each supplied fixture. bench/run.py defaults to one unless --runs is explicit. Repetition labels do not select new input seeds. New CLI versions, models, skill inventories, or fixtures define a new configuration; a successful current run does not reconstruct missing historical settings.

3. Score an output workspace

With an output workspace you already hold, configure native recalculation as in the installation instructions and call the frozen scorer directly:

python scoring/frozen-v7/scorer.py TASK_ID WORKSPACE

The command prints JSON. Inspect its passed, check results, and errors: the CLI does not use a failing verdict as its process exit code. Preserve the scorer fingerprint and workspace identity alongside the result. The standard runner's original grade and the frozen verdict are distinct fields.

This can reproduce a verdict only when the relevant output bytes and grading dependencies are available. Published hashes identify retained artifacts; they cannot reconstruct them. Regrading the same bytes measures evaluator sensitivity, not improvement in the agent.

4. Validate and run process experiments

Start with the process validator, which uses the reference policy, an inactive agent, and declared negative controls. For example:

python bench/validate_process.py --task procure-to-pay-week --seeds 0,1 --strict

It checks the specified controls, repeatable final state under each seed, and cited handbook clauses. The default seed is 0; more seeds must be explicit. These checks establish properties of the implemented contract, not independent practitioner agreement.

The process-run instructions distinguish the no-model oracle run from a paid agent run. The current runner starts local processes and has no container isolation mode. Use a separate checkout for validation or scenario regeneration while a campaign is active, because scenario caches are shared within a checkout.

The published pilot covers 6 tasks at seed 0, each repeated 5 times per system. Those repetitions are the same scenario, not 5 seed draws. Running --task all uses the currently implemented inventory, which can exceed the historical pilot. Declare an exact task list and seed for a comparable new campaign, and report changed conditions.

Exporting results is separate from execution. Follow the campaign declaration and publication checklist; do not substitute a new run for the released snapshot.

5. What the historical record cannot reproduce alone

The public release distributes ledger records, task packs, the frozen scorer, and provenance. It does not fully distribute the raw historical workspaces, traces, authenticated skill environment, private runtime binaries, or per-attempt container identities. The export process checked original result and receipt hashes against private source records; a public clone can verify the exported accounting, but cannot repeat that source comparison without access to those records.

Current runner defaults and a bare authenticated login are not replacements for a missing historical environment. The evidence audit states the resulting limits; methods defines the quantities a new campaign should report.

Full repository instructions

The following installation, adapter, build, process, and publication commands are rendered directly from README.md. Steps that invoke agents require model access; optional model-assisted auditing also calls a provider unless --dry-run is supplied.

Installation and track-specific commands

Requirements

Use Linux for official containerized runs and application hosting. Python 3.11 or 3.12, Docker Engine, Bash, and Git are required. Node 22 and the Codex CLI are installed in the agent image. Host-side grading uses Python plus the LibreOffice recalculation image. Local-process mode is available for development but is not a reference-answer isolation boundary; do not use it for official model evaluations on a host with benchmark answers or unrelated secrets accessible to the agent.

Paid model runs require your own authorized model access. Codex requires a benchmark-owned login. Proto requires a separately obtained compiled CLI runtime and a benchmark-owned configuration. The benchmark contains the adapters, not third-party or private agent implementations. Models and CLI versions may cease to be available; changed configurations must be reported as a new cell, not claimed to reproduce the historical campaign.

Install and verify without calling a model

git clone https://github.com/erphq/business-bench.git
cd business-bench
python3.12 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python -m unittest discover -s tests -v
python bench/export_campaign.py --verify
python bench/export_desk_campaign.py --verify
python bench/export_process_campaign.py --verify
python bench/validate_build.py
bash docker/build.sh
export BENCH_RECALC_DOCKER_IMAGE=bench-recalc:release
python bench/validate_tasks.py --strict

The unit tests include a real subprocess runner smoke test using a deterministic test adapter; they do not spend model credits. Full task validation checks generators and reference/empty outcomes. It can take several minutes with native workbook recalculation. Do not regenerate canonical inputs in place to make a validation failure disappear: investigate the failure and version any task changes explicitly.

Run the desk track with Codex

Use a dedicated account/home with only the authorized tools and skills for this benchmark. Do not copy a personal home wholesale. Authenticate with the CLI's normal login flow in homes/codex-sol before running. On a host where Codex is installed:

mkdir -p homes/codex-sol
printf '[features]\napps = false\nplugins = false\n' > homes/codex-sol/config.toml
CODEX_HOME="$PWD/homes/codex-sol" codex login
export BENCH_RECALC_DOCKER_IMAGE=bench-recalc:release
python bench/run.py --docker business-bench:release \
  --harness codex-sol --tasks address-standardize \
  --runs 1 --parallel 1 --label codex-smoke
python bench/run.py --docker business-bench:release \
  --harness codex-sol --tasks all \
  --runs 3 --parallel 2 --label codex-full
python bench/report.py results/codex-full

Every invocation needs a fresh label. Existing desk labels and attempt directories are refused instead of overwritten. A task failure is a valid recorded outcome; missing attempt records cause the runner to return an error. The adapter defaults to gpt-5.6-sol with high reasoning effort, but CODEX_MODEL can override it; record the effective configuration. Plugin/skill manifests are operator-supplied: this release does not assert that a bare login reproduces the historical campaign's full skill environment. Desk and build Codex adapters reuse the configured home; unlike process turns, the runner does not create a fresh Codex home per attempt. See the adapter contract before interpreting repetition isolation.

Run the benchmark with Proto

Proto is ERP.AI's agent; the benchmark runs its headless runtime through the adapters in harnesses/. These steps reproduce the proto-sol6-sub system (gpt-6-sol through your own ChatGPT subscription).

  1. Runtime. Obtain a Proto runtime build from ERP.AI: a folder containing index.mjs. Each published campaign records the SHA-256 of the build it used (bundle_sha256 in its provenance).

  2. Images (Docker): PROTO_RUNTIME=/path/to/runtime bash docker/build.sh, then export BENCH_RECALC_DOCKER_IMAGE=bench-recalc:release.

  3. Home. Proto reads its model settings from a benchmark-owned home, never your personal one:

    mkdir -p homes/proto-sol6-sub/.proto
echo '{"llmProvider": "codex", "selectedModel": "openai-codex/gpt-6-sol"}' > homes/proto-sol6-sub/.proto/config.json
CODEX_HOME=$PWD/homes/codex-sol codex login    # signs in once with your ChatGPT account

  4. Run one task, then all of them, with fresh labels:

    python bench/run.py --docker business-bench:release-proto --harness proto-sol6-sub --tasks payments-match-v2 --label my-smoke
python bench/run.py --docker business-bench:release-proto --harness proto-sol6-sub --tasks all --runs 3 --parallel 4 --label my-proto

    For Codex on the same model and account, use --harness codex-sol6.

  5. Score. python scoring/frozen-v7/scorer.py TASK_ID WORKSPACE gives the frozen verdict for an output workspace; the runner's own grade is recorded separately.

Runs use your own model access and may be billed. Do not commit homes/. A different runtime, model or home defines a new cell. Models through OpenRouter are set up as below.

Optional Proto adapters

Supply PROTO_RUNTIME as an absolute directory containing index.mjs and all runtime assets needed by that build. The optional image builder copies it into a temporary build context and produces business-bench:release-proto; it never downloads or vendors Proto source into this repository. The temporary context path is printed for operator-managed disposal.

# Set PROTO_RUNTIME to your separately supplied runtime directory first.
bash docker/build.sh
# Supply OPENROUTER_API_KEY through your secret manager/environment first.
BENCH_HOME_NAME=proto-glm BENCH_GLM_MODEL=z-ai/glm-5.3-flash \
  python bench/setup_home.py
BENCH_HOME_NAME=proto-deepseek BENCH_GLM_MODEL=deepseek/deepseek-v4.1-flash \
  python bench/setup_home.py
BENCH_HOME_NAME=proto-qwen BENCH_GLM_MODEL=qwen/qwen3.8-flash \
  python bench/setup_home.py
export BENCH_RECALC_DOCKER_IMAGE=bench-recalc:release
python bench/run.py --docker business-bench:release-proto \
  --harness proto-glm --tasks all --runs 3 --parallel 5 --label glm-full

Use proto-deepseek and proto-qwen with distinct labels for the other API cells. Subscription-route and platform adapters are also supplied, but require their corresponding authenticated homes; setup_home.py configures OpenRouter only, not subscription login or ERP application access. Never commit the resulting homes/ directories.

Run the build track

Use a dedicated Linux evaluation host and non-production data. Build runs use host networking; allocate a free port, arrange the intended external reachability, and do not expose an unreviewed generated app on a production host. Platform-backed runs need a dedicated test organization and appropriately scoped access.

python bench/build_run.py --task hvac-field-service \
  --harness codex-sol --docker business-bench:release \
  --label hvac-codex-1 --port 8100 --changes 3 --timeout 3600
python bench/build_grade.py \
  results/hvac-codex-1/hvac-field-service__codex-sol__t0 \
  --serve --docker business-bench:release --probe

Grade each turn, complete its tester sheet, and recheck previous requirements after every change. --serve executes the generated start command inside the specified container; inspect untrusted artifacts and use an isolated host. A responding URL or generated sheet is not a completed acceptance test. Report the initial checklist, new requirements, regressions, external reachability, and costs separately. Tester sheets contain generated app logins and must remain private.

Run the process track

Process tasks run an agent inside bb-erp over several turns on a business clock. process_run.py starts one bb-erp server per attempt, calls the harness adapter once per turn with ERP_URL, ERP_TOKEN and the erp command on PATH, lets the counterparty simulator act between turns, and grades the final database and audit log. It runs bb-erp and the agent as local processes, which is not an isolation boundary; container mode is on the roadmap.

python bench/validate_process.py --seeds 0,1,2,3,4 --strict
python bench/process_run.py --task procure-to-pay-week --harness oracle --label dev-oracle
# Paid run over all 7 currently implemented tasks; this is not the six-task historical pilot.
python bench/process_run.py --task all --harness codex-sol --runs 5 --parallel 2 --label my-campaign

validate_process.py checks, for each requested task and seed, that the oracle passes, an agent that does nothing fails, each negative control fails the checks it targets, two runs end in the same database, and every cited handbook clause exists. Its default is seed 0; the five seeds above are explicit. Real agents use the desk adapters (harnesses/<cell>.sh); set CODEX_BIN or BENCH_PROTO_CLI as the adapter describes. Use a separate checkout for validation or scenario regeneration while a campaign is active, because scenarios are cached in .cache/process/.

Before a paid campaign, declare its exact task list, seed, repetitions and cells. Before publication, create results/<label>/campaign.json with that declaration and the actual conditions, then run python bench/export_process_campaign.py --label <label>. The export checklist and description schema explain the required record. Export is a publication step, not part of running an agent.

Results and reproducibility

results/latest/ holds the desk release, results/desk/ later desk campaigns and results/process/ the published process campaigns; other runs stay ignored. Recompute them with python bench/export_campaign.py --verify, python bench/export_desk_campaign.py --verify and python bench/export_process_campaign.py --verify; these check the complete matrix and the arithmetic, not the original artifacts' correctness. The release ledger keeps per-check verdicts, original result hashes, resource records, and execution status, without private paths or logs.

The release includes the exact frozen scorer and its fingerprinted task definitions. Run python scoring/frozen-v7/scorer.py TASK_ID WORKSPACE with native recalculation configured to score an output workspace. The standard runner's original-grade field and this frozen verdict are distinct; retain both. The verifier checks the full frozen package fingerprint as well as result arithmetic. The original result and receipt hashes were checked against the server records for every exported attempt.

For a new official campaign, freeze the repository commit, task manifest, agent image digest, model/provider route, authorized skill inventory, budgets, dependency versions, and scorer before launch. Retain raw artifacts securely so a later scorer can produce a separately named scoring snapshot. Do not mix rescored subsets into the published aggregate. This completed comparison uses different model/harness configurations and non-contemporaneous cohorts; see provenance for the exact scope.

The optional bench/audit.py can prepare a reviewer bundle with --dry-run; without that option it sends task and run content to the configured model provider. Use only synthetic/authorized data. This model-assisted review is not independent practitioner adjudication and never changes the primary acceptance definition. regrade.py and recost.py can change local run records; use them only on a copy, never on a released snapshot. Desk/build raw attempt directories can retain authenticated homes, and all raw workspaces and traces require review before sharing; public exporters publish allowlisted records rather than those directories.

Rebuild the PDF

python docs/build_latex.py

The paper is compiled with XeLaTeX, using native booktabs tables, PGFPlots figures, numbered equations, linked cross-references, and a BibTeX bibliography. See paper/README.md for the TeX/Pandoc prerequisites and the direct .tex build. The shared manuscript and complete generated TeX sources are tracked. The paper embeds the bundled SIL Open Font License Libertinus Serif fonts; attribution and license are in docs/assets/fonts/. Generated previews and build intermediates remain ignored.

Site

The public site under site/ is generated from this repository: task metadata via python3 site/scripts/export_tasks.py, results from results/latest/, results/desk/ and results/process/, the process-track pages from tasks/process/ and docs/process/, and the paper from SPEC.md. Build with cd site && bun install && bun run build; deploy with bunx wrangler deploy (Cloudflare account access required). The Site workflow checks the export is fresh, runs the site tests, and builds on every push.