Self-audit
The public record supports checking score accounting and inspecting the evaluation rules. It does not supply the private historical outputs or independent acceptance judgments needed to validate every verdict. This is the authors' evidence assessment, not an external certification.
Conflict of interest and evaluator chronology. ERP.AI publishes the benchmark and develops Proto. The same organization constructed the primary frozen scorer after outputs existed. On those outputs, the observed lead changes from 16 accepted attempts under the original evaluator to 34 under the frozen evaluator, a shift of 3.21 percentage points. That is measurable evaluator sensitivity; its direction alone does not establish whether the revisions improve validity or introduce bias.
1. Public evidence and what it establishes
| Evidence | What a reader can check | What remains outside it |
|---|---|---|
| Attempt ledger, summary, provenance | Matrix completeness, verdict totals, per-check records, resource accounting, and the recorded cohort definitions. | Whether a historical output deserved its recorded verdict; hashes cannot reconstruct the output. |
| Desk comparison, 28 September 2026: ledger, provenance | The same checks for a separate 3-system campaign, including its declared paired bootstrap. | The same limits: one repetition, private artifacts, and a development-exposed task set. |
| Frozen scorer and manifest | The exact 847-file package, copied task definitions, and equivalence rules for 7 tasks. | Whether the declared tests cover the right business properties or were selected independently of observed outputs. |
| Task packs | 187 desk tasks and 20 build tasks: supplied inputs, requests, checks/checklists, generators, and references. | Representative business demand, independent practitioner agreement, or human completion times. |
| Adapters and recorded configurations | Disclosed model routes, runtime identifiers, sampling settings where available, and current integration behavior. | A complete historical environment: private runtime binaries, image digests, authenticated skills, raw traces, and output workspaces are not fully distributed. |
Run the public accounting checks without calling a model. The desk verifier and the verifier for later desk campaigns check the ledger hash, recomputed summary, scorer fingerprint, and scorer identities. Original-result and receipt identities were checked against private source files during export, according to provenance; a fresh public clone cannot repeat that private-file comparison.
2. Task validity
| Question | Published evidence | Unresolved question |
|---|---|---|
| Can the declared tests be satisfied? | The release notice reports reference-solution passes and untouched-workspace failures. | This establishes test-relative feasibility for those controls, not that an independent reader can infer every requirement from the request. |
| Are supplied inputs reproducible? | Repeated generation was checked under the validation environment. The notice lists 28 tasks with supplied bytes different from regenerated bytes. | Byte differences are not assumed harmless. Supplied release inputs remain authoritative; regenerated instances require their own identity. |
| Do tests agree with professional acceptance? | No independent practitioner agreement study or human completion baseline is published with this release. | Ambiguity, omitted requirements, plausible invalid outputs, and valid alternative representations need external adjudication. |
3. Evaluator sensitivity is observable; error rates are not
| System | Raw fail → pass | Ungraded → pass | Pass → fail | Net accepted |
|---|---|---|---|---|
| Proto / DeepSeek V4.1 Flash | 56 | 4 | 0 | +60 |
| Codex / GPT-5.6 sol | 40 | 3 | 1 | +42 |
The additional 18-attempt difference is an evaluator change, not improvement from rerunning either agent. The frozen package was designed to admit specified alternative representations and enforce artifact checks. Without independent judgments on sampled outputs, the number of corrected false negatives and introduced false positives remains unknown.
Native recalculation, readability checks, sentence-level predicates, and strict-validator controls address specific failure modes. They do not measure the residual false-acceptance or false-rejection rate. The process track has scripted negative controls, but those controls are not evidence that the historical desk scorer was comprehensively tested against plausible wrong answers.
Determinism, identity, and validity are separate
Provenance records a 1,341-attempt scorer replay with zero verdict differences. That is a report about the replayed inputs and environment, not proof of determinism across every runtime or of agreement with business judgment. The immutable manifest preserves the scorer version; its digest is b4720db00f2a55461ca70767a4baf8dd2d25aa9d301e8fe22ea13d21ccca358f.
4. Version and cohort limits
The two current cells share the reported task set and primary frozen scorer. Their models, harnesses, sampling controls, and cohort dates differ. Earlier snapshots and later development experiments are excluded in provenance; they should not be pooled into a single ranking or treated as controlled before-and-after changes. The separate 28 September 2026 desk comparison ran Proto and Codex on the same model, account and host. That narrows the differences for that pair; it uses the same exposed tasks, and its repetitions ran under different loads.
Recorded source cohorts
- Proto: broad-desk-v43-c8f62dd60 r1; repeat-desk-v43-c8f62dd60 local r1/r2 map to logical r2/r3
- Codex: full-1-codex-sol r1/r2/r3
See the configuration table. Repetition labels are not common random seeds; the comparison did not randomize concurrent executions of otherwise identical systems.
5. Execution status does not certify elicitation
| System | Attempts | Known abnormal | Unknown exit status | Accepted with known abnormal exit |
|---|---|---|---|---|
| Proto / DeepSeek V4.1 Flash | 561 | 3 | 0 | 0 |
| Codex / GPT-5.6 sol | 561 | 0 | 0 | 0 |
These records reveal interrupted executions, but low interruption counts do not establish adequate prompting, budgets, tool access, or skill configuration. Budget sweeps and controlled configuration comparisons would be needed to measure sensitivity to those choices.
6. Exposure, uncertainty, and generalization
| Claim boundary | Current evidence | Stronger evidence needed |
|---|---|---|
| Development exposure | The task set was used during development and is publicly inspectable. Synthetic generation does not establish absence of exposure. | Documented separation of development and evaluation inputs, sealed evaluation procedures, and independent review. |
| Repetition | 3 executions per supplied desk fixture. The same inputs recur; the paired task bootstrap retains repetitions together. | Additional seeds address new instances; held-out templates address a different boundary. Neither follows from repeating seed-identical inputs. |
| Statistical uncertainty | A descriptive interval under empirical task resampling. | A justified workload sampling frame, treatment of related task families, and predeclared comparisons for broader inference. |
| Operational usefulness | Declared artifact acceptance and captured resource accounting. | Owner acceptance, failure severity, reversibility, review/repair costs, and deployment constraints. |
| Process and build transfer | A separately reported local-mode process pilot and released build task packs. | Track-specific controlled evaluations and completed application acceptance; desk scores do not supply them. |
What would resolve the evidence gaps
- Validate the contract. Independently review a stratified sample of requests, checks, and accepted and rejected outputs from both systems; publish the sampling rule, disagreements, adjudications, and uncertainty.
- Probe evaluator failure. Test valid alternative representations and plausible invalid mutations such as identifier substitutions, duplicate settlements, or omitted authorization. Report false acceptance and rejection separately.
- Separate sources of variation. Freeze the scorer before new outputs exist, use sealed instances and held-out templates, and run controlled configuration comparisons with declared repetitions and budgets.
- Retain an inspectable experiment. Record task and image identities, runtime and skill inventories, sampling settings, and output retention/access conditions before execution.
- Measure usefulness. Establish human completion and review effort, task ambiguity, and the relation between test acceptance and professional judgment.
These are proposed studies, not completed assurances. See the roadmap and paper's validity discussion. Report a task or grading dispute with the release, task, check, and a reproducible counterexample. Changes to released tasks or scores require an explicit versioned correction.