What the desk results establish

Acceptance means that all required predicates pass. A predicate can contain several rules or a tolerance, so passing the contract does not prove every possible property of an artifact. The payment example shows an actual contract with a 90% agreement threshold inside one required predicate.

Averaging check outcomes gives partial credit to rejected artifacts. That is useful for diagnosis, but the mean fraction and complete acceptance answer different questions. Their difference is 6.96 points for Proto and 10.10 points for Codex.

The pooled fraction divides all passed checks by all required checks. It gives more weight to tasks with more predicates and cannot be substituted for the attempt-weighted mean. Duplicating a predicate can change either partial-check metric without changing acceptance. The number of failed checks also does not establish repair effort or failure severity.

A custom predicate can bundle many business rules. The ledger establishes which tests failed; it does not show that the artifact was almost correct, that a single edit would repair it, or that the agent skipped self-verification. Distinguishing an agent error from an overly strict check requires artifact review against the task's requirements. That independent review has not been completed.

At-least-one acceptance uses the grader to identify a successful output after observing all runs. It does not establish a practical retry policy: a user needs a way to recognize success. All-3 acceptance records repeatability on fixed inputs; it is not the probability that a monthly production workflow will succeed.

The all-repetitions interpretation follows τ-bench's pass^k. Here, different task difficulties and only 3 observations per fixture prevent a pooled pass rate from serving as a general reliability model. New fixtures and held-out templates test different forms of generalization.

Reports and drafting contribute 37 additional Proto passes, exceeding the net difference of 34. Across the other 138 tasks, the counts are 377/414 for Proto and 380/414 for Codex. This post hoc calculation locates the observed difference; it does not justify replacing the full comparison with a selected subset.

Weighting each category equally changes the gap from 6.06 to 3.55 points. Neither weighting measures business demand. Use the workload comparison to inspect this sensitivity, then inspect relevant task contracts before treating a category label as a match for your own work. The data do not isolate whether differences arise from models, harnesses, skills, or evaluator coverage.

Total captured estimated model spending divided by accepted attempts allocates spending on failures to the accepted outputs. It excludes human review, repairs, infrastructure, and uncaptured request usage. It is neither a provider invoice nor an expected cost for retrying until success.

Proto has lower estimated spending and median time, but more tokens, slightly more summed task time, and a worse p90. The cheaper model route therefore does not dominate every resource dimension. Resource accounting gives the price basis and missing observations.

The original gap is 2.85 points; the frozen gap is 6.06. The 3.21-point change comes from evaluating the same retained outputs differently, including recovery from original grader errors. It is not agent improvement and does not by itself establish scorer bias.

The frozen scorer was constructed after outputs existed by the organization developing Proto. Sharing one scorer makes the primary comparison consistent, while freezing its fingerprint makes later changes detectable. Neither establishes that the scoring choices were independent of the observed outputs. Independent adjudication is needed to decide which verdicts better reflect business acceptance. See the full transition accounting and self-audit.

Both systems pass all repetitions on 116/187 tasks; 15 tasks have at least one system fail every repetition, and 1 have neither system pass. Their acceptance counts differ on 59 tasks. These observations describe this public suite; high scores on a development-exposed task set do not establish that a class of business work is solved.

The next useful evidence is a scorer reviewed before execution, sealed fixtures and templates, and independent judgments of retained artifacts. A same-model harness experiment would address a different question from this system comparison. The research roadmap separates these planned tests from the released desk results and the process pilot.