Reply to a customer about a cracked skillet
The request
The task request supplied to the adapter, alongside the workspace below. Runtime instructions, authorized tools and skills also affect the system. Containerized runs withhold host-side references; local mode is not an isolation boundary.
Laura Rogers emailed us about a cracked skillet and a late delivery. Please draft my reply to her, sticking to our refund policy, and save it as response.md.
Workspace
- email_customer_complaint.txt
- orders_export_2026-08-24_to_2026-09-02.csv
- returns_and_refunds_policy.md
- support_saved_replies_2025.txt
Expected deliverables
- response.md
This task also has a task-specific grader (check.py).
Recorded acceptance checks
The table uses check names and outcomes from the frozen campaign ledger. Acceptance requires all required predicates to pass; a predicate may itself contain a tolerance or coverage threshold. Its scope is part of the measurement contract.
| Recorded check | Type | Required | Failed / observed |
|---|---|---|---|
| response.md exists | file_exists | yes | 0 / 6 |
| correct order number | text_contains_all | yes | 0 / 6 |
| refund amount per policy | text_sentence_matches | yes | 0 / 6 |
| processing time | text_sentence_matches | yes | 0 / 6 |
| no promise the policy forbids and no wrong order number | custom | yes | 4 / 6 |
Current task declaration (5 checks)
From task.yaml. Original runner grading and frozen retrospective scoring are separate. Read the frozen source for the rules behind the reported results.
- response.md exists —
file_exists, required - correct order number —
text_contains_all, required - refund amount per policy —
text_sentence_matches, required - processing time —
text_sentence_matches, required - no promise the policy forbids and no wrong order number —
custom, required
Recorded results, desk release
This is one of the 7 tasks with a task-specific equivalence grader in the frozen scorer, intended to accept equivalent representations. Whether an equivalence rule matches the business requirement remains open to review. Original and frozen verdicts can differ; see what the frozen scorer changed.
| System | Repetitions | Frozen / 3 | Original / 3 | Median minutes | Mean model estimate (USD) |
|---|---|---|---|---|---|
| Proto / DeepSeek V4.1 Flash | 1 | 0 | 0.7 | 0.0089 | |
| Codex / GPT-5.6 sol | 1 | 1 | 0.7 | 0.1304 |
These are repeated executions on the released fixture, not independent new business cases. Cost is captured model usage priced under the recorded assumptions, excluding infrastructure and human review. Per-attempt check verdicts, raw and frozen verdicts, usage, receipt and artifact hashes for this task are in results/latest/attempts.jsonl under "task": "refund-apology-letter".
Recorded results, desk comparison, 28 September 2026
| System | Repetitions | Frozen / 3 | Original / 3 | Median minutes |
|---|---|---|---|---|
| Proto / GPT-6 sol | 3 | 2 | 0.7 | |
| Codex / GPT-6 sol | 0 | 0 | 1.4 | |
| Proto / DeepSeek V4.1 Flash | 2 | 0 | 0.5 |
A separate campaign under the same frozen scorer, not pooled with the release. Conditions and data.