Dispute letter for a supplier invoice that does not match the PO
supplier-dispute-letter · timeout 1200 s · source
The ask
Sent verbatim as the prompt. The agent works in a copy of the workspace below and sees nothing else.
The Cascade invoice for our last welding supplies order doesn't match what we ordered and received. Please write the dispute letter to their AR team; Emily's note says what goes in it. Save it as dispute.md.
Workspace
- Cascade_invoice_INV-86187.pdf
- PO-2026-0509.xlsx
- note_from_ops.txt
- price_increase_notice_july2026.txt
- receiving_log_aug2026.csv
- supply_agreement_SA-2025-11_excerpt.md
Expected deliverables
- dispute.md
This task also has a task-specific grader (check.py).
Checks
The task passes only when every required check passes. Reference answers, tolerances, and the planted traps are in the repository, not here.
| Check | Type | Required | Failed (frozen) / 6 |
|---|---|---|---|
| dispute.md exists | file_exists | yes | 0 |
| invoice and PO referenced | text_contains_all | yes | 0 |
| disputed lines with amounts | custom | yes | 0 |
| total disputed and amount paid now | text_numbers_present | yes | 0 |
| price clause cited | text_sentence_matches | yes | 3 |
| freight-inclusive total absent | text_not_contains | yes | 0 |
Recorded results, complete desk comparison
pass pass, frozen scorer only fail timed out
| System | Repetitions | Frozen / 3 | Raw / 3 | Median minutes | Mean cost (USD) |
|---|---|---|---|---|---|
| Proto / DeepSeek V4.1 Flash | 3 | 3 | 1.6 | 0.0206 | |
| Codex / GPT-5.6 sol | 0 | 0 | 1.3 | 0.1510 |
Per-attempt check verdicts, raw and frozen verdicts, usage, receipt and artifact hashes for this task are in results/latest/attempts.jsonl under "task": "supplier-dispute-letter".