Website FAQ from the summer support tickets
faq-from-tickets · timeout 1200 s · source
The ask
Sent verbatim as the prompt. The agent works in a copy of the workspace below and sees nothing else.
Can you turn our July and August support tickets into an FAQ for the website? Dana's note says what she wants. Save it as faq.md.
Workspace
- helpdesk_tickets_2026-07-01_to_2026-08-31.csv
- note_from_dana.txt
- saved_replies_2026-03.txt
- subscription_policies.md
Expected deliverables
- faq.md
This task also has a task-specific grader (check.py).
Checks
The task passes only when every required check passes. Reference answers, tolerances, and the planted traps are in the repository, not here.
| Check | Type | Required | Failed (frozen) / 6 |
|---|---|---|---|
| faq.md exists | file_exists | yes | 0 |
| top five questions in order | custom | yes | 0 |
| pause answer uses the current limit | text_sentence_matches | yes | 0 |
| damaged coffee answer uses the current window | text_sentence_matches | yes | 0 |
| change deadline answer | text_sentence_matches | yes | 0 |
| cancel answer | text_sentence_matches | yes | 0 |
| free shipping answer uses the current threshold | text_sentence_matches | yes | 0 |
| withdrawn minimum absent | text_not_contains | yes | 0 |
Recorded results, complete desk comparison
pass pass, frozen scorer only fail timed out
| System | Repetitions | Frozen / 3 | Raw / 3 | Median minutes | Mean cost (USD) |
|---|---|---|---|---|---|
| Proto / DeepSeek V4.1 Flash | 3 | 0 | 1.8 | 0.0216 | |
| Codex / GPT-5.6 sol | 3 | 0 | 1.8 | 0.2744 |
Per-attempt check verdicts, raw and frozen verdicts, usage, receipt and artifact hashes for this task are in results/latest/attempts.jsonl under "task": "faq-from-tickets".