Tasks · desk · Extraction
Log vendor insurance certificates and flag lapsed policies
insurance-certificates · timeout 1800 s · source
The ask
Sent verbatim as the prompt. The agent works in a copy of the workspace below and sees nothing else.
Can you log the vendor insurance certificates into coi.csv for our compliance review? They're in the certificates folder; the compliance note says what goes in it.
Workspace
- certificates/COI_Hollowell_Electric_2026.pdf
- certificates/Hollowell COI 2025-09.pdf
- certificates/Pinnacle_Roofing_evidence_of_insurance.pdf
- certificates/Westbrook-COI.pdf
- certificates/pemberton_hvac_certificate.pdf
- certificates/scan_cedar_pine_coi.pdf
- compliance_note.txt
Expected deliverables
- coi.csv
Checks
The task passes only when every required check passes. Reference answers, tolerances, and the planted traps are in the repository, not here.
| Check | Type | Required | Failed (frozen) / 6 |
|---|---|---|---|
| requested columns | csv_columns | yes | 0 |
| one row per policy | csv_set_equal | yes | 0 |
| row count | csv_row_count | yes | 0 |
| insureds and carriers | csv_values_match | yes | 0 |
| coverage types | csv_values_match | yes | 0 |
| limits | csv_values_match | yes | 0 |
| policy dates | csv_values_match | yes | 0 |
| status | csv_values_match | yes | 0 |
Recorded results, complete desk comparison
pass pass, frozen scorer only fail timed out
| System | Repetitions | Frozen / 3 | Raw / 3 | Median minutes | Mean cost (USD) |
|---|---|---|---|---|---|
| Proto / DeepSeek V4.1 Flash | 3 | 3 | 1.8 | 0.0222 | |
| Codex / GPT-5.6 sol | 3 | 3 | 1.5 | 0.2111 |
Per-attempt check verdicts, raw and frozen verdicts, usage, receipt and artifact hashes for this task are in results/latest/attempts.jsonl under "task": "insurance-certificates".