Tasks · desk · Extraction
Log new vendors' tax forms with masked TINs
vendor-tax-forms · timeout 1800 s · source
The ask
Sent verbatim as the prompt. The agent works in a copy of the workspace below and sees nothing else.
We can't pay the new taproom vendors until their tax info is logged. Can you fill in vendor_tax.csv from the forms they sent? AP's instructions are in the folder.
Workspace
- ap_instructions.txt
- new_vendor_requests.csv
- vendor_tax_forms/Clearview_taxpayer_letter.pdf
- vendor_tax_forms/Deborah_Okafor_vendor_form.pdf
- vendor_tax_forms/Howard_Properties_tax_form.pdf
- vendor_tax_forms/KC_Sign_Works_tax_info.pdf
- vendor_tax_forms/RiversideLinen_taxform.pdf
- vendor_tax_forms/VendorForm_Torres.pdf
- vendor_tax_forms/scan_20260825_0912.pdf
- vendor_tax_forms/tax_form_current.pdf
Expected deliverables
- vendor_tax.csv
Checks
The task passes only when every required check passes. Reference answers, tolerances, and the planted traps are in the repository, not here.
| Check | Type | Required | Failed (frozen) / 6 |
|---|---|---|---|
| requested columns | csv_columns | yes | 0 |
| one row per vendor | csv_set_equal | yes | 0 |
| row count | csv_row_count | yes | 0 |
| legal and DBA names | csv_values_match | yes | 1 |
| entity types | csv_values_match | yes | 0 |
| TIN types | csv_values_match | yes | 0 |
| masked TINs | csv_values_match | yes | 0 |
| no full TIN in the file | text_not_contains | yes | 0 |
| tax addresses | csv_values_match | yes | 0 |
| form status | csv_values_match | yes | 0 |
Recorded results, complete desk comparison
pass pass, frozen scorer only fail timed out
| System | Repetitions | Frozen / 3 | Raw / 3 | Median minutes | Mean cost (USD) |
|---|---|---|---|---|---|
| Proto / DeepSeek V4.1 Flash | 3 | 3 | 1.4 | 0.0149 | |
| Codex / GPT-5.6 sol | 2 | 2 | 1.3 | 0.1868 |
Per-attempt check verdicts, raw and frozen verdicts, usage, receipt and artifact hashes for this task are in results/latest/attempts.jsonl under "task": "vendor-tax-forms".