16 LLMs, 48 test cases, four pipeline stages. The expensive models lost, the reasoning models lost, and three times a summary document carried a number the raw data contradicted.

The Findings

Two findings drove everything else.

Expensive models lost. gpt-5.6-sol costs 2x more than gpt-5.6-terra and scored lower. Gemini Pro costs 9x more than Mistral Large 2512 and scored lower. DeepSeek v4 Pro scored worse than its cheaper Flash sibling. Price is a proxy for compute, not for quality on your specific task.

Reasoning models lost. GLM-5.2 spent 85% of its completion tokens on internal thinking and still fabricated claims. Kimi K2 Thinking scored lower than regular Kimi K2 on both tasks, cost more, and took twice as long. The models that scored 100% on resume generation aren’t reasoning models — they’re just better at following instructions.

Act I: The Budget Brawl

I built a tool called SeekerOS. It’s evaluation infrastructure — it scrapes job boards, scores postings against a resume, runs an LLM to produce a verdict (APPLY / CONDITIONAL / MONITOR / SKIP), and generates a tailored resume for the ones worth applying to. The pipeline makes four LLM calls per job: company research, JD analysis, resume generation, and a traceability check that verifies every claim in the generated resume traces back to a master resume.

The problem: I didn’t know which model to use for each task. I was routing everything through a Kilo Gateway aggregator with whatever model I’d last configured, hoping for the best.

I decided to fix this properly. I’d heard about promptfoo — an evaluation framework that runs your prompts against a dataset of test cases and gives you pass/fail results. It supports LLM-as-judge for subjective grading, Python assertions for objective checks, and YAML configs that make it easy to swap models.

First I needed a golden dataset. I exported 34 JD analysis cases from my scan history — jobs where I’d already made an apply/skip decision that agreed with the LLM’s verdict. Hard cases were excluded by construction: the dataset contains only cases where my judgment and the model’s verdict already aligned. Each case has the JD text, the expected verdict, and metadata. For resume generation, I filtered to 14 APPLY/CONDITIONAL cases (you don’t generate resumes for SKIP jobs).

Then I wrote the promptfoo configs. JD analysis was straightforward: run the model, parse the JSON, check if the verdict matches. No judge needed — it’s a pure assertion. Resume generation was harder: I needed an LLM judge that reads the generated resume alongside my master resume and flags any unsupported or overstated claims. Any fabricated metric, any embellished skill, any invented technology = fail.

I pointed it at Kilo Gateway and let it rip across their budget models: deepseek-v4-flash, Qwen, Tencent Hy3, Poolside Laguna, Minimax. Thirteen models on a 6-test quick screen, then the top performers — plus GLM-5.2, which went straight to the full run — on the full 34+14 dataset.

The results were… sobering.

DeepSeek v4 Flash and GLM-5.2 tied at 61.8% on JD analysis. Not great, but the failures had a pattern: both models confused SKIP with CONDITIONAL. They’d upgrade borderline-reject jobs to “maybe review this.” Annoying but safe — you review a job you should’ve skipped, no harm done. The dangerous failure (CONDITIONAL → APPLY, applying to a bad fit) showed up 3 times in deepseek and once in GLM.

Resume generation was worse. GLM-5.2 topped the budget cohort at 71.4%. DeepSeek scored 42.9%. Every model fabricated metrics, overstated experience, or invented technologies. One model claimed “deep hands-on experience” with a cloud platform where the master resume says “broad familiarity.” Another invented Apache NiFi out of whole cloth. A third wrote “built” where the master resume says “led.”

The free model (Tencent Hy3) scored 57.1% at $0.00. The “pro” model (DeepSeek v4 Pro, discounted tier) scored 28.6% — worse than its cheaper “flash” sibling. Reasoning tokens were a double-edged sword: GLM-5.2 spent 85% of its completion tokens on internal thinking and didn’t produce better output. It just thought about it longer before making the same mistakes.

I had numbers. I had a leaderboard. But 61.8% wasn’t good enough.

Act II: The Judge Broke, Then I Broke the Judge

Before I could test frontier models, I had to fix the eval infrastructure. The judge was broken in three ways I discovered sequentially, each more frustrating than the last.

Problem 1: The judge couldn’t see the master resume. Promptfoo’s llm-rubric assertion only passes the model’s output to the judge — not the original input prompt. So the judge was evaluating resume claims in a vacuum, with no master resume to check against. Every resume passed because the judge had no reference. I fixed this by embedding the master resume text directly in the rubric value. Obvious in hindsight.

Problem 2: Reasoning models eat all the tokens. I tried using a reasoning model as the judge. The commit that fixed this (1e387e9, Jul 16) records that reasoning models “spend 99% of tokens on thinking when used as judges, leaving only 30-75 visible tokens for the rubric JSON” — though another note in the same commit body says “leaving no visible output,” and the underlying runs were overwritten by post-fix re-runs, so neither figure is independently verifiable. The direction was clear: almost all the budget went to thinking, little or nothing to the verdict. I bumped max_tokens from 4096 to 16000. Still not enough — the model would think for 15,000 tokens and emit a two-sentence verdict. I switched to minimax/minimax-m3, a non-reasoning model that reliably produces visible JSON output. Problem solved.

Problem 3: Models prepend “Thinking:” before their JSON. Qwen and several other reasoning models output their internal reasoning as visible text before the JSON payload. The JSON parser would choke on “Thinking: I need to evaluate this job…” and fail the test. I wrote a backwards-scanning extraction function that finds the last valid JSON object in the output. This caught the edge cases — but it also meant the eval parser was stricter than production. The production parser uses a forward raw_decode that is more permissive: it finds JSON the eval misses. I confirmed this by running all 14 parse failures through the production parser. Zero tests flipped from FAIL to PASS. But 2 tests that failed in eval would have passed in production (Gemini 2.5 Flash, tests 6 and 22 — missing confidence field that production ignores), and 3 tests parsed to a different verdict. The net effect: eval pass rates are slightly pessimistic for models with parse failures, not optimistic. The gap is tech debt — the eval parser should match production — but it does not change any model’s ranking.

Full deep-dive on all three judge failures — what broke, how I fixed it, and what I learned — in Part 3.

With the judge fixed, I extended the script to support direct API calls to OpenAI, Anthropic, and Grok — not just Kilo Gateway. Each provider has different auth, different base URLs, different model naming. I wrote a case statement in compare_models.sh that handles the routing, rewrites the promptfoo YAML configs on the fly, and sets the right judge model per provider.

Then I pointed it at OpenAI.

Act III: The Frontier Arrives

I ran three OpenAI models with the full dataset: gpt-5.6-sol (the expensive one), gpt-5.6-terra (the mid-tier), and gpt-5.6-luna (the cheapest). Temperature was set to 0.3 for JD analysis and 0.7 for resume generation, with max_tokens at 16000 for both.

gpt-5.6-terra scored 82.4% on JD analysis and 100% on resume generation. After seeing budget models stuck at 61.8%, terra cleared 80% on JD analysis — a directional jump, not a precise one. And 100% on resume generation — zero unsupported claims, zero fabricated metrics, zero invented technologies across all 14 test cases.

gpt-5.6-luna, the cheapest of the three, also scored 100% on resume generation and 67.6% on JD analysis. gpt-5.6-sol, the most expensive, underperformed both — 64.7% JD and 85.7% resume. More money doesn’t buy better results. Sol had 7 API errors from rate limiting (429/quota on tests 26–33), which probably dragged its score down, but even adjusting for that, it wasn’t better than terra.

The frontier models’ JD analysis failures were mostly the safe kind: SKIP → CONDITIONAL and APPLY → CONDITIONAL. They over-review, they don’t under-review. Terra had zero false positives. Luna had one — a single CONDITIONAL → APPLY. That’s the failure mode I can live with.

Then I ran Anthropic. Claude Sonnet 5 scored 70.6% on JD analysis and 100% on resume generation — joining terra and luna in the 100% resume club. Claude Haiku 4.5 scored 92.9% on resume generation at $1.29 per eval, but tanked on JD analysis at 38.2% — worse than every budget model. Haiku is too aggressive: 11 of its 21 JD failures were false positives, recommending APPLY for jobs that should be SKIP or CONDITIONAL. Not suitable for analysis. Its properties — non-reasoning, fast, cheap, reliable JSON — make it a natural judge candidate, though I haven’t yet measured its judge accuracy directly.

Sonnet 5 had an interesting failure pattern: it was too conservative, downgrading CONDITIONAL jobs to SKIP. The opposite of GPT models, which upgrade SKIP to CONDITIONAL. Same net effect (safe failure) but from the other direction.

The frontier models had won. But I still didn’t have a budget option I felt good about — and I wanted to know if any non-frontier model could close the gap.

Act IV: The Non-Frontier Surprise

I wasn’t done. I tested nine more models through Kilo Gateway: Mistral Medium 3.1, Mistral Large 2512, Kimi K2, Kimi K2 Thinking, Cohere Command A, Meta Llama 4 Maverick, and the three Gemini variants (Pro, 3.5 Flash, 2.5 Flash).

The results were surprising.

Mistral Large 2512 scored 76.5% on JD analysis — among the strongest JD results in the whole run, frontier models included. On combined accuracy (70.8%, 34/48), it ties gpt-5.6-sol at 1/15th the eval cost ($0.39 vs $5.91). On JD analysis specifically, Mistral leads sol outright (76.5% vs 64.7%) — it is not a tie there. Only 1 false positive. Zero parse failures. This is the model that changed my hybrid config.

Kimi K2 scored 73.5% on JD analysis at $0.63 — among the strongest non-frontier options, with the safest budget failure pattern I’ve seen (1 false positive, dominant failure is SKIP→CONDITIONAL over-review).

Gemini was a disaster across all three tiers. Gemini Pro scored 60.4% combined at $3.55 — no better than GLM-5.2 while costing slightly more ($3.26), and 9x the price of Mistral Large for a lower score. Gemini 3.5 Flash had 11 false positives on 13 JD failures — the most dangerous model tested. Gemini 2.5 Flash scored 50.0% combined, statistically indistinguishable from Llama 4 Maverick (47.9%). Priced like frontier models, performed like budget models.

Reasoning tokens are a net negative. I tested Kimi K2 Thinking (the reasoning variant) against regular Kimi K2. The thinking version scored lower on both tasks (62.5% vs 68.8% combined), cost more ($1.03 vs $0.63), took twice as long (37m vs 18m), and had 1 parse failure with 9 verdict mismatches. The initial findings document classified all 10 as parse failures; the raw JSON shows only 1 was a true parse failure (test 15). The other 9 were verdict mismatches — the model produced valid JSON with the wrong verdict. This confirmed the pattern across all reasoning models I tested: they think longer, cost more, and produce worse results for instruction-following tasks.

Cohere Command A tied DeepSeek at 56.2% combined (27/48 each) — at $1.87 vs DeepSeek’s $0.15. The match is exact on both sub-scores: 21/34 on JD analysis and 6/14 on resume generation, verified from the raw eval JSON. Cohere had 1 error in each config; DeepSeek had none. Same accuracy, 12x the cost.

Llama 4 Maverick was the cheapest ($0.10) and the lowest-scoring (47.9%). Zero false positives — it never said APPLY when it shouldn’t — but it also rarely said APPLY when it should. 14 of 18 failures were SKIP→CONDITIONAL. Not useful as a filter when it flags everything for review.

The Verdict

Here’s the complete leaderboard — all 16 models that ran the full 34+14 dataset, sorted by combined accuracy:

† gpt-5.6-sol had 7 API errors (429/quota) on tests 26–33. Eval Cost = total token cost of running both eval configs (34 JD tests + 14 resume tests) for that model, recomputed from list pricing. Not per-job production cost. Three additional models (Tencent Hy3:free, Poolside Laguna, DeepSeek v4 Pro:discounted) ran resume generation only and are omitted from this table. Seven models were attempted but failed (including Claude Opus 4.8, gpt-5.6-sol-pro, gpt-5.6-terra-pro, Gemini 3.1 Pro Preview). Two models were never attempted (Grok 4.3, Grok 4.5 — the script supports them, but the cost per run would exceed any model tested). Ten models were screened only and did not advance to the full dataset. See model_status_reconciliation.json for the complete table.

The cost analysis is where it gets interesting. My pipeline makes 4 LLM calls per job. At gpt-5.6-terra for everything, that’s $0.188 per job. At deepseek-v4-flash, it’s $0.009. But I don’t have to use the same model for everything.

The recommended hybrid: DeepSeek for company research ($0.002, ASSUMED), Mistral Large 2512 for JD analysis ($0.009), gpt-5.6-luna for resume generation ($0.024), Claude Haiku for the traceability judge ($0.012, ASSUMED). Total: ~$0.046 per job. That’s 4.1x cheaper than all-terra while maintaining 100% resume accuracy. The case for the hybrid rests on cost and failure profile — Mistral Large’s 1 false positive in 34 tests — not on a precise accuracy delta over terra.

At 100 jobs per week, the hybrid costs $4.59. The all-terra approach costs $18.83. That’s the difference between running it every day and thinking twice before scanning.

Method and Limitations

All costs in this post are list-price recomputations as of July 2026, not amounts billed. Token counts come from raw promptfoo eval JSON (tokenUsage fields, main model only, excluding judge/assertion tokens). The recomputation multiplies prompt tokens × input price per million + completion tokens × output price per million, where completion tokens include reasoning tokens. The sourcing artifact is cost_recomputation_v3.json (commit f0dd0f9).

Three pricing notes:

• gpt-5.6-sol: Raw JSON JD cost was lower than the recomputed figure because OpenAI’s prompt caching discounted 67,298 input tokens on tests 0–5 at 50% off. The recomputed $5.91 uses list price without cache discount for consistency across all models.
• claude-sonnet-5: Uses introductory $2/$10 per Mtok pricing (valid through Aug 31, 2026; evals ran Jul 13, 2026). Standard $3/$15 pricing gives $8.92. Introductory is the correct basis — that was the listed price when evals ran.
• meta-llama/llama-4-maverick: Priced at DeepInfra rates ($0.15/$0.60 per Mtok), the cheapest verified host in July 2026. This model is multi-host; pricing varies $0.15–$0.90 per Mtok input by provider. The $0.10 figure is a floor.

Company research and traceability judge legs are marked ASSUMED throughout — they were not evaluated separately. Per-job production costs use per-test average token counts from the eval, not production usage data.

Every number traces to raw JSON. Three times during this project, a summary layer carried a number the raw data contradicted: the initial findings document misclassified Kimi K2 Thinking’s failures (6 parse failures claimed, 1 actual — the other 5 were verdict mismatches) and mispriced Haiku ($0.26 instead of $1.29); a v2 cost artifact had wrong terra and luna pricing ($3.0/$15.0 instead of $2.5/$15.0, and $0.5/$2.0 instead of $1.0/$6.0); the draft post inherited both. The rule that caught every error: trace each number to the raw eval JSON, not to a document that summarized it.

The dataset is frozen at evals/golden_dataset_v1_34cases.yml (commit 1e387e9) — 34 JD analysis cases and 14 resume generation cases. Phase 1C resolutions are in phase1c_resolutions.json (commit f0dd0f9).

Statistical power. Each model ran the dataset once — no repeats. At n=34 (JD analysis), the 95% Wilson CI half-width is ~±15pp at typical pass rates. At n=14 (resume generation), it is ~±22pp. At n=48 (combined), it is ~±13pp. Every adjacent pair in the leaderboard has overlapping 95% confidence intervals — including terra (87.5%) and Llama (47.9%) at the extremes. The ranking is directional, not statistically resolved.

What the eval does establish, independent of the interval: cost figures are exact — token counts are counts, not estimates. Failure direction is categorical, not a rate comparison: terra’s zero false positives vs Gemini 3.5 Flash’s 11-of-13 is a difference in behavior, not a sampling artifact. Paired comparisons on the same dataset share variance and are more robust than leaderboard rank — Kimi K2 vs K2-Thinking is the strongest result in the post for exactly this reason (same prompts, same cases, one variable). And gross tiering (frontier above budget) holds directionally even where adjacent pairs don’t separate.

The same rule that caught the findings-document and v2 pricing errors also says the rankings are softer than the table’s two decimal places suggest.

What I Learned

The 6-test screen lies. Three models scored 83.3% on the 6-test screen (DeepSeek v4 Flash:discounted, DeepSeek v4 Pro:discounted, Tencent Hy3:free). DeepSeek v4 Flash dropped to 61.8% on the full 34. n=6 is useless; n=34 is directional, not precise. The small sample over-represents easy cases. Always run the full dataset before making decisions.

Reasoning tokens don’t correlate with accuracy. Models that think longer don’t produce better output. DeepSeek Pro scored worse than DeepSeek Flash. GLM spent 85% of its completion tokens on reasoning and still fabricated claims. Kimi K2 Thinking scored lower than regular Kimi K2 on both tasks. The frontier models that scored 100% on resume generation aren’t reasoning models — they’re just better at following instructions.

More expensive doesn’t mean better. gpt-5.6-sol costs 2x more than terra and scored lower. DeepSeek Pro costs more than Flash and scored lower. Gemini Pro costs 9x more than Mistral Large and scored lower. Price is a proxy for compute, not for quality on your specific task.

Judge selection is critical. A reasoning model as judge produces ”…” and fails silently. A non-reasoning model produces clean JSON. The judge needs to see the reference material (master resume) — if it can’t, every test passes and you learn nothing.

Failure patterns matter more than accuracy numbers. 61.8% with all safe failures (SKIP → CONDITIONAL) is better than 70% with dangerous failures (CONDITIONAL → APPLY). Look at the confusion matrix, not just the pass rate.

Promptfoo is a pre-deployment gate, not a runtime cost. The eval cost $0.10–$5.95 per model to run the full dataset. In production, that same $6 covers ~32 jobs with terra or ~680 jobs with deepseek. You run promptfoo when something changes — model, prompt, rules, resume — not on every job.

Trace every number to raw JSON. The initial findings document misclassified Kimi K2 Thinking’s failures and mispriced Haiku. A v2 cost artifact had wrong terra and luna pricing. The post inherited both. Each error was caught by going back to the raw eval JSON — not by trusting a summary of it.

The known open item: validating Haiku’s accuracy as a traceability judge. Its 92.9% resume generation score proves it understands the content — but judge accuracy is a different skill that requires its own eval. That’s the next test on my backlog.

The hybrid configuration gives me 100% resume accuracy at 4.1x lower cost than all-terra. Mistral Large for JD analysis, Luna for resume generation, DeepSeek for research, Haiku for the judge. The cost is low enough to run daily without thinking twice.