The Judge Problem: Why LLM-as-Judge Silently Fails

Part 3 of the series on running LLMs in systems that have to be correct. Part 1: From Chat Agent to Pipeline. Part 2: Benchmarking 16 LLMs.

Every resume passed. One hundred percent, green across the board. I had results. Every model passed.

The dashboard was green. The system was broken.

The judge couldn’t see the master resume. Promptfoo’s llm-rubric assertion passes the model’s output to the judge — and only the model’s output. The original prompt, the reference material, the source of truth? Invisible. The judge was evaluating resume claims in a vacuum. “Does this resume look good?” Yes. “Is every claim supported by the master resume?” How would I know — you didn’t give me the master resume. So it passed everything.

This is the worst kind of failure: a judge that fails open. Not “I can’t tell, so I’ll fail it” — that’s safe, that’s conservative, that catches your attention. “I can’t tell, so I’ll pass it” — that’s silent. You ship on green. You don’t even know you need to look.

I found this because I knew the budget models fabricated claims. I’d seen the output — invented metrics, overstated experience, technologies pulled from thin air. There was no way 100% was real. But if I hadn’t known that, I would have taken the green dashboard at face value. I would have shipped resumes from the cheapest model and wondered why companies weren’t responding.

Three judge failures, each more frustrating than the last. Here’s what broke, how I fixed it, and what I learned.

Symptom: 100% pass rate on resume generation across all models. Every model, including ones I knew fabricated claims, scored perfect.

Diagnosis: Promptfoo’s llm-rubric assertion sends the judge two things: the model’s output and the rubric string you wrote. It does not send the original prompt, the input variables, or any reference material. My rubric said “check if claims are supported by the master resume” — but the master resume was in the prompt, not the rubric. The judge literally could not see it.

As of promptfoo 0.121.18 (published Jul 8, 2026 — the version installed when I ran these evals), llm-rubric supports test variables in the rubric value via {{varName}} syntax. You can pass reference material as a test variable and template it into the rubric. But the default behavior — what happens when you write a rubric and run the eval — is still output-only. The judge sees the output and your rubric text. Nothing else. If you don’t know to inject the reference, you get a blind judge.

Fix: Embed the master resume directly in the rubric value. Not elegant, but explicit:

# evals/promptfoo/helpers/golden_dataset_tests.py:236-251
return {
    "type": "llm-rubric",
    "value": (
        "Judge the generated resume for traceability. Use the following "
        "system prompt as the rubric:\n\n"
        + _JUDGE_SYSTEM_PROMPT
        + "\n\n=== MASTER RESUME (source of truth) ===\n"
        + _load_master_resume_for_judge()
        + "\n\n=== END MASTER RESUME ===\n\n"
        "The generated resume is the model output below. "
        "If ANY claim in the generated resume is 'unsupported' or 'overstated' "
        "per the judge's verdict (comparing against the master resume above), "
        "the test FAILS. Only 'supported' claims pass. "
        "IMPORTANT: Output ONLY the JSON verdict object with no thinking, "
        "no reasoning, and no text before or after the JSON. "
        "Format: {\"reason\": \"...\", \"pass\": true/false, \"score\": 1.0/0.0}"
    ),
    ...
}

The fix is one function call — _load_master_resume_for_judge() reads the resume file and concatenates it into the rubric string. The diff is 27 lines. The commit body explains: “Judge was failing because it couldn’t see the master resume (only sees model output, not the original prompt).”

After the fix, the pass rates dropped from 100% to 28-71% depending on the model — and that range includes models like DeepSeek v4 Pro (28.6%) that ran resume generation only and don’t appear in the full-dataset leaderboard in Part 2 (where the resume-generation range is 42.9%-100% across the 16 models that ran both stages). The green dashboard became a real leaderboard.

Lesson: A judge that can’t see the reference material fails open — everything passes. This is the inversion that makes it dangerous. A broken assertion that fails closed (everything fails) is annoying but safe. A broken assertion that fails open is a false sense of security. You trust the results. You ship on green. You don’t even know to look.

Failure 2: The Judge That Thought Too Much

Symptom: Some judge runs produced ”…” as the entire visible output. The rubric parser couldn’t extract a verdict, so the test failed — but not because the resume was bad. Because the judge ran out of tokens before it could speak.

Diagnosis: I was using reasoning models as judges. Reasoning models — DeepSeek, Qwen, the Kilo Gateway auto-routed models — spend their completion tokens on internal thinking before producing visible output. With max_tokens set to 4096, the model would think for 4,000 tokens and emit a few dozen tokens of actual verdict. Sometimes zero. Sometimes just ”…” — the model’s way of saying “I thought about it but had no budget left to answer.”

The commit that fixed this (commit 1e387e9, Jul 16) records two observations about the severity. One note says reasoning models “use all 4096 tokens for thinking, leaving no visible output.” Another note in the same commit body says they “spend 99% of tokens on thinking when used as judges, leaving only 30-75 visible tokens for the rubric JSON.” The two figures conflict — 0 visible tokens vs. 30-75 — and the underlying runs were overwritten by post-fix re-runs, so neither is independently verifiable from the persisted eval JSON. The direction is clear regardless: reasoning models spent almost all their budget on thinking and produced little or no visible output. The exact ratio is lost.

The same commit bumped max_tokens from 4096 to 16000 for the judge. Bumping to 16000 helped. But it didn’t solve the root cause. A reasoning model with 16000 tokens thinks for 15,000 and emits a two-sentence verdict. You’re paying for 15,000 tokens of internal monologue to get a pass/fail you could get from a non-reasoning model in 200 tokens.

The real fix was swapping the judge model entirely. I switched to minimax/minimax-m3, a non-reasoning model that reliably produces visible JSON:

# evals/compare_models.sh:92
JUDGE_DEFAULT="minimax/minimax-m3"

The commit message: “kilo-auto/efficient and kilo-auto/balanced sometimes route to reasoning models that produce only ’…’ as visible output. minimax-m3 reliably produces detailed JSON verdicts with real reasons.”

After the swap, judge outputs went from ”…” to typically a few hundred tokens (median 501 across 265 judge calls, computed from the tokensUsed.completion fields in the resume generation eval JSON) of JSON verdicts with specific claim-by-claim reasoning. The judge could finally speak.

Lesson: Reasoning models are bad judges for structured-output tasks. They spend their budget thinking, not answering. A non-reasoning model with 200 tokens of visible output beats a reasoning model with 15,000 tokens of invisible thought. Pick your judge for output reliability, not intelligence.

Failure 3: When Nobody Speaks JSON

Symptom: After Failures 1 and 2 were fixed — reference material embedded, non-reasoning judge installed — 4 of 265 judge calls (~1.5%) still failed with “Could not extract JSON from llm-rubric response.” DeepSeek v4 Pro (discounted tier), Gemini 3.5 Flash, Kimi K2, and Tencent Hy3:free each hit this once. The judge’s own response couldn’t be parsed.

3a — The judge that couldn’t be parsed

Diagnosis: My first assumption was reasoning prefixes — the same root cause as Failure 2. Wrong. All four failures ran under minimax/minimax-m3, a non-reasoning judge. completionDetails.reasoning is 0 in all four grading results. The judge produced substantial visible output — 471, 1139, 1096, and 269 completion tokens for DeepSeek v4 Pro, Gemini 3.5 Flash, Kimi K2, and Tencent Hy3:free respectively. The judge spoke. The parser couldn’t extract JSON from whatever it said.

What it actually said, I can’t tell you. Promptfoo discards the judge’s raw response on grader error. The eval JSON records graderError: true and the reason string, but not the judge’s output. Four failures, four unknowable payloads. This is an observability gap in the eval tooling: a parse failure without the raw response can’t be diagnosed or reproduced.

This is still unfixed. The remediation is to log the judge’s raw response before promptfoo discards it — but as of this writing, that is not implemented and not on any tracked backlog. The eval helper in golden_dataset_tests.py has no raw-response capture, and no commit or issue addresses it.

Lesson: Structured output from any LLM is probabilistic — even from a non-reasoning judge with 269-1139 tokens of visible output. The parser is part of the judge, and a parse failure is an infrastructure failure, not a model-quality verdict. And capture the judge’s raw responses: a grader error without the payload is undiagnosable. The eval infrastructure needs the same observability discipline as production infrastructure.

3b — The model that couldn’t be parsed

A different failure class from 3a: this one is on the model-under-test side, not the judge side. Reasoning models like GLM-5.2 and DeepSeek prepend their internal reasoning as visible text before the JSON payload. Here’s an example from a GLM-5.2 JD analysis run (company name redacted):

Thinking: Let me analyze this job posting against the candidate's profile carefully.

**Company:** [COMPANY]
**Title:** SRE (Site Reliability Engineer) - the posting describes an SRE role foc...
[500 words of reasoning]
{"verdict": "APPLY", "weighted_score": 8.2, ...}

The eval’s JSON extraction function handles this by scanning backwards from the end of the output for the first valid JSON object:

# evals/promptfoo/helpers/golden_dataset_tests.py:112-153 (trimmed)
text = output.strip()
if text.startswith("```"):
    # Strip markdown fences
    ...
# Handle reasoning/thinking prefixes (Qwen, DeepSeek, etc.)
if text and text[0] not in "{[":
    # Try last fenced block first
    fence_idx = text.rfind("```")
    ...
    # Fall back: scan backwards for a brace that starts valid JSON
    search_pos = len(text)
    while search_pos > 0:
        idx = text.rfind("{", 0, search_pos)
        if idx < 0:
            break
        candidate = text[idx:].rstrip("` \n\r")
        try:
            json.loads(candidate)
            text = candidate
            break
        except json.JSONDecodeError:
            search_pos = idx

This works. But it’s stricter than production.

The tech-debt delta: Production code uses extract_json_text (backend/seeker_os/llm/json_utils.py), a forward-scanning raw_decode parser that handles markdown fences, reasoning prefixes, prose preamble, and trailing text in one pass. The eval’s backwards-scanning extractor validates each candidate with json.loads before accepting it — production’s raw_decode does not. The audit (parser_permissiveness.json, commit 8369bac) compared both parsers against the same corpus: zero outputs parse under eval but fail under production. Production parses more.

The draft of this post originally described three production parsers — _strip_code_fences, brace-matching _extract_json, and an inline regex — as if they were current. Commit b80c25c (Jul 16) consolidated all of them into extract_json_text before the audit ran. The draft was describing code that no longer existed. Same failure mode as the rest of this post: a claim about state nobody re-verified.

This means the eval can fail tests that production would pass. A model that prepends reasoning and has valid JSON at the end fails in eval but parses in production. The gap is tech debt — the eval parser should match production — but it does not change any model’s ranking. Eval pass rates are slightly pessimistic for models with parse failures, not optimistic.

Lesson: The direction of the eval/production parser gap matters — and it has to be measured, not assumed. My eval ended up stricter than production, which is the safe direction: models that fail in eval might pass in production, but not the reverse. I arrived there by accident. If the gap had gone the other way — eval more permissive than production — I would have shipped models that pass eval but fail in production. Verify which direction your gap runs. Don’t assume it’s safe.

Before You Trust Your Judge

Verify the judge can see the reference material. Send a test case you know should fail. If it passes, your judge is blind. Don’t assume the framework passes context to the judge — verify it.
Don’t use a reasoning model as judge — and know that max_tokens won’t save you if you do. Reasoning models spend tokens on thinking, not verdicts. The obvious mitigation for a reasoning judge that runs out of tokens is to raise max_tokens. I bumped from 4096 to 16000. It helped — but it didn’t solve the problem. A reasoning model with 16000 tokens thinks for 15,000 and emits a two-sentence verdict. You’re paying for 15,000 tokens of internal monologue to get a pass/fail you could get from a non-reasoning model in 200 tokens. Swap to a non-reasoning judge. If you must use a reasoning judge, give it room — but understand you’re papering over the root cause, not fixing it.
Test your parser against real model output. Don’t test with clean JSON. Find a model that prepends “Thinking:” and run your parser against it. If it fails, you’ve found the bug before production does.
Match your eval parser to your production parser. Mine ended up stricter, which is the safe direction — but I got there by accident. Measure which way your gap runs before you trust either set of results.
Watch for the 100% pass rate. If every model scores perfect on your first run, something is wrong with the judge, not something is right with every model. A 100% pass rate is a red flag, not a green light.
Log your judge’s raw responses. Frameworks may discard them on parse failure — a grader error without the payload is undiagnosable.

What’s Next

The three failures here — blind judge, silent judge, unparseable judge — were discovered and fixed between Jul 13 (commit 00d1949, initial eval setup) and Jul 16 (commit 1e387e9, consolidated fixes). The parser consolidation followed the same day (commit b80c25c). I learned more about eval infrastructure during those three days than during the model evaluation itself. The audit that confirmed the eval/production gap ran Jul 27 (commit 8369bac) — a separate pass, eleven days after the last of them.

The judge is the foundation. If the judge is broken, every result is fiction. Test the judge before you trust the results.

But here’s the limit of what this post’s method can check. I can verify a judge is not blind — give it a test case that should fail and see if it passes. I can verify a judge is not silent — check that it produces visible, parseable output. What I cannot verify is whether the judge is right. A judge that sees the reference material and speaks clearly might still agree with the model when it shouldn’t, or disagree when it shouldn’t. Measuring that requires a reference standard — a set of cases where a human has adjudicated every claim — and I do not have one. That is a different problem from the three failures here, and it needs a different kind of eval. It’s on the backlog.