The first post in a series on running LLMs in systems that have to be correct.

I watch so many around me struggle to find a job. There are numerous tools and methodologies to assist individuals in a job search, but I’ve never seen one that approached the problem the way an engineer would: as a data pipeline with correctness requirements. So I decided to build one.

I had a second motive. I wanted a real project — with real stakes, real data, and real failure modes — to sharpen my understanding of operating LLMs in production-shaped systems. A toy project teaches you the API. A system that has to be correct teaches you the discipline. So I dogfooded it: my own profile, my own resume, my own pipeline of postings became the test data.

This is the story of building that tool, what broke, and what it taught me about operating LLMs in systems where correctness matters.

The False Start: Clawford

My first attempt was Clawford — an openclaw conversational agent that managed the whole workflow through chat. I’d built it on an agent framework, gave it skills for scanning job boards, scoring postings, and managing an opportunity pipeline. It ran on a dedicated machine, posted results to a private GitHub repo, and sent me Telegram notifications. It was genuinely useful, and it was the wrong architecture.

The state lived in conversation. The list of jobs I’d seen, the scores I’d assigned, the companies I’d rejected — all of it was threaded through chat history and files on disk, managed by the agent’s own logic. When I asked “what’s in my pipeline?”, it would list files, read status markers, and summarize. When I said “pursue this one,” it would scaffold a folder, move a file, and update a status field. The agent was the system of record.

That’s the problem. An agent is nondeterministic. The same command could produce different folder structures, different file contents, different status updates. The scoring was reliable because I’d pushed it into deterministic scripts, but the workflow around it — the state transitions, the file management, the status tracking — rode on chat memory and LLM-generated file operations. There was no audit trail. If a status field got corrupted, I couldn’t tell when or why. If a file went missing, there was no transaction log to replay.

The breaking point was state integrity. The COMMANDS.md file in the repo has a warning that says it all: “Always verify status.md writes with a read-back before committing — benchmark caught models silently checking wrong checkboxes.” The agent could write a status file that said “applied” when I hadn’t applied. It could move a file to the wrong folder. It could lose track of where a job was in the pipeline. And because state lived in files managed by LLM-driven operations, there was no transaction log, no audit trail, no way to replay what happened.

Resume generation was still manual — testing that workflow meant pasting a JD and my master resume into Claude.ai and tailoring by hand. I wanted to automate it, and I knew that automating it through an agent would mean trusting the agent not to fabricate claims. I needed to know that every claim was traceable. I needed to know that the status field said “applied” because I actually applied, not because the agent hallucinated a status update. Correctness-critical steps can’t ride on chat memory.

An agent is a great interface and a terrible system of record. I tore it down and rebuilt.

The Rebuild: A Pipeline, Not a Conversation

SeekerOS is a pipeline. Jobs flow through deterministic stages, and LLM calls happen at defined points inside scaffolding that catches errors, logs everything, and ensures the model never writes state directly — its outputs are parsed, validated, and written by pipeline code.

The stages:

1. Ingest — Fetch job cards from an aggregator that indexes postings across the major ATS platforms (Greenhouse, Lever, Ashby, Workday, and others). Dedup through layered checks — URL hash, composite source key, fuzzy title-plus-company match. Insert into SQLite. No LLM here — just HTTP and SQL.
2. Filter — Deterministic card-level hard filters: title, location, commitment, compensation floor. This is the gate that decides which jobs are worth a JD fetch. No LLM.
3. Fetch JD — Pull the full job description. ATS API where available, HTML extraction otherwise. A content hash on the full JD text catches reposts here. Still no LLM.
4. Score — A heuristic scoring engine applies a configurable rubric to the full JD: positive modifiers for AWS, Terraform, remote, strong comp; negative modifiers for on-site-only, low comp, management-heavy. Pure regex, no LLM. The score decides ranking — and which jobs are worth spending LLM budget on for analysis.
5. Analyze — First LLM call. The JD analyzer reads the full description and produces a structured verdict: APPLY, CONDITIONAL, MONITOR, or SKIP, with a weighted score, fit signals, and gaps. The verdict acts as a cap on the composite score — a SKIP is capped low no matter how high the heuristic score.
6. Company Research — Second LLM call. A research agent gathers company context — funding, layoffs, tech stack, engineering culture — and produces a signed delta that adjusts the score.
7. Resume Generation — Third LLM call. The generator reads the master resume and the JD, produces a tailored resume. The prompt is explicit: reorder and reframe, never fabricate. But prompts aren’t enforcement.
8. Traceability Check — Fourth LLM call. A separate model extracts every factual claim from the generated resume and checks each one against the master resume. Unsupported or overstated claims are flagged as violations. This is the guardrail that makes the honesty requirement real.

The stack is FastAPI and SQLite on the backend, Next.js on the frontend, and an LLM gateway that routes tasks to specific models. Config-driven — no hardcoded personal values.

(There’s also a lightweight metadata-extraction LLM call when I add a job manually by URL — five call sites total, four decision points. The fifth just pulls structured fields like title, location, and comp range out of the fetched JD text when the ATS doesn’t provide them.)

War Story 1: The Queue That Lied

The ready queue is where jobs wait for my review — scored, analyzed, sorted by net score. One day I noticed a SKIP-verdicted job sitting at the top of the queue, above three APPLY jobs I wanted to look at first.

The cause was a NULL coalescing bug. The net score — the composite that incorporates the AI verdict cap — was stored in a separate column. Jobs that hadn’t been analyzed yet had net_score = NULL. The sort query was ORDER BY score DESC — the raw heuristic score, not the net score. So a job with a heuristic score of, say, 8.0 and a SKIP verdict (capped to 3.0 in net score) ranked above a job with a heuristic score of 6.5 and an APPLY verdict (no cap, net score 6.5). The scores here are illustrative — the cap values are real, straight from the rubric config. The verdict cap was computed and stored, but the sort ignored it.

The fix was one line: ORDER BY COALESCE(net_score, score) DESC. Fall back to the heuristic score when net score is NULL (pre-analysis jobs), but use the capped score when it exists. The SKIP dropped from the top of the queue to the bottom, where it belonged.

This is a classic separation-of-concerns bug. The scoring logic was correct — the cap was computed and stored. The display logic was correct — the UI showed the net score. The sort logic was the gap. Three systems touching the same data, one of them using the wrong column.

War Story 2: The Status That Wouldn’t Stick

The second bug was worse because it was silent. I’d mark a job as “interested” or “reviewing” — a pre-apply decision — and the status would never persist. I’d come back to the job later and it would still say “ready,” as if I’d never touched it. My decisions were being silently dropped.

The cause was endpoint routing. A job’s status moves through two distinct phases: the pipeline phase (discovered, filtered, jd_fetched, ready) and the application lifecycle (applied, interviewing, offer, and so on). The application lifecycle has a dedicated transition endpoint with strict validation — it only accepts legal post-apply transitions. The frontend was routing the pre-apply statuses — “reviewing” and “interested” — through that same post-apply endpoint. The validation did exactly what it was designed to do: it rejected them. The fix was to route pre-apply statuses through the general update endpoint, and reserve the transition endpoint for the post-apply state machine it was built to guard.

There’s a companion guardrail on the pipeline side: the auto-analysis policy has analysis_excluded_statuses — a configurable list of statuses considered decided-dead, like “rejected.” Jobs in those statuses are excluded from auto-analysis candidate selection, so the pipeline doesn’t waste LLM spend re-analyzing jobs that are already off the board.

The lesson: a status field isn’t just data — it’s a commitment, and it deserves a state machine. But validation that’s strict in the wrong place is its own failure mode. The endpoint correctly rejected an illegal transition; the system incorrectly asked it to perform a legal one. My intent was valid — the plumbing dropped it. Silent rejection of valid user intent is a bug, not a safety feature.

The Traceability Checker

The honesty guardrail is the centerpiece. “The AI must not lie on my resume” sounds like a prompt instruction — and it is. But prompts aren’t enforcement. A prompt says “don’t fabricate.” A model mostly complies. But “mostly” isn’t a guardrail.

The traceability checker is a separate LLM call that runs after resume generation. It extracts every factual claim from the generated resume and judges each one against the master resume. The verdict per claim: supported, unsupported, or overstated. Unsupported and overstated claims are HIGH-severity violations.

Take a real bullet from my master resume: “Owned release management and SRE for Hilton’s web and mobile app rebuild — driving the engineering, process, and cultural changes that took the organization from monthly releases to one or more per day, as needed — release-on-demand in practice.” A generator that wants to impress will offer “architected the migration from monthly to daily releases, cutting release cycle time 30x” — same facts, invented ownership claim, derived metric it was never given. That’s an overstated verdict. An invented metric — “reduced deployment time by 80%” — is unsupported. The checker flags both.

This isn’t a prompt. It’s a validator. The generator and the checker are separate models, separate prompts, separate calls. The generator is incentivized to produce a good resume. The checker is incentivized to find lies. They’re not colluding — they’re adversarial. If the generator invents a metric, the checker catches it. If the generator overstates a role, the checker flags it. Nothing physically stops me from sending a resume with violations. But nothing gets sent without the checker’s verdict in front of me — the contract isn’t “the machine won’t let me lie,” it’s “I can’t lie by accident.”

Why a validator and not a better prompt? Because a prompt is a request. A validator is a contract. I can test a validator — feed it a resume with a known fabrication and verify it catches it. I can’t test a prompt. The validator is the difference between “I asked the model not to lie” and “I proved the model didn’t lie.”

What This Is

SeekerOS started as a tool to solve a problem I kept watching people fight — tracking a modern job search without losing state, decisions, or honesty along the way. But it’s equally a working lab for the discipline of running LLMs in systems that have to be correct. Every lesson in this series came from building it:

• The pipeline structure that puts LLM calls in deterministic scaffolding.
• The scoring system that uses cheap deterministic heuristics to decide where LLM budget gets spent, and LLMs only for the nuanced analysis.
• The traceability checker that enforces honesty as a contract, not a request.
• The observability layer that logs every LLM call — model, prompt, response, tokens, latency.
• The evaluation framework that benchmarks models against a golden dataset before they touch production.

To be continued:

• LLM Observability: Seeing What Your Models Actually Do
• The Quest for the Right LLM: A Promptfoo Story
• The Judge Problem: Why LLM-as-Judge Silently Fails

The agent was a great interface and a terrible system of record. The pipeline is a decent system of record and a terrible interface. The right answer is probably both — agent on top, pipeline underneath. But that’s a later post. For now, the pipeline works, the guardrails hold, and the honesty contract is checked by code, not left to trust.