The Eval Harness You Need Before an LLM Feature Ships

Here's a failure mode that doesn't exist in normal software: your code didn't change, and it broke anyway. A provider ships a new model version, or a user phrases something in a way you never tested, and your LLM feature quietly starts returning garbage — no exception, no stack trace, no alert. It just gets subtly worse, and you find out from a customer.

Traditional tests don't catch this because there's no deterministic "right answer" to assert against. You need an eval harness. Here's how we build one before an LLM feature goes to production.

Evals are tests for non-deterministic output

A unit test asserts f(2) === 4. An eval asserts "for this input, the output satisfies these properties" — because the exact text will vary run to run. The unit of an eval is a case: an input, and one or more checks the output must pass.

We keep a versioned set of cases as plain data (JSON or a table), and run the real model against them:

interface EvalCase {
  id: string;
  input: string;
  checks: Check[];   // each returns pass/fail + a reason
}

Three kinds of check cover most of what matters, cheapest first:

• Deterministic — does the output parse as valid JSON? Match a schema? Contain a required field? Stay under a length? These are free, fast, and catch the majority of regressions. Run them first.
• Assertion — for inputs with a known answer, does the output contain it? Did the classifier pick the right label? Exact where you can, fuzzy where you must.
• Model-graded — for open-ended output, a second LLM call scores it against a rubric ("is this answer grounded in the provided context? 0–1"). Powerful but slow and costs tokens, so it's the last resort, not the default.

The discipline is the same one that keeps agent economics sane: layer a free check in front of every paid one. Most regressions trip a deterministic check and never need a model-graded pass.

Structured output is your first guardrail

The cheapest guardrail is not letting the model return free-form text where you need data. We validate every structured generation against a schema before it can reach a user — the same Zod-at-the-boundary discipline we use everywhere:

const result = ResponseSchema.safeParse(modelOutput);
if (!result.success) {
  // retry once, then fail the job — never ship a malformed result
  return { status: 'failed', reason: 'schema_validation' };
}

A malformed generation becomes a failed job, not a broken response the frontend has to defend against. The schema is a contract the model must satisfy or the output doesn't count.

Run evals in CI, gate the deploy

The whole point is to catch regressions before users do. So the eval suite runs in CI on every change that touches a prompt, a model version, or the surrounding logic. We set a threshold — say, 95% of cases must pass — and a drop below it fails the build like any other test.

This is what makes a model upgrade safe. When a provider releases a new version, you don't cross your fingers and ship it. You point the eval suite at it, see that 3 cases regressed, and decide with evidence instead of vibes. Without evals, a model swap is a blind gamble on production.

Guardrails at runtime, not just build time

Evals catch regressions before deploy. Guardrails catch bad output at runtime, in front of the user:

• Input validation — reject or sanitize inputs that are obviously out of scope before spending a token.
• Output validation — the schema check above, plus content checks (no leaked system prompt, no PII echo, length bounds).
• Fallbacks — when validation fails, degrade gracefully: retry, return a safe canned response, or escalate to a human. Never surface the raw failure.
• Observability — log every call with inputs, outputs, and timing. When something does slip through, the logs are the only place the silent failure becomes visible.

What to copy

1. Treat evals as tests for non-deterministic output — a versioned set of cases with pass/fail checks, run against the real model.
2. Cheapest check first: deterministic → assertion → model-graded. Most regressions never reach the expensive one.
3. Validate structured output against a schema at the boundary. Malformed = failed job, never a shipped result.
4. Run evals in CI and gate the deploy on a pass threshold. This is what makes a model upgrade a decision instead of a gamble.
5. Guardrails at runtime: validate in, validate out, fall back gracefully, log everything.

The reason LLM features feel scary to ship is that they can fail without failing loudly. An eval harness plus runtime guardrails turns that invisible risk into a visible, testable one — which is the entire difference between a demo and something you'd put in front of a paying customer.

FAQ

What's the difference between an eval and a unit test? A unit test asserts an exact output (f(2) === 4). An eval asserts that a non-deterministic output satisfies properties (valid schema, contains the right answer, grounded in context) because the exact text varies each run. Evals are how you test LLM output without a single fixed correct answer.

Do I really need evals for a small LLM feature? If the feature touches customers or money, yes. Even a dozen deterministic checks (does it parse, does it match the schema, is it in range) will catch the most common silent regressions from model updates and edge-case inputs. Start small; the harness grows with the feature.

How do evals make model upgrades safe? Instead of hoping a new model version behaves, you run your eval suite against it and see exactly which cases regress. A model swap becomes an evidence-based decision with a measured pass rate, not a blind deploy you find out about from users.

---

Shipping an LLM feature that has to be reliable, not just impressive? Building the eval-and-guardrail layer is exactly the work we do — the unglamorous part that decides whether a feature survives production.