Testing non-deterministic systems like LLM apps.
Why exact-match tests fail on AI output, and how to build a suite that is stable, honest and keeps learning.
Testing non-deterministic systems, such as applications built on large language models (LLMs), means checking properties of the output instead of exact matches. You describe what every acceptable answer must contain and must avoid, run each case more than once, and read pass rates instead of single results.
The first test most people write against a language model asserts that the output equals an expected string. It passes a few times, then the model phrases the answer differently and the suite goes red, even though nothing is wrong. The usual response is to delete the test. The better response is to notice that it was asserting the wrong thing.
Why do equality assertions fail on LLM output?
Equality assertions fail because they assume there is exactly one correct output, when for most of what a language model does there is a set of acceptable outputs. A model generates text by sampling, picking each next word or word fragment from a range of likely candidates, so the same prompt can produce different wording on each run. Two replies can both be right while sharing few words.
An exact-match test cannot tell a correct rewording from a wrong answer. It fails when the model rephrases, and it can pass on a bad answer if the expected string was copied from a run that was already wrong.
Why deleting the test is the wrong fix
Deleting a flaky test removes the check along with the bad assertion. The test existed because someone cared about something specific: that the reply quoted the right order number, perhaps, or did not promise a refund. That concern did not go away when the wording changed.
Teams that delete these tests tend to fall back on reading a few outputs by eye before each release. That misses regressions from later prompt changes, and it does not scale past the one person who remembers what good looked like. Keep the intent of the test and change the assertion.
Describe the set of acceptable outputs
A good test for a non-deterministic system describes the boundaries of the acceptable set rather than picking one member of it. This is less exotic than it sounds. Nobody asserts the exact bytes a PDF generator produces; they assert it has four pages and the total on page three is correct.
A useful test asks whether the output could be wrong in a way that matters.
Each of the assertions below survives the model choosing different words.
- Required facts are present. The reply contains the customer's order number and the delivery date on record.
- Claims stay inside the source. Every figure in a summary appears in the document being summarised.
- The format holds. Valid JSON with exactly the four fields requested and dates in the agreed format.
- Limits are respected. Under the length limit, in the customer's language, with at most one follow-up question.
- Forbidden content is absent. No competitor named, no refund promised, no legal advice given.
- It fails safely. When the answer is not in the supplied material, it says so or escalates instead of guessing.
Some of these need only ordinary code: a JSON parser, a pattern match, a word count. Others, such as whether every claim is supported by the source, need judgement.
Where exact matches still belong
Equality is still the right assertion wherever a task has one correct answer. A classification label, an invoice number pulled from a document and the name of the tool an agent chose should all match exactly. Normalise first (trim whitespace, ignore case where it does not matter, convert dates to one format) so the test fails on substance rather than formatting.
The skill is splitting each response into the parts that must be exact and the parts that only need to be acceptable.
How do you test an LLM application?
You test an LLM application in layers: ordinary tests for the code around the model with the model stubbed out, structural checks on real model output, and judged checks of quality on a schedule. Splitting the suite this way keeps the flaky parts contained and the fast parts fast.
The deterministic layer
Everything around the model is ordinary software: parsing, retrieval filters, permission checks, escalation rules, arithmetic validation and the code that turns model output into an action. Test it with the model replaced by a stub, a stand-in that returns fixed responses, including malformed and empty ones. These tests are fast, cheap and do not flake.
This layer is also where many serious bugs live. A permission check that lets the wrong document through is not subtle, and it does not need a model to detect. Teams that skip this layer and jump straight to judging answer quality end up with a suite that is slow, expensive and silent about the bugs it could have caught.
The structural layer
Here the real model runs, and the assertions are about shape: valid JSON, required fields present, values in range, no empty strings where content was required. Shape is stable enough to check on every commit, provided the set is small enough to keep the run quick and the cost modest.
Many providers offer structured output modes that constrain the model to a schema. Depending on the provider, these reduce or remove format failures, but a valid field can still hold the wrong value.
The judgement layer
This layer asks whether the answer is any good: accurate, complete, supported by its sources and appropriate in tone. It is slow, costs money on every run and is somewhat noisy, so it runs nightly and before a release rather than on every push. Most grading is done by a second model acting as a judge, checked against a smaller amount of grading by people.
| Layer | What it checks | When it runs | Cost and noise |
|---|---|---|---|
| Deterministic | What it checks: Parsing, permissions, rules and actions, with the model stubbed | When it runs: Every commit | Cost and noise: Low cost, no noise |
| Structural | What it checks: Shape of real model output: schema, fields, ranges | When it runs: Every commit, on a small set | Cost and noise: Moderate cost, little noise |
| Judgement | What it checks: Accuracy, grounding and tone, graded by a model | When it runs: Nightly and before a release | Cost and noise: Higher cost, some noise |
| Human review | What it checks: A sample of outputs, and the judge's own grades | When it runs: Before a release and when calibrating | Cost and noise: Highest cost, graders can disagree |
Build an evaluation set from real cases
An evaluation set is a fixed collection of inputs, each paired with what a correct response must contain or avoid, that the system is run against repeatedly. Running a system against one is what people usually mean by evals. Its value depends almost entirely on where the cases come from.
Invented cases are usually written by the same people who wrote the prompt, so they test what the team already expected. Real cases come from the inputs the system will actually see: historic support tickets, recent documents, questions asked during the pilot. Remove or mask personal data according to your data handling rules before any of it goes into a repository.
What goes into the set
A useful set mixes several kinds of case, and each kind catches a different problem.
| Case type | Where it comes from | What it catches |
|---|---|---|
| Typical | Where it comes from: A sample of ordinary real inputs | What it catches: Regressions in everyday quality |
| Edge case | Where it comes from: Long, short, multilingual or badly formatted inputs | What it catches: Brittle prompts and fragile parsing |
| Known failure | Where it comes from: Past errors, escalations and corrections | What it catches: The same mistake coming back |
| Should not answer | Where it comes from: Questions outside scope or outside the source material | What it catches: Guessing instead of declining or escalating |
| Adversarial | Where it comes from: Attempts to override instructions or extract data | What it catches: Prompt injection and data leaks |
Give each case its expected properties, a note of its source and a category. Categories matter because an overall pass rate can hide a collapse in one small group of cases.
Version the set alongside the code
Keep the evaluation set in the same repository as the prompts and code, and change it through the same review process. A result then means something specific: this prompt version, with this model setting, against this version of the set. When a pass rate moves, you can tell whether the system changed or the test did.
Hold some cases back. If every prompt change is tuned against the full set, the prompt slowly learns the set rather than the task. A held-out portion that only runs before a release gives an honest second reading.
How do you use an LLM as a judge?
You use an LLM as a judge by giving a second model the output, the source material and a set of narrow yes-or-no questions, then checking its grades against grades from people. It is the practical way to check quality at a volume nobody can read by hand.
Ask the judge narrow questions
A judge given a vague rubric produces vague grades. Asking for a score out of ten tends to yield middling-to-high numbers almost regardless of input. Specific binary questions produce usable signal: does the answer address the question asked, is every claim supported by the supplied context, does it say clearly when the context is insufficient.
judge:
model: not the model that wrote the answer
given: question, source context, reference answer
answers_the_question yes/no
every_claim_in_context yes/no
admits_when_unsupported yes/no
stays_in_scope yes/no
reason: one sentence for every no
Give the judge the original question, the source material and, where one exists, a reference answer written by a person. Ask for a one-sentence reason with every "no" so failing grades can be checked quickly. Anything code can check, such as length or format, should be checked with code.
Calibrate the judge against human grades
A judge is itself a model, so it needs testing too. Have people who know the domain grade a sample of outputs against the same questions, run the judge on that sample, and compare. Where they disagree, read the cases: usually a question was ambiguous, or the judge lacked context a person had.
Re-check that agreement whenever the judge's model, its prompt or the provider changes. A judge that has silently drifted is worse than no judge at all, because the suite keeps producing confident numbers that no longer mean what they did.
Keep the author and the judge apart
Do not let a model grade its own output. A model judging text written in its own style tends to be generous to its own habits, and it shares the blind spots that produced the error. Use a different model for the judge. Where the judge compares two answers side by side, run the comparison in both orders, since judges can favour the answer shown first or the longer one.
Judge runs cost money, and the cost multiplies. As an illustrative example with round numbers, 200 cases run five times with one judge call each is 1,000 judge calls for a single nightly run. Keep judged runs off the commit path, test whether a smaller judge agrees with people just as well on your questions, and apply the habits in keeping model costs down.
Read pass rates instead of single results
Because the output varies, one run of a case tells you little. Run each case several times and record how often it passes. A case that passes every time and one that passes three runs out of five are different findings, though a single run might show both as green.
Agree thresholds before anyone looks at results, or whatever the system scored quietly becomes good enough. Agree them with whoever owns the business outcome, and set them per category.
- Safety and permission properties must pass on every run. Leaking another customer's data on one run in five is a serious defect whatever the average says.
- Format properties should pass on every run once structured output is in place.
- Quality properties get a floor agreed in advance, and a release must not fall below the previous baseline by more than the normal run-to-run wobble.
Small sets make rates jumpy. In a set of twenty cases, one case flipping moves the pass rate by five percentage points, so look at the direction across several runs, and at which cases moved, before deciding the system got worse.
| Pattern | What it usually means | What to do |
|---|---|---|
| Passes every run | What it usually means: The case is handled reliably | What to do: Keep it as a regression guard |
| Fails every run | What it usually means: A consistent gap in the prompt, data or code | What to do: Fix the cause, then keep the case |
| Passes some runs | What it usually means: The behaviour is fragile for this kind of input | What to do: Tighten instructions or add examples, then rerun |
| Falls after a change | What it usually means: The change probably caused a regression | What to do: Compare outputs from before and after the change |
| Falls with no change | What it usually means: Inputs or the provider's model may have shifted | What to do: Check provider notices and recent production inputs |
What temperature and seeds do and do not guarantee
Lowering the temperature setting reduces variation, but it does not make output identical. Temperature controls how adventurous the sampling is: at low values the model favours its most likely next word, so repeated runs look more alike. Even at zero, many hosted models can return different outputs for the same input, for reasons on the provider's side such as how requests are batched and computed.
Some providers offer a seed parameter that fixes the random choices made during sampling. Where it exists it is typically described as best effort, and it does nothing once the provider updates the model behind a name. Some models do not let you set temperature or seed at all, so check what your provider documents.
If tests run at temperature zero and production runs at a higher setting, the tests check a configuration you do not ship. And a system that gives the same wrong answer every time is stable, not correct. Test the configuration you actually run, and rely on properties and pass rates for confidence.
Testing retrieval and agents on their own
When a system retrieves documents or acts through tools, the final answer is the last link in a chain. Testing only that answer tells you something went wrong without telling you where.
How do you test retrieval?
You test retrieval by checking whether the right passages came back for a question, separately from whatever the model then wrote. Retrieval-augmented generation (RAG) means the system searches your documents for relevant passages and hands them to the model as context. If the passage holding the answer never arrived, no prompt change fixes the answer.
Build a retrieval set of real questions, each labelled with the passages that should be found, and check each result against these properties.
- Found. The expected passage is among the results handed to the model.
- Ranked. It appears near the top, since lower passages may be cut when context is limited.
- Excluded. Documents the user is not permitted to see are kept out, every time.
- Current. Replaced documents no longer come back.
With a fixed index, retrieval is far more repeatable than generation, so these tests can run often. Rerun them whenever the chunking (how documents are split into passages), the embedding model (which turns text into numbers for search) or the index changes. If you are still deciding whether retrieval is the right approach, weigh the options in choosing between fine-tuning and retrieval.
How do you test an AI agent?
You test an AI agent by asserting on the trace of what it did (which tools it called, with what arguments, in what order, and when it stopped) as well as on its final message. An agent is a model that decides its own next step and acts through tools such as a database lookup or a refund system. Run these tests against stubbed or sandboxed tools, never live ones.
- Tool calls. The right tool for the situation, with valid arguments that match the case.
- Step order where it matters. Identity verified before an account changes, stock checked before an order is confirmed. Assert the constraint rather than one exact sequence, because several paths can be valid.
- Stopping conditions. It stops when done, stops at the step limit, and hands over rather than looping on a failed call.
- Tool failures. Make stubbed tools time out, return errors, return nothing or return malformed data. The agent should retry within limits, report or escalate, and never claim an action succeeded when it did not.
Designing agents so these behaviours can be tested from the start is part of how we build custom AI agents.
What runs on each commit, nightly and before a release
Not every test belongs on every push. Cheap, stable checks run constantly, while slow, expensive and noisier checks run less often but always before anything reaches users.
| When | What runs | Why then |
|---|---|---|
| Every commit | What runs: Deterministic tests and a small structural smoke set | Why then: Fast feedback on the bugs code can catch |
| Nightly | What runs: Full structural set and judged evaluation, with repeated runs | Why then: Catches quality drift without slowing developers down |
| Before a release | What runs: Everything, including held-out cases and human review of a sample | Why then: The go or no-go decision against agreed thresholds |
| Model or provider change | What runs: The full release run plus judge recalibration | Why then: The model underneath changed even if the code did not |
| In production | What runs: Structural checks on live output, the judge on a sample | Why then: Real inputs drift away from the test set |
Treat a model change, a prompt change or a provider's model update like a code release, because each changes behaviour just as much. Where the provider allows it, pin a specific model version rather than an alias that points at whatever is newest, so the date of a change is one you chose. Pinned versions are eventually retired, but providers usually give notice.
Why do test suites go green forever?
A test suite goes green forever when every case in it is one the system already handled on the day it was written. It passes permanently while quality quietly declines on inputs nobody thought to include. That is more dangerous than a flaky suite, because a flaky suite at least draws attention to itself.
Grow the suite from production
The fix is to make the suite grow from real use. Every escalation, every complaint and every case someone had to correct becomes a test case with the correct answer attached. If the system hands work to people, those handoffs are a steady supply of hard cases, which is one more reason to design escalation paths for agents that record how each case was resolved.
Do that consistently and the suite becomes a record of everything the system has got wrong, which is exactly what it should be checking. A failing test then means a specific customer's case breaking again, which is far easier to prioritise than a dip in a quality score.
The numbers to watch
Track the number of cases and the pass rate side by side over time. A rising count with a steady pass rate is a system getting more capable. A flat count is a suite that has stopped learning, and it will stay green while the system drifts underneath it. Break both numbers down by category, since a healthy total can hide one area getting steadily worse.
Monitoring in production is where testing continues
Monitoring a live AI system is the same testing discipline applied to real traffic, because production inputs change in ways no evaluation set anticipates. Structural checks are cheap enough to run on every response, and judged checks can run on a sample. Report both against the same categories and thresholds as the suite.
Some production signals tell you more than any score.
- Escalation rate. A sudden change often means the inputs changed.
- Correction rate. How often people overrule or edit the output, broken down by category.
- Explicit feedback. Complaints, negative ratings and follow-up messages saying the answer was wrong.
- Cost and latency. Rising tokens per request can point to a prompt, retrieval or looping problem before quality visibly drops.
Log enough to replay a failure: the input, retrieved passages, tool calls, output, prompt version and model setting, kept according to your data retention rules. A failure you can replay turns into a test case quickly. One you cannot replay turns into an argument about what probably happened.
A plan for setting up LLM evaluation
The order below does the cheap, reliable work first.
- Write down what wrong looks like. List the failures that would matter, such as wrong figures, invented claims, leaked data or a wrong action, ranked by harm.
- Separate the code from the model call. Make the call replaceable with a stub and test the surrounding code as ordinary software.
- Add structural checks on real output. Schema, fields, ranges, length and forbidden content, on a small set run on every commit.
- Collect a first evaluation set from real cases. Include edge cases, known failures and cases the system should decline, and commit it to the repository.
- Write judge questions and calibrate them. Yes-or-no questions, checked against a sample graded by hand.
- Agree thresholds before the first run that counts. Per category, with whoever owns the outcome, in writing.
- Run cases repeatedly and record pass rates. Store each result with the prompt version, model setting and set version.
- Test retrieval and tools on their own. Add retrieval sets and agent trace tests where the system has them.
- Wire up the schedule. Every commit, nightly, before a release and on any model change.
- Feed production back in. Turn escalations and corrections into cases regularly, and watch case count and pass rate together.
Without some version of this, the decision to go live rests on opinion, which is a common way for AI pilots to stall before production.
Key takeaways
- Test non-deterministic output by asserting properties every acceptable answer shares, and keep exact matches for fields with one right answer.
- Test the code around the model with the model stubbed, because many serious bugs live there and those tests do not flake.
- Build the evaluation set from real cases, including edge cases and known failures, and version it with the code.
- Use a model as a judge only with narrow yes-or-no questions calibrated against human grades, and never let a model grade its own output.
- Run each case several times, read pass rates, and agree thresholds before seeing any results.
- Low temperature and fixed seeds reduce variation but do not guarantee identical output, especially across model updates.
- Grow the suite from production escalations and corrections, and watch case count and pass rate together.
Where to start with testing an AI system
If you have nothing yet, start with the deterministic layer and a small evaluation set drawn from real inputs. Both are cheap, they catch the plainest bugs, and they give every later step something to stand on. Add the judge once there is a set worth judging.
If a system is already live without this, pull recent escalations and corrections, turn them into cases, and see what passes. An independent AI strategy and review can look at the test suite, the thresholds and the monitoring together.
If you would like help designing tests for an AI system, schedule a call and we will talk through where your current checks stop and what to add first.
Frequently asked questions
You can unit test the code around an LLM, but not the model's wording in the usual sense. Stub the model call and test parsing, permissions, rules and actions as ordinary software. For real model output, use property checks such as valid structure, required facts and forbidden content, and run them several times, because the same input can produce different wording on each run.
An LLM eval is a repeatable test that runs a fixed set of inputs through an AI system and scores the outputs against defined criteria. The criteria can be checked by code, by a second model acting as a judge, or by people. Evals are usually run several times per case, read as pass rates, and compared against a previous baseline.
LLM as a judge is reliable enough to be useful when its questions are narrow and it has been calibrated against human grades. Vague rubrics such as scores out of ten produce vague results. Ask specific yes-or-no questions, give the judge the source material, compare its grades with people's on a sample, and recheck that agreement whenever the judge model or prompt changes.
No, setting temperature to zero reduces variation but does not guarantee identical output. Many hosted models can still return different responses to the same input because of how the provider processes requests, and a model update changes behaviour regardless of settings. Seed parameters, where offered, are usually best effort. Test the configuration you run in production and rely on pass rates rather than repeatability.
There is no fixed number, but an evaluation set needs enough real cases to cover each category of input and each failure that matters. A small set of well-chosen real cases, including edge cases and known failures, is a better start than a large invented one. Small sets make pass rates jumpy, so grow the set steadily from production escalations and corrections.
Test for hallucinations, meaning confident claims with no basis in the source, by checking whether every claim in the answer is supported by the material the system was given. Code can verify figures, names and references directly. A judge model can answer a yes-or-no question on whether each claim appears in the context. Include cases where the answer is absent and check the system says so.
Yes, mock the LLM when testing the code around it, and use the real model when testing its output. A stub that returns fixed, malformed and empty responses makes tests of parsing, permissions and error handling fast and stable. Structural and quality checks must run against the real model and configuration you ship, otherwise they only prove that the stub behaves.
Run cheap structural checks on every commit, full judged evaluations nightly, and the complete suite with human review before every release. Also rerun everything when the model, the prompt or the provider changes, since behaviour can shift without any code change. In production, keep running structural checks on live output and judge a sample, so testing continues after launch.