Change Order

How to test an LLM search summary for invented results

6 September 2026

Your search endpoint returns five records. The user asks for ten. The model writes a polished list with a few extra projects that your application never retrieved.

The previous case study describes that failure in a production search feature. This follow-up asks a narrower question: what can an automated test actually catch?

Here is a small, runnable example. It uses synthetic records and deliberately bad outputs, requires no API key, and does not claim to measure how often a real model hallucinates.

Download the complete test file, then run it with Node.js 20 or later:

node --test summary-contract.test.mjs

The file contains eight tests. They cover a restricted output contract, including rejection cases. The tests pass when unsafe output is rejected; they do not expect the suite itself to fail.

Start with what the user can verify

For this interface, the rule is:

Every project selected by the model must belong to the records currently shown to the user.

That is a product decision, not a universal rule for search. An application that supports citations to additional records might choose a different boundary. Here, the five visible cards are the only project references the summary is allowed to make.

Use the exact snapshot that produced those cards. Checking against the entire database would allow a real but invisible sixth record through. Checking against a newer search response could validate the wrong page of results.

The first regression case is intentionally small:

const visibleIds = new Set(['P1', 'P2', 'P3', 'P4', 'P5']);
const selectedIds = ['P1', 'P2', 'P3', 'P4', 'P5', 'P6'];

const valid = selectedIds.every(id => visibleIds.has(id));
// false: P6 was not shown to the user

This catches an unseen ID. It does not catch every invented claim.

A valid citation can still accompany a false statement

Consider this output:

Imaginary Palace (P1) is the largest hotel.

Suppose P1 is a valid visible ID, but its real name is Harbour Hotel. A checker that only extracts IDs will accept the citation while missing both the invented name and the unsupported ranking.

This is a limitation of the earlier article’s citation approach: asking for reference IDs can make output easier to inspect, but a model can reuse a valid ID beside an incorrect claim. The instruction alone does not enforce correctness.

The download includes a test demonstrating that blind spot. It is useful to keep examples of what your validator misses alongside examples of what it rejects.

One enforceable option: let the model select, let the application name

For a deliberately restricted version of the feature, accept only this shape:

{ "project_ids": ["P2"] }

The application checks the shape, rejects unknown or duplicate IDs, and resolves display names from the same trusted rows used by the cards. The total also comes from the server. Additional fields, including free-form prose, are rejected.

In the downloadable example, the rendered text becomes:

47 matching projects. 5 shown above. Selected: Market Hotel.

That removes model-generated names from this output path. It also sacrifices the flexibility of a narrative summary. This is an alternative design to evaluate, not a claim that this implementation shipped in the original production system.

If the model’s selection is unnecessary, skip the model entirely and render the search results directly. If selection is useful, remember that a valid selection can still be irrelevant to the user’s question. Membership validation does not establish relevance or ranking quality.

For a real UI, render these strings as text or through your framework’s escaping, not by inserting them as raw HTML. Trusted record identity does not make a record’s text safe HTML.

Keep the cases that expose the boundary

The example checks:

These are deterministic tests of the application boundary. They are inexpensive to run when you change the renderer, validator, or response contract.

When integrating this pattern, handle rejection explicitly: retain the valid search cards, show a short fallback such as “The summary is unavailable,” and record the validation failure. Do not fall back to displaying the rejected prose.

Evaluate the actual model separately

Passing these tests says nothing about whether your current prompt usually returns valid output. For that, run your real model adapter against fixed inputs and validate each response.

Include queries that ask for more records than the preview contains, requests for rankings unsupported by the supplied data, empty results, and requests for names that are absent. Include straightforward queries too, so a change that rejects everything does not look successful.

For every run, record the query, exact input snapshot, prompt version, model identifier, raw response, and validation result. Repeat cases because one successful response is weak evidence for a variable system. Keep transport errors separate from invalid model outputs.

Report both the number of runs and failures: “0 invalid selections in 50 runs of these fixtures” is a bounded observation. It is not proof that the model cannot hallucinate. No live-model evaluation results are included here.

If you retain free-form prose, you need additional checks for names, amounts, locations, and the relationships between them. Exact string matching misses paraphrases; another model judging the text can also make mistakes. Use these checks as evaluation evidence, and make the remaining uncertainty explicit.

Make the next failure easy to reproduce

Start by saving one problematic query and its exact visible records. Turn it into a fixture. Write down the claim your interface can enforce, and add a test showing how the old behavior crosses that boundary.

You do not need to solve every kind of hallucination to stop an invented sixth record reaching this particular UI. You do need to be precise about which guarantee your code provides.

Get the eight runnable tests, or read the original debugging story for the prompt and data-flow context.