LLM engineering
How to build an LLM eval set when you have no labelled data
You cannot improve what you cannot measure, and nobody hands you a labelled set. How to build a useful one in about a day.
6 min readByteWeave Studio
Most LLM features are shipped on vibes. Someone tries a dozen prompts, the outputs look reasonable, and it goes out. Then a prompt changes, something degrades in a way nobody notices for three weeks, and there is no way to tell whether the new version is better or worse than the old one.
The blocker is usually stated as a lack of labelled data. In practice you do not need a research-grade dataset. You need about sixty examples and a rule for judging them, and that is a day of work.
Start from failures, not from coverage
Do not try to sample representatively. Go and collect the cases where the current system already looks wrong: support threads, the Slack messages where someone pasted a bad output, your own notes from testing. Twenty real failures are worth more than two hundred cases that all pass.
Add the awkward inputs everyone knows about and nobody writes down — the empty input, the input in the wrong language, the one with a table pasted into it, the customer who writes in all lowercase without punctuation. These are cheap to collect and they are where regressions land.
Write the grading rule before the answers
For each case, write what a correct output must contain and what it must not. Not a model answer — a checkable rule. "Must state the refund window in days", "must not invent an order number", "must decline if the policy is not in the retrieved context".
This is where most eval sets go wrong. A single reference answer forces you into fuzzy string similarity, which punishes valid rewordings and rewards nothing you care about. A short list of assertions per case is easier to write and far easier to trust.
Grade with a model, but check the grader
Running assertions through a model as judge is fine and scales well. What is not fine is trusting it unverified. Hand-grade thirty cases yourself, run the judge on the same thirty, and measure agreement. Below roughly 90%, fix the rubric before you fix the system, because you are otherwise tuning against noise.
Keep a handful of deliberately unanswerable cases where the correct behaviour is to decline. Models that have been tuned towards helpfulness will answer them anyway, and this is the fastest way to notice that a prompt change has made your system more confident and less correct.
Make it cheap enough to run every time
An eval that takes an hour gets run before releases. An eval that takes ninety seconds gets run on every prompt change, which is when it is actually useful. Keep the set small, run it in parallel, and put the result somewhere visible.
Then let it grow only from real incidents. Every time something goes wrong in production, the fix is not complete until that case is in the set. After six months of that discipline the eval set is a genuine description of your problem, and it cost you nothing to design.
- LLM
- Evaluation
- Testing
- Prompt engineering
More reading
LLM engineering
Why your RAG demo works and your RAG product doesn’t
A retrieval demo on twenty documents proves almost nothing about the same system on twenty thousand. The four things that break in between.
6 min read
LLM engineering
Fine-tuning, RAG or prompting: choosing without wasting a quarter
Three techniques that solve different problems and get picked for the wrong reasons. A decision order that avoids the expensive mistake.
6 min read
Document AI
Why your invoice OCR works in testing and fails in production
The test folder is clean exports. The real intake is phone photographs. What changes between the two, and how to find out before a client does.
5 min read
Have a problem
worth solving?
Tell us what you're building. We'll help you figure out what's possible — and say so if we're not the right people for it.