Eval (evaluation)

A repeatable test of whether the AI is any good: a fixed set of real questions with known correct answers, run through the system and scored — before a change and after it, the same way every time.

The test drive you give every van on the forecourt, on the same route, so the comparison actually means something.

Why it matters: without evals, "it seems better now" is a feeling. With them you can prove a change helped, and — more to the point — catch the changes that quietly broke something that used to work. That happens constantly: you improve one thing, three others get worse, and nobody notices for a fortnight. Evals are also how you learn your real accuracy before a customer does. This is exactly where "passed its demo" and "actually works" part company.

Related: Golden set · Benchmark