Write the evaluation harness before the model

Most teams meet their evaluation problem late. The model works in a notebook, someone demos it, and the question of whether it is actually good enough arrives after the deadline has been set. By then the harness is written under pressure, against a system that already exists, by someone who wants it to pass.

Start from the failure, not the metric

The useful first question is not “what score do we want” but “what does a bad answer look like here, and who notices”. Write ten of those cases down in plain language before touching a metric. On the programmes we teach, this exercise regularly kills a project idea in the first fortnight, which is the cheapest moment for it to die.

Only once those cases exist does a metric mean anything. A score of 0.91 tells you nothing on its own. A score of 0.91 with the four cases you actually care about failing tells you a great deal.

Three properties of a harness worth keeping

• It runs in under a minute. A suite that takes twenty minutes is a suite that gets skipped before a release.
• It fails loudly on the cases you wrote by hand. Aggregate scores hide exactly the regressions you were most worried about.
• It is versioned with the prompt and the model id. An evaluation result that does not record what it evaluated is a number without a subject.

What this costs

Roughly two weeks at the start of a project. Teams find that hard to justify until the first time a model update silently degrades a behaviour that no one had written a test for, and the regression is found by a customer rather than by CI.

We teach this in the first module of the evaluation track, before any modelling work, because the ordering is the lesson.