Evals are the spec
For AI features, the eval set is the most precise requirements doc you'll write.
With traditional software, a spec describes behavior and engineers implement it. With LLM features, you can't fully specify behavior in prose. The model will meet inputs no PRD imagined.
So I write the eval set first. A good one is a few hundred real examples, each with what a great response looks like and why. Writing it forces the decisions a PRD lets you skip. What counts as resolved? When is a partial answer acceptable? Which mistakes are fine, and which ones end the pilot?
The eval set becomes the shared language between PM, engineering and stakeholders. "Is it ready?" stops being a debate about demos. It becomes a question with a number: are we above the release gate on the cases that matter?
Two lessons I've learned along the way. First, an LLM judge is only useful once you've checked it against human ratings; until then you're measuring the judge, not the product. Second, the eval set is never finished. Every bad conversation in production is a new test case.