Write the Eval Before You Shop for Models

Leaderboard wins are entertaining. They are a weak proxy for your product’s actual tasks.

Before comparing GPT-this or open-source-that, write twenty to fifty real examples: inputs, expected behavior, and failure modes that matter to users.

Score retrieval separately from generation when you use RAG. A beautiful answer on bad evidence is still a product bug.

Track regression over time. The model you choose is less important than knowing when an upgrade quietly hurts precision on the cases you care about.

If you cannot define pass/fail for a change, you are optimizing for vibes. Vibes do not scale.