Eval Set Starter Kit

Run ten example cases against a model, check every answer, and fail the build when quality drops below a frozen baseline.

What's inside

Ten example cases from a made-up retailer's help center
A scorer that checks the facts an answer must contain and what it must never say
A frozen baseline to compare every run against
A gate that fails the build when the pass rate falls below the baseline or a confident wrong answer appears
The gate set up for GitHub Actions and for Azure DevOps
An offline stub model that needs no key, and 18 unit tests for the scorer and the gate

Engineers who change prompts or models and want proof that the answers did not get worse before the change ships.

Guides are shared under CC BY 4.0, so you can copy and adapt them with credit. Code and config files are MIT licensed.