All work
Evaluation
Open source
2024
An open-source harness for regression-testing LLM features before every deploy, so prompt changes stop breaking production quietly.
Scroll to resolve
3.1k
GitHub stars
Client
Open source
Role
Creator & maintainer
Stack
Python · Pytest plugin · DuckDB
Year
2024
The problem
Every team I worked with shipped prompt changes on vibes and found the regressions from support tickets.
What I built
Evalbench turns a folder of examples into a test suite: graded by rules, by a judge model, or by a human, with diffs between runs stored in DuckDB.
The result
It’s now used by teams at 40+ companies and has 3,100 stars on GitHub.