Background

Run it yourself

A clone, Docker, three provider keys, and the commands in order.

Setup

Node 22, Docker, and keys: Anthropic for the agents, OpenAI for embeddings, OpenRouter for the Matcher. Every harness tool refuses the live database by name; the worst a mistake can do is cost money.

git clone https://github.com/minerval-ai/minerval && cd minerval && npm ci
docker compose up -d                     # Postgres 16 + pgvector
cp .env.example .env                     # add ANTHROPIC_API_KEY, OPENAI_API_KEY, OPENROUTER_API_KEY
npm run corpus:reset                     # the separate database

Detail

The evals, in a sensible order

npm run corpus:golden -- --profile=production                    # cents
npm run corpus:run -- blackholes --profile=production --limit=1   # read the meter
npm run corpus:run -- blackholes --profile=production --score     # a baseline; three times
npm run corpus:calibrate -- review                               # then read the judge
npm run corpus:property -- idempotency blackholes --profile=production
npm run corpus:swap -- eggs --agent=steward --model=claude-sonnet-5 --profile=production
npm run corpus:contributions -- blackholes
npm run predictions -- seed --corpus --drain

How this guide is updated

Commit the result under corpus/, run the sync, commit what it writes under web/content/evals/. The guide renders from those files at build time.

npx tsx scripts/sync-frontend-content.ts     # corpus/ → web/content/evals/

Harness docs: corpus/README.md · the rubric · SCORING.md.