A number here is worth exactly what its protocol is worth, so every run names its own dataset cut, its model and how it was scored. That is what lets you tell a real lift from a saturated benchmark — and a comparable pair from two runs that never met. A work in progress; more runs land here.
The loop does not change between these. The task does, and with it the only thing PromptPotter actually needs from you — a way to tell a correct answer from a plausible one.
Correct means one canonical record, from any surface form
Names, codes and abbreviations arriving from a dozen upstream systems, all meant to resolve to the same thing. The failure mode is quiet — a near-match that reads as right and is not — so the scoring has to be stricter than a human skim.
Measured on TermNorm
Correct means the final answer, after multi-step work
Competition math, logic chains, extra-hard tasks. The model can usually do the individual steps; the prompt decides whether it commits to the right ones and leaves enough work on the page to be checked.
Correct means an answer no single passage contains
The evidence is split across documents, so the prompt has to make the model gather before it answers instead of settling for the first passage that looks relevant.
Measured on HotpotQA
Correct means whatever your scorer says it is
Anything you can measure. Point a connector at the backend you already run, bring a labelled set, and the same loop optimizes against your own definition of a good answer.
Start with the developer docs
Six datasets, each with its own failure mode. Two are still running, and say so.
| Benchmark | What it measures | Result | Gain | Run |
|---|---|---|---|---|
| TermNorm | Entity normalization on a five-step backend | 0.94 composite | +0.31 | Round 4, up from 0.63 · $2.41 · 12 of 312 hard samples still missing |
| AIME | Competition math | 60% pass | +24 | Round 1 · one single-axis edit: a “combinatorial count” cue |
| JustLogic | Logical depth ≥ 6 | 44% pass | +17 | Round 4, after L2 fired, up from 27% · L1 alone stalled at 31% |
| GSM8K | Grade-school math | ≥95% pass | 1 round | Groq · gpt-oss-120b · $0.34, larger than a free run |
| BBEH | Big-Bench Extra Hard | In validation | — | Held-out test split · public reference 14.8% |
| HotpotQA | Multi-hop QA across paragraphs | Queued | — | M11 Track 1 · Groq · gpt-oss-120b · vs. promptfoo, promptolution |
None of these is a discount. Each one is a decision about where the next dollar buys the most evidence — and each is a setting you own, in a config file on your own machine, billed to your own key.
Absolute accuracy hides more than it shows — lifting 60→75 and 90→93 look alike until you know the ceiling. So every run reports four metrics, and we hold the model, the split and the scoring fixed across every method we compare. No cross-paper number mixing.
correct / total. The figure papers report — but meaningless without the base model and origin prompt.
(opt − base) / (ceiling − base). The fraction of available improvement actually realized — normalizes across models.
Headroom per query. How economically the gain was found — fewer LLM calls for the same lift.
Queries to reach 90% of the final headroom. Separates "finds good prompts" from "finds them fast."
Numbers here are preliminary — the BBEH head-to-head is mid-run, and the table fills in as runs land. The protocol and the held-out splits are public.
These are public so you can judge the protocol behind them. The number worth having is the one from your own examples, where you already know the right answers and nobody else's split is involved.