PromptPotter
GitHub Try it free

Benchmarks. What was measured, on what, and how to check it.

A number here is worth exactly what its protocol is worth, so every row names its dataset cut, its model and how it was scored.

Transcribed from the protocol on . The protocol is the record

Screened

from the protocol

Public benchmark cuts scored once with nothing optimized, to see whether a gain there could mean anything.

20 cuts screened

Admitted
4
Held
6
Turned away
10

The admission bar

from the protocol

The cheapest model a campaign would ship on. If it already solves a dataset, or cannot reach it, the dataset does not run.

25 rows per screen

Model
openai/gpt-oss-20b
Reasoning
effort low
Sampling
temperature 0

Optimized results

No optimized result is published yet. Runs are scheduled; until one lands, this page is a protocol and a roster.

0 of 4 admitted datasets have a published run

Measured
one round, 34 examples

One round, replayed

A headline number

from the protocol

What a result has to stand on before it may appear in the table below.

200+ rows scored

Tuning runs
50 to 100 rows
Tuning figures shown
never

20 public benchmark cuts screened. 4 admitted.

Each one is scored first with nothing optimized. If the cheapest model already solves it, or cannot reach it at all, no gain measured there means anything, so it does not run. The table also lists every dataset folder the repo ships, screened or not.

Show datasets by verdict

from the protocol An origin score is what the starting point scored before any optimizing. None of them is a result. Unless a row says otherwise, the score comes from our admission screen: openai/gpt-oss-20b, reasoning effort low, temperature 0, 25-sample slice. A row with no folder has no scorer in the repo yet.

Datasets, what is scored on each, and what the protocol decided
Dataset Task Scored by Model pinned Origin Origin score on a scale of 0 to 100 Verdict
BBEH mini# datasets/bbeh Hard reasoning puzzles, 23 task families 460 rows label_match the final answer against the key mistral-small-3.2-24b-instruct mistralai · effort low · temperature 0 28% Admitted Headline benchmark · scored before the dataset moved to another model, to be re-measured
JustLogic, depth 2 to 4# datasets/justlogic-d234 Three-way deduction: true, false or uncertain 600 rows label_match the label, less a small charge for tokens gpt-oss-20b:nitro openai · temperature 0 50 to 62.5% Admitted Focus instrument · floor 35% · from the recon slice, to be re-read on the current wiring
AIME 2025# datasets/aime_2025 Competition maths, integer answer 30 problems aime_match the boxed integer, compared exactly gpt-oss-20b:nitro openai · temperature 0.7 30% Admitted Wired · 30 problems, too small to split
SealQA LongSeal# datasets/sealqa-longseal-12 Search questions over a 12-document haystack, one agent episode each 254 tasks label_match the answer, as the sealqa judge grades it qwen3.7-flash:nitro qwen · 4 turns · judge: gpt-oss-120b 26.3 to 33% published figure Admitted Published figure · judge-graded agent episode
PlanBench task 1# no folder: screened only Symbolic planning: write a valid plan Not wired gpt-oss-20b 36% Held Needs a plan validator
NaturalPlan# no folder: screened only Planning in plain language: trips, calendars, meetings Not wired gpt-oss-20b 36% Held Meeting planning alone (43%) is the clean cut
MuSiQue 3-hop# no folder: screened only Three-hop questions over supplied paragraphs Not wired gpt-oss-20b 38% Held Overlaps BBEH’s reading tasks
BBEH mini, medium effort# no folder: screened only Hard reasoning puzzles, 23 task families Not wired gpt-oss-20b 44% Held In range, but up to 220 s per sample
BoardgameQA depth 3# no folder: screened only Three-way rule reasoning: proved, disproved or unknown Not wired gpt-oss-20b 64% Held Revisit as a recovery target
GSM8K# datasets/gsm8k Grade-school maths word problems 1,319 test rows gsm8k_match the final number, compared exactly gpt-oss-120b openai · effort medium · temperature 0 78% literature figure Held Literature figure · verdict withdrawn, to be measured
MMLU-ProX Swahili# no folder: screened only Multiple-choice questions in Swahili Not wired gpt-oss-20b 36% Turned away In range, wrong axis: it would learn translation tricks
JustLogic d6–7# no folder: screened only Three-way deduction at depth 6 to 7 Not wired gpt-oss-20b 44% Turned away Floor · answers “Uncertain” on about 80% of rows
PopQA# no folder: screened only Factual recall about little-known entities Not wired gpt-oss-20b 44% Turned away Memorised · 10 of 11 hits were one entity
JustLogic d≥4, early cut# no folder: screened only Three-way deduction at depth 4 and up Not wired gpt-oss-20b 52% Turned away Label bias · always-uncertain scores 44%
BoardgameQA depth 2–3# no folder: screened only Three-way rule reasoning: proved, disproved or unknown Not wired gpt-oss-20b 60% Turned away One class is never predicted
ExploreToM# no folder: screened only Tracking who believes what in a story Not wired gpt-oss-20b 68% Turned away Ceiling
AR-LSAT# no folder: screened only LSAT analytical-reasoning puzzles Not wired gpt-oss-20b 72% Turned away Saturated · solved directly
FOLIO# no folder: screened only First-order logic over short premises Not wired gpt-oss-20b 80% Turned away Saturated · reproduced on two providers
MuSR# no folder: screened only Multi-step reasoning over a narrative Not wired gpt-oss-20b 81% Turned away Ceiling · skewed answer key
SATBench# no folder: screened only Satisfiability puzzles in plain language Not wired gpt-oss-20b 100% Turned away Saturated at every filter
JustLogic, depth 2 to 4, held# datasets/justlogic-d234-held The same deduction task under a wider held-out set 1,500 rows, 900 of them held out label_match the label, less a small charge for tokens gpt-oss-20b:nitro openai · temperature 0 None not in the roster Not screened The held-out instrument for the head-to-head
Reactome TypeQL# datasets/reactome-typeql-42 Writing database queries that are then executed 42 questions accuracy_r2 the executed query’s result, graded by TypeDB’s harness mimo-v2.6-flash xiaomi · effort none · development model only None not in the roster Not screened Outside the head-to-head: read against TypeDB’s published rows
SpreadsheetBench, 20 tasks# datasets/spreadsheetbench-s20 Editing a workbook in a container, one agent episode each 20 tasks env_reward the task’s own verifier, less a small charge for tokens qwen3.7-flash qwen · effort none · 20 turns None not in the roster Not screened The search panel for agent skills
SpreadsheetBench, 10 tasks# datasets/spreadsheetbench-s10 Editing a workbook in a container, one agent episode each 10 tasks env_reward the task’s own verifier, less a small charge for tokens mercury-2.5 inception · temperature 0 · 20 turns None not in the roster Not screened An instrument check, too small to search on
Terminal-Bench, regex-log# datasets/harbor-tbench-regex-log One terminal task, run as an agent episode 1 task env_reward the task’s own verifier gpt-oss-120b openai · temperature 0 · 30 turns None not in the roster Not screened A connector check, not a number to publish
LCA TermNorm# datasets/lca-termnorm Matching bill-of-materials names, through a multi-step retrieval pipeline size not stated in the repo rr how high the right entry ranks gpt-oss-120b openai · temperature 0.3 None not in the roster Not screened The regression set for multi-step pipelines
Email tagging# datasets/email-tagging Routing an email into one of four categories 15 rows label_match the category against the key gpt-oss-20b openai · temperature 0 None not in the roster Not screened A pilot, not a curated benchmark
Screen taste# datasets/screen-taste-v0 Ordering ten films or series from ten the person likes 20 rows list_rr how high the held-out title ranks gpt-oss-20b:nitro openai · temperature 0 None not in the roster Not screened The demo dataset
PromptPotter on itself# datasets/promptpotter-self Tuning the optimizer’s own prompts; each sample is a whole inner campaign 6 inner campaigns mean_round_delta the inner campaign’s gain per round, scaled to 0 to 1 None declared no model of its own: it runs JustLogic, depth 2 to 4 None not in the roster Not screened The self-optimization instrument
PromptPotter on itself, fixture# datasets/promptpotter-self-e2e The same recursion cut down to one cell, for the browser test 1 inner campaign mean_round_delta the inner campaign’s gain per round, scaled to 0 to 1 None declared no model of its own: it runs JustLogic, depth 2 to 4 None not in the roster Not screened A test fixture. No result is read off it

Three numbers. A definition beside each.

What a published result will consist of. Nothing is priced per query: a lift is priced in wall clock and dollars. What we report

The numbers PromptPotter reports, from the protocol
Number What it says Read from Example
Absolute accuracy Correct over total on the test set, for the best version found. Higher is better. results_*.json, per method and model AIME 2025 starts at 30%: the same ratio, on the 25-row screen and before any optimizing.
Rounds to separable The first round whose pick separates from the starting point on the shared panel. Lower is better. This is the one to quote. index.json::final.rounds_to_separable No run is published yet, so there is no value to show.
Rounds to improved The first round that crowned a winner. A promotion verdict with no interval, so it is not a result on its own. index.json::final.rounds_to_improved No run is published yet, so there is no value to show.

Run one yourself. On your own key.

Python 3.13 or later, an OpenRouter key, and a pipeline backend to score against. Any folder name from the table works in place of aime_2025.

Terminal
# the repo, with the public-dataset loaders
git clone https://github.com/PromptPotter/prompt-potter-optimizer
cd prompt-potter-optimizer
pip install -e ".[all,benchmarks]"

# a campaign on one dataset folder, then its rounds
python -m promptpotter new aime_2025
python -m promptpotter resume

# grade the starting point and the pick on the held-out rows
python -m promptpotter bench