Build on it. What it records, where it keeps it, what holds it shut.
It never edits your code, and it runs on your machine against your keys. This page is what a developer asks next.
Watch every call the optimizer makes. On your disk first.
A campaign spends your money while you are not looking at it. So every optimizer call, backend match and escalation check emits a structured trace: to your disk always, to Langfuse when you configure it.
Cloud tracing is off until you set the keys. When you set them the payloads go too: prompts, candidates, queries, ground truth, answers. The host is a plain setting. Leave it unset and nothing leaves the machine.
| Sink | State | How |
|---|---|---|
| Local mirror | always on | langfuse/events.jsonl in the cycle folder. A pure mirror: nothing reads it back. |
| Langfuse | off until configured | Set LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_HOST, and install the observability extra. |
| MLflow | off by default | Set MLFLOW_ENABLED=true. |
Shape of a trace
- Cycle one trace
- Round a span
- L1, L2, L3 or critique call an LLM observation
- Backend match a span
What lands in it
- Optimizer calls
- prompt out, candidates back, token counts, per call and not per round
- Backend matches
- query, params, result, and any warnings the backend attached
- Escalations
- which rule fired, its priority, the reason, and the signals it read
- Scores
- origin, per round, per evaluator, hit or miss per query
pip install -e ".[observability]"Where runs live. Files on your disk.
A campaign is an object on disk. Everything it writes sits under one
user-data root: $PROMPTPOTTER_HOME if you set it, the
checkout’s .promptpotter/ when you run from source,
otherwise the app-data folder of your operating system.
| File | What it holds | For a tool reading it |
|---|---|---|
rounds/round_NNNN.json | One file per closed round. | stable |
index.json | The cycle’s index, with the final result stamped on it. | stable |
log.md | One digest per round, readable as text. | stable |
export.json | The winning version, with the measurements behind it. | stable |
dashboard.json | The live view the app reads while a run works. | live |
readout.log | The run readout the terminal printed. | live |
.runtime/ | The ledger and working state. May change between versions. | internal |
The stable surface What each file holds, and who writes it
The providers it calls.
- OpenRouter, the default
OPENROUTER_API_KEY - Groq
GROQ_API_KEY - OpenAI
OPENAI_API_KEY - Anthropic
ANTHROPIC_API_KEY
Verifiable, not certified.
There is no SOC 2 report, no ISO 27001 certificate and no third-party penetration test. What exists instead is every enforcement point written down, in a repository you can read.
| Control | How it is enforced | Written down in |
|---|---|---|
| Sign-in | Google or GitHub OIDC, verified at the API boundary. No session means a 401 before any handler runs. | ADR-0002 |
| Sessions | The cookie carries a random server-side id and nothing else: no JWT, no signed envelope. HttpOnly, Secure on https, SameSite=Lax. | ADR-0002 |
| Who gets in | Sign-up is open, and signing up is the grant. An account is bounded by the free tier’s lifetime ceilings, in dollars and in tokens. A blocklist is re-read on every request. | Access model |
| No inbound admin door | Editing the blocklist has no public endpoint at all. The box reaches out over a locked channel instead. Stated as one rule: a control-plane change never has an inbound door open to the internet. | ADR-0004 |
| The perimeter | A Cloudflare tunnel, outbound only, proxies to loopback. The app binds 127.0.0.1. No ports forwarded, no static IP. | Access model |
| Tenant isolation | Structural, not a permission check. Every store is rooted at the tenant’s own directory. Wrong owner and does-not-exist both return 404, so the API never confirms a stranger’s campaign is real. | Access model |
| Delegation attenuates | You can hand a colleague part of your own access, never more than you hold, with a spend ceiling per grant. Rights shrink as they are passed on; they cannot grow. | ADR-0005 |
| Headers | nosniff, X-Frame-Options DENY, HSTS on https, a strict CSP on the API and no-store on every API response. | Access model |
What is not true yet.
- No third-party audit. No SOC 2, no ISO 27001, no penetration-test report.
- Web-launched runs share the API process. A run launched from the web UI currently shares the API process, its environment file and its provider keys. The CLI runs as its own process; the split for web-launched runs is designed and not yet built.
- Per-user API keys are not shipped. Today an install carries one set of provider keys.
Change the code. The contracts and the gate.
The developer docs live in the optimizer repo, beside the code they describe. Each row opens on GitHub.
- Stable surface The connector protocol, the CLI flags and the read paths pinned not to change under you.
- Node standard The node interface every pipeline step implements.
- Adding a surface Golden-path recipes: the fixed set of edits for each surface, and the contract test that guards it.
- PromptPotter as a DSPy optimizer PromptPotterOpt, for a dspy.Module you already have.
- Architecture The architecture reference.
- Conventions Rules contributors follow that are not derivable from reading the code.
To contribute, install with the dev extra and run the gate. It runs every check CI runs.
git clone https://github.com/PromptPotter/prompt-potter-optimizer.git
cd prompt-potter-optimizer
pip install -e ".[all,dev]"
# every CI check
python scripts/gate.py