PromptPotter
GitHub Try it free

It goes behind what you already use, not in place of it.

You are probably the one being asked whether to recommend it. Sometimes the honest answer is a different tool with this sitting behind it.

You already have something that works. This makes it work better.

Point it at something you already run that involves an AI answering a question. Give it examples where you know what the answer should have been. It writes a batch of alternative versions, tries each one against those examples, keeps what scored higher, and repeats. What comes back measurably beats what you started with, and shows its working.

The thing it hands back is a setting, not software. You look at it, and you decide whether to use it.

What it connects to today.

Recommending the right tool and recommending this one are usually the same conversation. Where something else is the better answer, it normally still works underneath it.

  • Langfuse

    Tracing and observability

    Yes Supported, off by default

    Every call it makes is traced. Traces are written to your own disk with no Langfuse instance involved, and stream to a Langfuse project the moment you add credentials. If Langfuse is the right answer for a client, this sits behind it rather than competing with it.

  • MLflow

    Experiment tracking

    Yes Supported, off by default

    A peer to the Langfuse sink. Flip it on and every round lands in your own MLflow server alongside whatever else you track there.

  • OpenAI · Anthropic · Groq · OpenRouter

    Model providers

    Yes All four, on your own keys

    OpenRouter is the default because it reaches the most models for the least setup. Anything OpenAI-compatible works; Anthropic has its own client. You bring the key, and you can see the spend as it happens.

  • LangChain · LangSmith · LlamaIndex

    Frameworks and their tooling

    No No integration

    There is no built-in support for these today, and we would rather say so than imply one. If what you built with them can answer over HTTP and say which settings it will let a caller change, the general path below still applies.

  • MCP

    Tool protocol

    Later Not shipped

    Planned, not written. It will show up on this page when it lands and not before.

The tools that do the same job.

Some tools also turn a score into a better version. These are the ones a fair recommendation weighs it against, described by what each says about itself.

Placed by column and band only; where a name sits inside its cell is not a ranking. Each placement rests on something the tool's own documentation states, listed below.
Every placement, and why
Each placement, with the public fact it rests on.
Tool How far it goes Can you run it yourself Why it sits there
PromptPotter Searches, stops early, and prices every call on one ledger Open source, runs on your own machines A stopping rule ends the spend once one version clearly leads, every call is priced on one spend ledger, and scores sit on one scale. Open source on GitHub.
OpikComet Searches for a better version Open source, runs on your own machines Six search methods in its Agent Optimizer, each run to the budget you set. Apache-2.0, free to self-host.
DSPy · GEPA Searches for a better version Open source, runs on your own machines GEPA searches inside DSPy programs, run to a budget you set. An open-source Python library.
OpenEvolve Searches for a better version Open source, runs on your own machines An open rebuild of the AlphaEvolve loop, run for the iterations you set. Open source.
Promptolution · CAPO Searches for a better version Open source, runs on your own machines Races candidates and drops losers under a token budget. Open source.
Langfuse Watches and scores Open source, runs on your own machines Traces, LLM-as-a-judge scores and experiments on your examples; a person writes the change. Open source, self-hostable.
LangSmith + Promptim Searches for a better version Open library, needs a hosted platform Promptim keeps an edit when the held-out score improves, and repeats. The library is open; the examples and results live in LangSmith, a hosted platform.
AlphaEvolveGoogle Cloud Searches for a better version Open library, needs a hosted platform Evolutionary search over code: Gemini proposes edits on Google’s side, your own evaluator scores them, for the program count you set. A managed Google Cloud service with an open client library.
Palantir AIP Watches and scores Closed, by contract or invitation AIP Evals scores AI workflows inside Palantir Foundry and grid-searches the prompt and model variants a person writes. Closed, under an enterprise contract.

  • Opik

    Comet · watch, score, optimize

    Open Apache-2.0, free to self-host

    An open-source platform for watching and scoring AI apps and agents, with an Agent Optimizer built in: six methods behind one API (GEPA, Evolutionary, Few-Shot Bayesian, MetaPrompt, HRPO and Parameter Optimization), LLM-as-a-judge metrics, test suites, and tuning of tool and MCP definitions. It pitches itself as the tool that fixes a problem, not only the one that finds it.

    Where this differs. Opik offers a menu of methods, each run to the budget you set. This one decides when to stop paying for evidence, puts every call on one spend ledger, and scores every version on one scale, so none can win by drawing easier questions.

  • Promptim

    LangChain · experimental library

    Hosted Runs on LangSmith

    Starts from your current version, has a model propose edits, keeps an edit only if the score on a held-out set improves, and repeats. A person can add feedback along the way. Your examples live in LangSmith.

    Where this differs. Here, a stopping rule ends the spend once one version clearly leads across many questions, what the run spent sits beside the result, and it runs on your own machines with no platform account.

  • AlphaEvolve

    Google Cloud · managed service

    Hosted Runs on Google Cloud

    Gemini proposes changes to a program, your own evaluator scores them, and the survivors seed the next generation. It is aimed at code and algorithms, with generation on Google’s side and scoring on yours.

    Where this differs. The same family of loop, aimed at the prompts and pipelines most teams ship, run end to end on their own machines. OpenEvolve is the open rebuild.

  • Palantir AIP

    Palantir · enterprise platform

    Closed Enterprise contract

    Inside Foundry, AIP Evals scores AI workflows against test suites and grid-searches every combination of the prompts and models a person writes.

    Where this differs. If you already run on Palantir, that is where this work lives for you. Outside it, this does the scoring and the search in the open, with no contract to sign first.

Compared with Google’s AlphaEvolve.

The two systems share a paradigm and part on what they are pointed at, which is why each wins a row.

Full comparison, with sources

What each can do, not a benchmark result. The AlphaEvolve column is read from its published description; “partly” is the same capability at the code level.
Capability PromptPotter AlphaEvolve
Pointed at prompts and pipelines code and algorithms
Runs on your own machines Google Cloud
Source-code algorithms no yes
The wording and the settings, tuned together yes partly
Stops paying for a losing version early yes partly
A rejected proposal is taught, not only discarded yes partly
Carries what it learned across runs yes yes
Every measurement priced yes not documented
Open, and steerable mid-run yes no
Runs from your editor yes no

Two questions decide it.

  1. Can it answer a question when asked over HTTP? Send it a question plus a set of settings, get an answer back. That is the whole requirement on this side.
  2. Can it say which settings a caller may change? A list, published once, of the knobs you are willing to let something else turn. That list is the permission boundary: nothing outside it is ever touched.

If both answers are yes, it can be optimized with no change to PromptPotter and no PromptPotter-shaped code in your system. If your system is Python and you would rather wire it in directly than stand up an HTTP surface, there is a plugin path that does the same job from your own package, without forking anything.

Worth knowing before you install one: a connector is ordinary trusted code, not a sandbox. Loading one runs it inside the same process, where it can see your provider keys. Treat a third-party connector the way you would treat any dependency you give credentials to.

Try it free See the app

The exact shape of both sides is written down: connecting your own system, the settings list, and the full source.