DSPy — Automatic Prompt Generation
This article is for people who build agents and LLM pipelines and are tired of tuning prompts by hand. For a simple task a prompt along the lines of "You are a helpful assistant, answer the questions" is enough. But once an agent calls functions and handles unusual requests, the prompt grows into several pages of rules, stops working after a model switch, and everything has to be tuned again. How to write such prompts by hand is covered in the article on prompt engineering, and how an agent calls tools in the article on RAG and function calling.
DSPy (GitHub) offers another way. You describe the task in code: what comes in, what has to come out, and which steps the program consists of. DSPy assembles the prompt text itself. And if you have examples with correct answers and a function that scores an answer, an optimizer picks, for every step, the instruction and examples that score higher. Switched the model? Run the tuning again instead of rewriting the text.
The framework comes from the Stanford NLP group. Its author is Omar Khattab, and the first paper on it appeared in October 2023. The name stands for Declarative Self-improving Python, and the project's motto is "programming, not prompting".
The article follows one example from start to finish: a support agent for an online shop that answers customers' questions from a knowledge base in SQLite. First we go through the core concepts, then connect a model, describe the task, build an agent with a tool, learn to score it, tune it with two optimizers, MIPROv2 and GEPA, and save the result. The code was checked on DSPy 3.4.1, the latest release as of November 2026, with the local Gemma 4 E4B model in LM Studio. Program outputs and measurements in the article come from that run; where a number is taken from the documentation or counted with a fake model, the text says so.
Pros: the prompt stops being "magic text": in its place you have code, a metric and a reproducible result. Switching models becomes a re-run of the tuning, with a score before and after. A small model after tuning often catches up with a large one that nobody tuned.
Cons: you need examples with correct answers and a way to score an answer; without them there is nothing to tune. Tuning takes from tens of minutes to hours and can cost money. The API moves fast: between versions 3.0 and 3.5 the agents, model connection and adapters were reworked. The final prompt is written by a machine, and editing it by hand is awkward.
What DSPy is made of
Which concepts you need to know
DSPy has its own vocabulary, and the documentation is hard to read without it. These are the words you will meet below:
- LM (language model): the connection to a model, meaning which server, which model and with which parameters.
- Signature: a description of the task, meaning which fields come in, which have to be returned and what they mean. It replaces the prompt text.
- Module: a way to solve the task a signature describes.
Predictsimply asks the model,ChainOfThoughtasks it to reason first,ReActturns the model into an agent that calls tools. - Predictor: one model call inside a module, with its own signature, instruction and examples. A simple module has one, an agent has several. Optimizers tune predictors.
- Adapter: the translator between a signature and the model. It turns a signature into chat messages and the model's answer back into Python fields.
- Example (
dspy.Example): one record of the data set, a question and its correct answer. - Metric: an ordinary Python function that says how good the program's answer to one example is.
- Optimizer: an algorithm that runs the program on the examples over and over and changes the instructions and examples in the prompts (some also change the model's weights) to raise the metric. Older versions called optimizers teleprompters, and the word still shows up in the code.
This is how they connect. A module executes a signature. The adapter turns it into messages for the LM and parses the answer. The metric scores the result on the examples. The optimizer rewrites instructions and picks examples inside the predictors until the metric stops growing.
Connecting a model
How to install DSPy and connect a model
DSPy installs as an ordinary Python package and runs on Python 3.10 through 3.15:
pip install -U "dspy[optuna]"The optuna extra is needed by the MIPROv2 optimizer. Since version 3.2 this library is not installed by default, and without it MIPROv2 fails with an import error. The GEPA optimizer comes with DSPy, except on Python 3.15: its gepa package supports only versions up to 3.14 so far.
Now let's connect a model. In the example it is Gemma 4 E4B running in LM Studio on the same machine:
import dspy
# The openai/ prefix means "a server with the same API as OpenAI",# followed by the name of the model on that server.lm = dspy.LM( "openai/google/gemma-4-e4b", api_base="http://localhost:1234/v1", api_key="lm-studio", # LM Studio does not check the key, but the field is required temperature=0,)# From now on every module uses this model unless told otherwise.dspy.configure(lm=lm)A model is given as a "provider/model" string. For a cloud model only the string changes, for example openai/gpt-5.4-nano, and there is no server address to set. The key comes from the provider's environment variable, such as OPENAI_API_KEY. Under the hood DSPy reaches models through the LiteLLM library and, since version 3.4, through its own lm15 engine wherever it can. Both read the same model string, so the code does not depend on the engine.
temperature=0 almost removes randomness: the model picks the most likely answer. The answers will still not be exactly the same (in our runs the agent sometimes searched with different words when asked the same question again), but the spread gets smaller. That is convenient for an agent and for an honest before-and-after comparison.
Why a repeated run is almost free
DSPy remembers the model's answers. The cache lives in memory and on disk, in the ~/.dspy_cache folder, and takes up to 30 GB by default. If a request repeats exactly (the same prompt, the same model, the same parameters), it does not go to the model, and the answer comes from the cache.
For optimization this is a big saving: an interrupted run can be started again, and everything already computed repeats instantly. But the cache gets in the way of measuring speed and checking how stable the model is: the second run turns out suspiciously fast and exactly the same as the first. For such measurements the cache is turned off with cache=False in dspy.LM.
Signature
What a signature is
A signature describes a task like a function: what goes in and what comes out. The shortest form is a string with the inputs and outputs separated by an arrow:
qa = dspy.Predict("question -> answer")result = qa(question="What is the capital of France?")print(result.answer) # ParisPredict is the simplest module: one model call with no reasoning and no tools. The result is a Prediction object whose fields are named after the signature's outputs.
What DSPy sends to the model
We did not write a prompt, but the model received one. One line shows exactly what went to the model and what came back:
dspy.inspect_history(n=1) # n is how many recent calls to showSystem message:
Your input fields are:1. `question` (str):Your output fields are:1. `answer` (str):All interactions will be structured in the following way, with the appropriate values filled in.
[[ ## question ## ]]{question}
[[ ## answer ## ]]{answer}
[[ ## completed ## ]]In adhering to this structure, your objective is: Given the fields `question`, produce the fields `answer`.
User message:
[[ ## question ## ]]What is the capital of France?
Respond with the corresponding output fields, starting with the field `[[ ## answer ## ]]`, and then ending with the marker for `[[ ## completed ## ]]`.
Response:
[[ ## answer ## ]]Paris
[[ ## completed ## ]]This is the work of ChatAdapter, the adapter DSPy uses by default. In the system message it lists the fields, shows the format and states the objective. So far the objective is built from the field names, because we gave no instruction. The user message holds the values of the input fields. Markers like [[ ## answer ## ]] are there to find the fields in the answer later: the adapter looks for them and splits the text into fields.
If the model breaks the format and the adapter cannot parse the answer, ChatAdapter repeats the request through JSONAdapter on its own. That adapter asks for the answer as JSON and, if the server supports it, turns on strict JSON mode. There are other adapters too, such as XMLAdapter. To switch the adapter for the whole program: dspy.configure(adapter=dspy.JSONAdapter()).
Small models sometimes break the format without anyone noticing. In one of our runs Gemma 4 E4B did not put a line break before the final marker, and the last field arrived with a tail: $15[[ ## completed ## ]]. There was no error, just junk in the answer. So the first thing to do with a new model is to look at inspect_history with your own eyes.
How to write a signature as a class
For a real task the signature is written as a class. Then the fields get types and descriptions, and the class docstring becomes the instruction for the model:
from typing import Literal
class ClassifyTicket(dspy.Signature): """Classify a customer support ticket."""
text: str = dspy.InputField() # Literal limits the answer to a list of values, bool to True and False. topic: Literal["delivery", "refund", "payment", "other"] = dspy.OutputField() urgent: bool = dspy.OutputField(desc="the customer is blocked or very upset")
classify = dspy.Predict(ClassifyTicket)result = classify(text="My parcel is two weeks late and I need it for a birthday tomorrow!")print(result.topic, result.urgent) # delivery Trueresult.urgent here is a real bool, not the string "True": the adapter converts the answer to the field's type. int, float, list[str], dict, Literal and Pydantic models all work. If the answer cannot be converted to the type, the adapter reports an error instead of returning junk.
For the model to know about the constraints, the adapter writes them into the prompt. This is how the fields of this signature look in the system message:
[[ ## topic ## ]]{topic} # note: the value you produce must exactly match (no extra characters) one of: delivery; refund; payment; other
[[ ## urgent ## ]]{urgent} # note: the value you produce must be True or FalseThe docstring, Classify a customer support ticket., became the objective in the system message instead of the faceless "Given the fields...". This is the instruction the optimizer will rewrite later. It does not touch field descriptions (desc), so they are the place for whatever must stay unchanged.
We write docstrings and field descriptions in English: this is text for the model, not for the programmer. For small models English is usually the more reliable choice.
How Predict differs from ChainOfThought
ChainOfThought takes the same signature but adds one more field in front of the outputs, reasoning. The model writes its reasoning first and then the answer:
think = dspy.ChainOfThought(ClassifyTicket)result = think(text="Can I pay with Apple Pay?")print(result.topic, result.urgent) # payment Falseprint(result.reasoning) # The customer is asking a question regarding accepted methods of payment...The signature did not have to change. That is the point of modules: the task is described once, and the way to solve it changes in one line. Reasoning helps with tasks where something has to be counted or compared, but it makes the answer longer and the call slower.
An agent with tools
How to prepare the knowledge base
An agent needs a source of facts. In the example it is the shop's knowledge base in SQLite: twelve short articles on returns, delivery, payment, warranty and so on. Here are the first four and the table setup:
import sqlite3
ARTICLES = [ ("Refunds", "You can return an item within 30 days of delivery for a full refund. " "The money goes back to the original payment method within 5 business days " "after the warehouse receives the item."), ("Return shipping", "Return shipping is free for defective items. For any other return, " "a flat fee of $6 is deducted from the refund."), ("Delivery times", "Standard delivery takes 3 to 5 business days. Express delivery takes " "1 to 2 business days and costs $12."), ("Payment methods", "We accept Visa, Mastercard, PayPal and Apple Pay. " "Cash on delivery is not available."), # Eight more articles in the same shape: free shipping, shipping abroad, # warranty, order changes, gift cards, loyalty points, price match, support.]
# FTS5 is SQLite's built-in full-text search. A table like this# stores the text and builds a word index over it right away.db = sqlite3.connect("kb.db")db.execute("DROP TABLE IF EXISTS articles")db.execute("CREATE VIRTUAL TABLE articles USING fts5(title, text)")db.executemany("INSERT INTO articles VALUES (?, ?)", ARTICLES)db.commit()db.close()FTS5 is usually already part of the SQLite that ships with Python, so there is nothing to install. Full-text search is needed because the model writes the query to the database, and it phrases it in its own words.
How to describe a tool
A tool for DSPy is an ordinary Python function. All that matters is that it has a clear name, type hints on its arguments and a docstring: DSPy shows all of them to the model, and the model uses them to decide when and how to call the function.
import refrom contextlib import closing
def search_kb(query: str) -> list[str]: """Search the shop's knowledge base and return up to 3 matching articles.""" # 1. Keep only the words of the query: FTS5 would take quotes # and asterisks for its own syntax. words = re.findall(r"\w+", query) if not words: return [] # 2. OR: an article matches if it has at least one word of the query. # bm25 ranks articles with more matches higher. match = " OR ".join(f'"{word}"' for word in words) # closing closes the connection: a plain with on sqlite3 manages only the transaction. with closing(sqlite3.connect("kb.db")) as db: rows = db.execute( "SELECT title, text FROM articles WHERE articles MATCH ? " "ORDER BY bm25(articles) LIMIT 3", (match,), ).fetchall() # 3. The model gets a list of "title: text" strings. return [f"{title}: {text}" for title, text in rows]
print(search_kb("cash on delivery")[0]) # Payment methods: We accept Visa, Mastercard, PayPal and Apple Pay. ...Why not LIKE '%...%'? The model makes up the query itself, for example refund period. LIKE looks for that phrase as a whole, while in the article the words "refund" and "period" stand apart or are missing altogether. The agent would get an empty list and conclude that the knowledge base has no answer. Word search with ranking forgives someone else's wording. In a real project this is usually vector search, covered in the article on vector databases, but for DSPy that changes nothing: the tool is still a function.
This is how DSPy describes the tool to the model. It is a fragment of the agent's prompt, covered below:
(1) search_kb, whose description is <desc>Search the shop's knowledge base and return up to 3 matching articles.</desc>. It takes arguments {'query': {'type': 'string'}}.(2) finish, whose description is <desc>Marks the task as complete. That is, signals that all information for producing the outputs, i.e. `answer`, are now available to be extracted.</desc>. It takes arguments {}.The function name, the docstring and the argument schema come from the code. So a poor name like f1 or an empty docstring hurts the agent just as a badly written prompt would. We did not declare the finish tool: ReAct adds it itself, so that the model can say "I have enough information".
How to build the agent
The agent's signature describes the whole task without mentioning tools:
class AnswerFromKb(dspy.Signature): """Answer the customer's question using only the shop's knowledge base."""
question: str = dspy.InputField() answer: str = dspy.OutputField(desc="a short answer, no explanations")
agent = dspy.ReAct(AnswerFromKb, tools=[search_kb], max_iters=5)result = agent(question="How long is the refund period?")print(result.answer) # You can return an item within 30 days of delivery for a full refund.max_iters limits the number of steps: if the model never says finish, the agent stops after the fifth tool call. The default is twenty steps.
The answer is correct but long, even though the field description asked for a short one with no explanations. A small model often skips such requests. We will come back to this when we write the metric: this is exactly the kind of thing optimization fixes.
What happens inside ReAct
ReAct (from reasoning and acting) is a loop in which the model takes turns thinking and calling tools. In DSPy 3.4 it works like this:
- The model gets the question, the list of tools and the history of previous steps. On the first step the history is empty.
- The model writes a thought (
next_thought), picks a tool (next_tool_name) and the arguments for it (next_tool_args). - DSPy calls the function and appends its result (
observation) to the history. - Steps 2 and 3 repeat until the model picks
finishormax_itersruns out. - A separate model call,
extract, reads the whole history and fills in the signature's output fields, in our caseanswer. It works likeChainOfThought, so it also reasons first.
So there are two predictors inside the agent: react picks the steps and extract writes the answer. Each has its own instruction and its own examples, and the optimizer tunes both. This will matter when we look at the result of the optimization.
The history of steps is in the trajectory field (long lines in the output are shortened):
for key, value in result.trajectory.items(): print(f"{key}: {value}")thought_0: The user wants to know the duration of the refund period. I need to search the knowledge base using relevant keywords like "refund period" or "return policy" to find this information.tool_name_0: search_kbtool_args_0: {'query': 'refund period'}observation_0: ['Price match: If you find a lower price at another official retailer within 14 days of purchase, we refund the difference.', 'Return shipping: ...', 'Refunds: You can return an item within 30 days of delivery for a full refund. ...']thought_1: The previous search provided a clear answer in observation [3]. It states: "You can return an item within 30 days of delivery for a full refund." ... I will finish the task.tool_name_1: finishtool_args_1: {}observation_1: Completed.The model searched for refund period, and the right article came third: the word "refund" appears in three articles, and "period" in none. The model still found the answer and finished. The three model calls took about nine seconds.
Two more things ReAct does on its own. If a tool raises an exception, the agent does not crash: the error text goes into the history instead of the result, and the model can try another way. If the history no longer fits the model's context, ReAct drops the oldest steps and tries again, up to three times.
What changes in DSPy 3.5
In DSPy's main branch, which will become version 3.5, ReAct has already been rewritten. The new implementation also exists in 3.4 as dspy.ReActV2, marked experimental. The differences:
- Previous steps reach the model as a real conversation with function calls, and if the adapter supports it, the provider's built-in function calling is used. One step can call several tools.
- Instead of
finishand a separateextractstep, the model calls asubmittool and passes the final output fields in it directly. The agent has a single predictor. - Instead of
trajectory, the result will havehistoryandtermination_reason, the reason the loop stopped.
The code in this article survives the upgrade, except for the loop over trajectory above. A saved optimization result does not: the new agent has different predictors, and the tuning has to be run again. The 3.4 documentation recommends taking dspy.ReActV2 for new agents already, if you are fine with an experimental API and have pinned the DSPy version in your dependencies. In 3.5 the ReActV2 name stays as a deprecated alias, and 3.6 removes it.
Data
What the data set looks like
The data set is a list of questions with correct answers. The simplest place to keep it is JSON:
[ { "question": "How long is the refund period?", "answer": "30 days" }, { "question": "How much does it cost to return a jacket that I just didn't like?", "answer": "$6" }]DSPy works not with dictionaries but with dspy.Example objects:
import json
with open("dataset.json", encoding="utf-8") as f: examples = [dspy.Example(**row).with_inputs("question") for row in json.load(f)]
print(len(examples)) # 40print(examples[0].inputs()) # Example({'question': 'How long is the refund period?'}) (input_keys={'question'})print(examples[0].labels()) # Example({'answer': '30 days'}) (input_keys=None)with_inputs("question") says which fields of the example are the program's inputs. Only they reach the agent when it runs. The remaining fields, here answer, stay correct answers that the metric will see. Forget with_inputs, and DSPy will not know what to pass in: every run ends with the error "Inputs have not been set for this example". On a set of fewer than ten examples the evaluation simply shows zero. With ten or more, dspy.Evaluate gives up at the tenth error and raises "Execution cancelled due to errors or interruption", and the real cause is visible only in the ERROR lines above it.
How many examples you need
The earlier DSPy documentation gave these guidelines: thirty examples often already bring substantial value, but you should aim for at least three hundred, and for MIPROv2 two hundred or more, so that the optimizer does not overfit. The 3.4 documentation, rewritten around GEPA, no longer has these pages; their text survives only in the repository. The new documentation gives no universal numbers.
Variety matters more than quantity. Our set has forty questions, and it includes:
- the same facts asked in different words: "How long is a gift card valid?" and "When does a gift card expire?";
- two-step questions where the agent has to find something and calculate: "If I return a sweater I don't like, how much of its $40 price do I get back?", with $34 as the correct answer;
- questions where the model has to work out which article fits, even though the question uses different words.
A real set should also have questions the knowledge base has no answer to: the agent must honestly say "I don't know" instead of inventing a rule. Forty examples are too few for real tuning, but just right for a demonstration run on a local model: one MIPROv2 run on them took a little over an hour.
How to split the data
The data is split into three parts, and each has its own role:
- training (train): the optimizer takes examples for the prompt and material for new instructions from it;
- validation (val): the optimizer compares candidates on it and picks the best one;
- test: the optimizer never sees it, and the final score is computed on it.
Without a test set the result is misleading. The optimizer chose the prompt that did best on the validation set, so the score on it is always inflated. Only a set it has not seen gives an honest picture.
The proportions depend on the optimizer. For most prompt optimizers, MIPROv2 included, the earlier documentation recommended an unusual split: twenty percent for training and eighty for validation. Prompt optimizers easily adapt to a small training set, and a stable score on a large validation set matters more. For GEPA it is the other way round; we will come back to it below. The split for MIPROv2:
import random
# Shuffle with a fixed seed so that the split is reproducible.random.Random(0).shuffle(examples)
# 1. Put a fifth aside for the final check.test_size = len(examples) // 5testset, rest = examples[:test_size], examples[test_size:]
# 2. The rest for MIPROv2: 20% for training, 80% for validation.train_size = len(rest) // 5trainset, valset = rest[:train_size], rest[train_size:]
print(len(trainset), len(valset), len(testset)) # 6 26 8Metric
What a metric is
A metric is a function that receives an example with the correct answer and the program's answer and returns a score: True/False or a number from 0 to 1. What counts as a good answer is up to us. Here an answer is good if it contains the correct value and is short, as we asked in the signature:
def metric(example, prediction, trace=None): answer = prediction.answer.lower() # 1. The answer contains the correct value, for example "30 days". correct = example.answer.lower() in answer # 2. The answer is short: no longer than ten words. short = len(answer.split()) <= 10 return correct and shortThe third argument, trace, is the history of calls inside the program. During ordinary evaluation it is None. The optimizer passes it when it decides whether a successful run is good enough to become an example in the prompt, and at that moment it needs a strict yes or no. Our metric already returns a bool, so it can ignore trace. If a metric returns a fractional score, trace is used to turn it into a yes or no: for example, return score >= 0.8 if trace is not None else score.
How to score the agent
dspy.Evaluate scores a program on a data set. It runs the program on every example, in several threads in parallel, and computes the average:
evaluate = dspy.Evaluate(devset=testset, metric=metric, num_threads=4)report = evaluate(agent)print(report.score) # 50.0report.score is the percentage of good answers. report.results holds "example, answer, score" triples, which are handy for going through the failures:
for example, prediction, score in report.results: if not score: print(example.question, "->", prediction.answer)On the test set the agent got half of the questions right. On all forty the result is nearly the same, 57.5%, and the full run took eight minutes. The failures there are of three kinds:
How long is a gift card valid? -> Gift cards are valid for 3 years from the date of purchase.What is the price of express shipping? -> The knowledge base confirms that express delivery is never free, but it does not provide the specific price.How many loyalty points do I earn per dollar? -> One point per dollarThe first is the most common: the answer is right but longer than ten words. The second is a real mistake: the article with the price was in the knowledge base, but the model did not find it. The third is the metric's mistake: "One point" means the same as "1 point", but the substring did not match. Such false failures spoil the optimization: the optimizer will diligently teach the model to write in digits what was correct in words too. So before tuning a program, look through the failures yourself and make sure the metric punishes exactly what it should.
When you need an LLM judge
A substring check works well for short facts. For free-form answers, where one thought can be put a hundred ways, it is replaced by a judge: another DSPy program that compares the answer with the reference by meaning.
class JudgeAnswer(dspy.Signature): """Decide whether the answer gives the same facts as the reference answer."""
question: str = dspy.InputField() reference: str = dspy.InputField() answer: str = dspy.InputField() is_correct: bool = dspy.OutputField()
judge = dspy.ChainOfThought(JudgeAnswer)
def judge_metric(example, prediction, trace=None): verdict = judge(question=example.question, reference=example.answer, answer=prediction.answer) return verdict.is_correctA judge is expensive: every score is now one more model call, and an optimization run makes thousands of them. A judge also makes mistakes, so it gets a stronger model than the one it scores. A separate model for the judge is set with dspy.context(lm=...) around the call. DSPy also has ready-made metrics: dspy.evaluate.answer_exact_match compares answers after normalizing the text, and dspy.evaluate.SemanticF1 is a judge that measures, by meaning, how fully the answer covers the reference and whether it adds anything extra. They expect fields with particular names, so writing your own ten-line metric is often simpler.
Optimization
Which optimizers there are
An optimizer can change three things: instructions, the examples in the prompt (called demonstrations, or few-shot examples) and the model's weights. Each one works with one or two of them:
| Optimizer | What it changes | When to use it |
|---|---|---|
LabeledFewShot | examples from the data as they are | an honest baseline, makes no model calls |
BootstrapFewShot | examples from successful runs of the program | a sensible first step, almost always better than no examples |
BootstrapFewShotWithRandomSearch | the same, many sets, the best on validation | when a single set of examples gives unstable results |
MIPROv2 | instructions and examples together | when both the wording and the examples need tuning |
GEPA | instructions only, from the metric's text feedback | when the metric can explain a mistake in words and there is a strong model for the analysis |
SIMBA | rules in the instruction and examples for the hardest cases | when the program fails on particular kinds of questions |
COPRO | instructions only, by brute force | when the examples are already good and only the wording is in the way |
BootstrapFinetune | the model's weights | the last lever: needs a model that can be fine-tuned |
BetterTogether | a chain of prompt optimizers and fine-tuning | when prompts and fine-tuning have already been tried separately |
DSPy does not pick an optimizer for you. The documentation advises starting simple: if LabeledFewShot or BootstrapFewShot already gives the result you need, heavy optimization is wasted money. Fine-tuning the weights is the last resort: it costs far more, and its gain over GEPA or MIPROv2 is often small. Mind the price as well: according to the documentation, a single GEPA or MIPROv2 run on cloud models can cost hundreds of dollars.
How MIPROv2 works
MIPROv2 (Multiprompt Instruction Proposal Optimizer, version 2) tunes instructions and examples at the same time. It works in three steps:
- Collecting examples. The optimizer runs the program on the training set and keeps the runs the metric accepted. They become example sets for each predictor, with every intermediate step: which tool was called and what it returned.
- Proposing instructions. An "author" model gets a short description of the data, the program's code, the collected examples and a random tip such as "be concise" or "be creative", and writes several versions of the instruction for each predictor.
- Searching for the best combination. Bayesian search (the TPE algorithm from the Optuna library) tries combinations of "instruction plus example set" for all predictors at once. It remembers which choices score high and tries similar ones more often.
This is how it runs:
evaluate = dspy.Evaluate(devset=testset, metric=metric, num_threads=4)print("before:", evaluate(agent).score) # before: 50.0
optimizer = dspy.MIPROv2(metric=metric, auto="light", num_threads=4)tuned = optimizer.compile(agent, trainset=trainset, valset=valset)
print("after:", evaluate(tuned).score) # after: 87.5tuned.save("agent.json")compile does not change the original agent; it returns a tuned copy. That is why agent and tuned can be compared on the same test set.
On the test set the share of good answers went from 50 to 87.5%: the agent now handles seven of the eight questions. On the validation set the best candidate scored the full 100% against the original 57.7%. The gap between 100 and 87.5 is exactly the inflated score discussed in the section on splitting the data: the candidate was picked on the validation set, so it looks better there than on questions it has not seen.
The answers became what the signature asked for:
How long is the refund period? -> 30 daysHow long after placing an order can I cancel it? -> Within 1 hourHow long does delivery to Canada take? -> 7 to 14 business daysWhen can I reach customer support? -> By chat, from 9:00 to 21:00 CET, seven days a week.The last answer is correct, but it has eleven words, and the metric did not count it.
What exactly changed is covered below, in the section on the result. In short: both predictors got new instructions of over two thousand characters each. For the extract step this is a jump from a single line, while the react instruction was long from the start, about 1,200 characters, because ReAct puts the description of the tools into it; the optimizer rewrote it completely. Each predictor also got four examples. The first comes from a successful recorded run of the agent: for extract it is the whole search history with the final answer, for react only the first step, the question and the first tool call. The other three are questions from the training set with correct answers such as "$6" and "free". Only extract sees them: they contain none of the fields the react step produces, so the adapter leaves them out of its prompt, and react in effect works with a single example.
The trial log shows what worked. In two trials the react step is set up identically, while the extract step has no examples and only its instruction changes. With the old instruction the validation score is 53.8%, with the new one 88.5%. The best result came from the new instruction combined with examples.
The run took 65 minutes on a laptop graphics card (RTX 5070 Ti): 14 minutes to collect examples and instructions and about 50 for the twenty trials. The agent ran 555 times in total. The timing is not a benchmark: the card was busy with other work as well. The instruction versions were written by the same Gemma 4 E4B, since we did not set a separate "author" model.
What a MIPROv2 run costs
The auto parameter sets the budget: light, medium or heavy. In light mode MIPROv2 collects six example sets for our agent, prepares three instruction candidates for each of the two predictors, the original one and two new ones, and makes twenty trials. The number of trials comes from a formula: two variables per predictor (instruction and examples), four in total, which gives 2 × 4 × log₂ 6 ≈ 20 trials.
How each trial is scored depends on the size of the validation set. light mode takes at most a hundred examples from it. If there are more than fifty, each trial is checked on a random batch of 35 examples, and every five trials the best candidate gets a full check. If there are fifty or fewer, as in our case, each trial is checked on the whole set.
Then it is simple arithmetic. On a set of three hundred examples (60 test, 48 training, 192 validation), MIPROv2 in light mode runs the agent about 1,230 times. We counted that with a fake model that answers instantly and caches nothing. Each agent run is at least three model calls: the search, finish and the answer. That makes about four thousand calls. With a real model some repeats come from the cache, but on a cloud model the rest is money spent.
Two things lower the price noticeably. The first is splitting the models. The instruction versions are written by prompt_model, and the program runs on task_model. The "author" is called rarely, so it can be a stronger and pricier model, while the program stays on a cheap one:
optimizer = dspy.MIPROv2( metric=metric, auto="light", prompt_model=dspy.LM("openai/gpt-5.4"), # writes the instruction versions task_model=lm, # runs the agent num_threads=4,)The second is the cache mentioned above. Restarted after an interruption, a run goes through the part already computed instantly and for free.
How GEPA works
GEPA (from Genetic-Pareto) is the optimizer the introduction in the DSPy documentation is now built around. It changes only instructions, and it does so differently from MIPROv2: instead of sorting through ready-made versions, it reads how the program went wrong and rewrites the instruction so that the mistake does not happen again. The loop looks like this:
- GEPA takes one of the best candidates and runs it on three examples from the training set.
- The metric scores the answers and explains in words what is wrong.
- A strong model (
reflection_lm) reads the program's steps, the answers and the explanations, and writes a new instruction for one of the predictors. - If the same examples go better, the new candidate is checked on the whole validation set and joins the pool.
- The pool holds the candidates that are best on at least one example; this is the "Pareto front" from the name. Now and then GEPA merges two candidates that are strong on different examples.
The key difference is the metric. It can return not just a score but also feedback, and the feedback goes word for word to the model that writes the instruction. A metric for GEPA has five arguments: besides the usual ones, it gets the name of the predictor being tuned and its part of the call history.
def gepa_metric(gold, pred, trace=None, pred_name=None, pred_trace=None): answer = pred.answer.lower() if gold.answer.lower() not in answer: return dspy.Prediction(score=0.0, feedback=f"Wrong. The correct answer is {gold.answer}.") words = len(answer.split()) if words > 10: return dspy.Prediction(score=0.5, feedback=f"Correct, but {words} words long. Use at most 10.") return dspy.Prediction(score=1.0, feedback="Correct and short.")The metric may return a plain number too, but then GEPA sees only "This trajectory got a score of 0.5", and its main advantage is gone. The feedback is in English for the same reason as the instructions: a model reads it.
The split for GEPA is the opposite of the one for MIPROv2. Every new candidate is checked on the whole validation set, so a large validation set eats the budget. The documentation advises keeping it as small as possible, as long as it resembles real questions, and putting everything else into training:
# Same test set. The rest is split the other way round:# three quarters for training, a quarter for validation.val_size = len(rest) // 4gepa_train, gepa_val = rest[:-val_size], rest[-val_size:]
optimizer = dspy.GEPA( metric=gepa_metric, reflection_lm=dspy.LM("openai/gpt-5.4", temperature=1.0, max_tokens=32000), auto="light", num_threads=4,)tuned_gepa = optimizer.compile(agent, trainset=gepa_train, valset=gepa_val)print("after GEPA:", evaluate(tuned_gepa).score)What this looks like in practice is shown by our cut-down run on the same forty questions: 24 in the training set, 8 in the validation set, a budget of max_metric_calls=200 instead of auto="light", and the same Gemma 4 E4B as reflection_lm instead of a strong cloud model. The run broke off at the seventh iteration, so it has no final score, but the first minutes of its log show the mechanics well:
Iteration 0: Base program full valset score: 0.6875 over 8 / 8 examplesIteration 1: Selected program 0 score: 0.6875Iteration 1: Proposed new text for react: You are an intelligent Customer Service Agent whose sole purpose is to answer customer queries using information exclusively sourced from the shop's Knowledge Base (KB). ...Iteration 1: New subsample score 2.0 is not better than old score 2.5, skippingIteration 2: Selected program 0 score: 0.6875Iteration 2: All subsample scores perfect for parent 0. Skipping.Iteration 3: Selected program 0 score: 0.6875Iteration 3: Proposed new text for extract.predict: You are an expert Customer Service AI agent tasked with answering customer inquiries using only the provided shop Knowledge Base (KB). ...Iteration 3: New subsample score 1.5 is not better than old score 2.0, skippingThe original agent scored 0.6875 on validation: that is the average of our metric, where a long correct answer is worth 0.5. On the first iteration the reflection model rewrote the instruction of the react step. On the same three examples the new candidate scored 2.0 in total against the previous 2.5, and GEPA threw it away without spending a check on the validation set. On the second iteration all three sampled examples were already solved perfectly, there was nothing to learn from, and GEPA skipped the turn. Before the run stopped, it proposed a new instruction four times, twice for each predictor, and none passed the check on three examples. This is how GEPA saves its budget: only candidates that have already proven themselves on a small batch get the expensive check on the whole validation set. The reflection model matters a lot too. In our run it was the same small Gemma, and none of its instructions turned out better than the original. No wonder the documentation recommends a strong model for this role.
GEPA also has auto="light", and it prints the budget when it starts: "approx N metric calls of the program". Despite the wording, these are program runs, one per example checked. For our agent and three hundred examples split as in the code above (60 test, 180 training, 60 validation) it is about 1,090 runs. The metric function itself is called more often, about 1,550 times: GEPA calls it a second time for feedback on a particular predictor, and these calls do not count against the budget. Instead of auto you can set the budget directly: max_metric_calls in the same program runs, or max_full_evals as a number of full passes over the sets.
What to know about GEPA before running it:
reflection_lmis required. Since version 3.4.1 it has to be a DSPy model that can answer in JSON. The documentation recommends a strong model: it is called rarely, a few dozen times per run.- GEPA is still marked experimental, and it does not install on Python 3.15.
- GEPA does not touch the examples in the prompt. If you need them too, GEPA is combined with
BootstrapFewShotthroughBetterTogether. - With
log_dirset, GEPA saves every candidate there and can resume an interrupted run from the same point.
How to find out what was spent
DSPy can count tokens. Tracking is switched on for the duration of a call through dspy.context:
with dspy.context(track_usage=True): result = agent(question="Do you ship to Canada?")
print(result.get_lm_usage()) # {'openai/google/gemma-4-e4b': {'prompt_tokens': 2019, 'completion_tokens': 605, ...}}The agent made three model calls: 2,019 tokens went out in the requests and 605 came back in the answers. 351 of those 605 are hidden reasoning: Gemma 4 thinks to itself first and only then answers, so each of its steps is noticeably slower than such a short answer suggests. Answers from the cache are not counted: if the question was asked before, the dictionary comes back empty.
The numbers are collected per model. For a cloud model the price of every call is in the model's history, lm.history. The detailed course of an optimization (candidates, scores, prompts) can be sent to MLflow: call mlflow.dspy.autolog(log_compiles=True, log_evals=True, log_traces_from_compile=True) before the run. Called without arguments, it records neither the optimization nor the evaluation results: these three flags are off by default.
The result
How to save and load the tuned agent
tuned.save("agent.json") writes what the optimizer picked: the instructions and examples of every predictor, plus the Python and DSPy versions the tuning ran on. The program's code does not go into the file. So when loading, the agent is first created the same way as for tuning, and then the saved state is applied to it:
agent = dspy.ReAct(AnswerFromKb, tools=[search_kb], max_iters=5)agent.load("agent.json")print(agent(question="How long is the refund period?").answer) # 30 daysThe file is small and human-readable, and it is convenient to keep in the repository next to the code: changes to the instructions show up in an ordinary diff. It never contains API keys. On load DSPy also drops server addresses unless you explicitly pass allow_unsafe_lm_state=True. The model the agent will run on is set as usual, with dspy.configure. If the file was saved by another version of DSPy, loading prints a warning, but the agent loads.
If the side that loads the agent does not have your code, there is a second way: tuned.save("agent/", save_program=True) saves the whole program with cloudpickle, and dspy.load("agent/", allow_pickle=True) restores it. Without your code this works only for classes defined in the script that did the saving: cloudpickle stores those in full, but for a class from an imported module it stores only a reference, and loading without that module fails with ModuleNotFoundError. If the signature and the tools live in your own module, list it when saving: tuned.save("agent/", save_program=True, modules_to_serialize=[my_module]). Such a file contains executable code, so load it only from a source you trust.
How to see what the optimizer changed
The instructions and examples of every predictor are available right from the code:
for name, predictor in tuned.named_predictors(): print(name, "| demos:", len(predictor.demos)) print(predictor.signature.instructions)This is what came out of our MIPROv2 run (long passages skipped):
react | demos: 4You are a highly specialized AI Agent Planner (Reasoning Engine). Your sole function is to analyze a customer's question and the entire conversation history (`trajectory`) accumulated so far to determine the single, most logical next step needed to find the answer using only the available tools....**Tool Definitions:**1. **`search_kb`**: Use this when you need to actively retrieve new information from the knowledge base. The argument must be a specific, focused search query (e.g., `'return period'`). Do not use general terms; focus on the core concepts of the question or observations.2. **`finish`**: Use this only when your reasoning confirms that you have gathered sufficient evidence across all parts of the trajectory to formulate a definitive answer, and no further tool calls are required....extract.predict | demos: 4You are a sophisticated AI knowledge synthesis engine responsible for compiling the final answer to a customer query based *exclusively* on an agent's complete investigative history (the 'trajectory'). ......2. **Answer (Final Output):** * Provide a final answer that is extremely concise, direct, and factual. * The `answer` must directly address the initial `Question`. * Do not include explanations, filler words, or unnecessary introductory phrases in the final `answer`. ...A model wrote these instructions, and it shows: they are long, with headings and repetition. But the instruction of the extract step says outright that the answer must be extremely short and without introductions, which is exactly what the metric was taking points off for.
Look at the react step. Its original instruction contained the list of tools that ReAct built from the code: names, docstrings, argument schemas. The optimizer rewrote the instruction completely, and now the tools are described in its own words. The agent works, but if you add a tool to the code or rename an argument, the saved instruction will not know about it, and the tuning has to be repeated.
The same is in agent.json. It is worth reading after every tuning run. The optimizer fits the prompt to the metric, not to your intentions: if the metric is wrong somewhere, it shows up in the instruction first.
When to run the optimization again
Tuning is tied to the model, the data and the code. It is run again when:
- the model changed, or its version at the provider did;
- the tools, the signature or the structure of the program changed;
- the knowledge base or the users' questions now have something the data did not;
- DSPy was updated in a way that changes the predictors, as will happen with
ReActin 3.5.
Each time, compare the score on the test set before and after. If the new model is no better than the old one after tuning, there is no need to switch: that decision is also made by the numbers, not by impression.
What else DSPy has
A lot of useful things stayed outside the example:
dspy.Refineanddspy.BestOfNrun a module several times and pick the answer that passed the check best.dspy.RLMis a module for tasks with a huge context. The data sits in variables of a Python interpreter, and the model writes code to look through it and calls a model on the pieces it needs. It replacesCodeActandProgramOfThought, which 3.5 removes.- Modules can be called asynchronously with
acall, stream their answer piece by piece throughdspy.streamify, and take tools from MCP servers throughdspy.Tool.from_mcp_tool. - Modules compose into programs like layers in PyTorch: your own class, a subclass of
dspy.Module, calls several modules inforward, and the optimizer tunes all of them at once.
When you don't need DSPy
DSPy pays off when a task repeats, can be measured, and the model will change. It is unnecessary if:
- there are no examples with correct answers and no way to score an answer. Without them you only get a convenient way to write calls, and the main thing, optimization, is out of reach;
- the task is a one-off, or the prompt already works;
- the prompt text has to be exactly what a person wrote, for example because of legal wording;
- you need full control over every token of the request: the prompt an adapter builds is longer and more templated than a hand-written one.
Keep the limitations in mind even when DSPy fits. A prompt tuned for one model does not carry over to another. On a small data set the optimizer overfits, and without a separate test set you will not see it. The API changes from version to version, so pin the DSPy version in your dependencies and run the evaluation again after every upgrade.