Self-improving agents: RL on the config, not the weights

Say “self-improving agent” in a meeting and you get one of two pictures: science fiction — an agent rewriting its own source at 3am — or a training run, with rollout farms and a GPU bill. I mean neither. There’s a third door, and it’s been standing open in my harness for months.

The harness article ended on a provocation: the operations stack — streaming, tracing, feedback — had quietly produced everything a learning system needs. Behavior: every run persisted as a lossless trace. Cost: priced per span. Judgment: thumbs keyed to the exact trace they judge. Add the eval service and you have a measurable objective. Add the config and you have parameters.

Read that list again with an RL hat on. An objective — the eval score. A signal — trace-keyed feedback, fresh every day. A parameter space — the config: system prompts, skill instructions, memory. An environment — production traffic. That is a reinforcement-learning setup in everything except the part people assume: nothing in it touches model weights.

This article is the full design for the step that closes the loop. I call it dreaming: a nightly job that replays the day’s trajectories, distills what worked and what failed into durable memories and sharper skill instructions, and ships every improvement as a reviewable diff. It is the part of my harness I’m building right now, and this is the design I’m building it to.

The objective: make the reward speak

Every learning loop starts with a number to climb. Mine comes from the eval service in the harness article: golden datasets, experiments, pass rates. Formally:

score = w₁·accuracy + w₂·relevancy + w₃·safety + …

Each dimension is an eval metric run over golden cases. The weights are not a technical artifact — they are product priorities made explicit. A support agent might carry safety at 0.5; an internal research tool might not. The weights get argued about in review, which is exactly where that argument belongs. And the goldens themselves come from the loop the harness already runs: seeded by hand, grown from production traces and thumbs-down feedback — so the objective stays anchored to what real users actually asked, not to what a benchmark imagines they might.

For a while I assumed the score was enough. The result that changed my mind is GEPA (ICLR 2026 Oral), whose central observation is that a scalar reward starves the optimizer: a 0.4 says that you failed and nothing else. GEPA’s eval metrics return text alongside the number — which cases failed, what the judge disliked, what pattern connects the failures — and a reflection model reads that feedback, plus the execution traces, before proposing the next candidate. Their thesis: language is a richer learning medium than a sparse scalar. Their headline result (below) suggests the medium matters as much as the method.

So the first concrete change dreaming forced on my harness predates the nightly job itself. Every eval metric now returns two things:

def refund_metric(example, pred) -> dict:
    score = judge_score(pred.answer, example.golden)   # 0..1
    notes = judge_notes(pred.answer, example.golden)   # WHY it scored
    return {"score": score, "feedback": notes}

# a scalar tells the optimizer THAT it failed;
# the feedback tells it what to try next

The judge was already an LLM; asking it to show its work costs one prompt line. Design rule: make the reward speak.

What the field calls this

I called this “dreaming” on a slide before I knew what the literature called it. The literature, it turns out, has been busy.

The umbrella term is self-evolving agents — the survey that organizes the space (published in TMLR) asks three questions of any system: what evolves (prompts, memory, tools, weights), when (during a task or between tasks), and how (what signal drives the update). In that taxonomy, my design is inter-test-time self-evolution: the agent learns between tasks, offline, from accumulated experience — never mid-conversation. And it updates through what an earlier survey named the in-context half of the In-Weight vs In-Context split: improvements land in prompts, memory, and instructions rather than in parameters — explicitly motivated, in their words and my economics, by avoiding training costs.

The core operation has a name too. The memory-evolution survey codifies cross-trajectory abstraction — compressing sets of similar trajectories into general, reusable rules — as the frontier of its Storage → Reflection → Experience progression. That is dreaming’s distillation step, named and formalized by someone else, which is always a comforting thing to discover.

And there’s a lineage. Reflexion (2023) had an agent critique its own failed attempt and retry. ExpeL (AAAI-24) moved the learning offline — gather trajectories, distill insights in batch, recall them at inference, zero weight updates — and is the closest published ancestor of what I’m building. The 2025 crop (Agent Workflow Memory, ReasoningBank) distills reusable strategies from successes and failures into memory banks.

None of these papers says “dreaming”; the name is mine. But that’s the useful takeaway from this section: if you build this, you are not off the map. The map has names on it.

The dreaming design

The job runs nightly, offline, while nobody is waiting on an answer. Five stages.

Select. Yesterday’s trajectories come out of the archive — and a decision from a different article pays its second dividend here. Compaction never deletes; it flips a flag. So every run the agent has ever made is preserved losslessly, and every one is replayable tonight. What gets selected: every thumbs-down (a repro case with its full span tree attached), every eval failure, and a sample of clean wins, because contrast needs both sides. Selections are grouped by task similarity — the abstraction step works on sets of similar attempts, not on single stories. A group in practice is small: a dozen refund conversations, five runs of the same skill against the same integration. Enough attempts to show a pattern; few enough that the distiller actually reads them.

Distill. An offline agent replays each group and writes insights. The recipe is ExpeL’s: put failures and successes of the same kind of task side by side and ask what separates them. The output format is the discipline: never a summary, always a rule — “confirm the order id before calling any refund tool”, “state the date filter explicitly when the search spans years” — or a concrete edit to a skill’s instructions. Rules generalize to tomorrow’s traffic. Summaries only compress yesterday’s.

Pool. Insights land in a pool with a lifecycle, and the lifecycle is load-bearing:

@dataclass
class Insight:
    text: str            # "confirm the order id before any refund tool"
    importance: int      # born at 2; +1 on upvote, -1 on downvote
    sources: list[str]   # trajectory ids that produced it
    scope: str           # the skill or agent it attaches to
# pruned from the pool when importance reaches 0

Born at importance 2, upvoted when a later run that used the insight succeeds, downvoted when one fails, deleted at zero — ExpeL’s ADD/EDIT/UPVOTE/DOWNVOTE mechanism, copied without shame. And the pool only ever changes by delta. ACE measured what happens when a model iteratively rewrites its own accumulated context: 18,282 tokens of working playbook collapsed to 122 after a single rewrite — they call it context collapse, and its quieter sibling brevity bias (summarization steadily dropping the domain detail that made the context useful). Incremental edits only. Nothing ever rewrites the pool wholesale.

Gate. A proposed delta has to earn its merge. It runs against held-out goldens the optimizer has never seen — the best-documented failure mode of prompt optimization is gains that don’t transfer beyond the cases it optimized against — and then against the CI eval floor from the harness article. Fail either, and the delta is discarded, not negotiated with.

Commit. What survives becomes a pull request: memory rows, skill-instruction edits, prompt deltas. A human reads a diff of what the agent learned overnight — which is a strange and wonderful artifact to review. Dreaming proposes; people and pipelines dispose.

The whole job, in the shape I’m building it:

async def dream():                        # nightly, offline
    groups = select_trajectories()        # fails, wins, grouped by task
    for g in groups:
        insights = distill(g)             # contrastive fail/success pairs
        pool.propose(insights)            # ADD / EDIT / UPVOTE / DOWNVOTE
    delta = pool.pending_delta()          # never a full rewrite
    if await gate(delta):                 # held-out goldens + CI floor
        open_review(delta)                # a PR; humans hold the merge

To be precise about what’s real: the eval service, the feedback keying, the memory pools, and the archive all run in production today. The nightly job that connects them is the part I’m building now — this article is its design doc, published.

Why not weight-RL

The obvious question: if I want reinforcement learning, why not do it properly — GRPO the model on my own trajectories?

Because of what weight-RL demands, and what it produces. It demands a rollout farm (thousands of sampled attempts per update), a GPU training budget, and a discipline most product teams don’t staff. It produces a checkpoint — an artifact welded to one base model, opaque to review, and expensive to redo when the base model improves underneath you. In a year where the base model improves twice, that last property is the killer.

Then there’s the result that reframed the choice. GEPA reports beating GRPO — actual weight-updating RL — by 6% on average and up to 20%, using up to 35× fewer rollouts. It’s the authors’ own comparison, so hold it loosely; but it survived peer review as an ICLR Oral, and its logic matches the mechanism from earlier: a reflection step reading textual feedback learns more per rollout than a policy gradient reading a scalar. Practice signals point the same way. DSPy’s docs report Shopify converting a GPT-5 task to a small, GEPA-optimized Qwen — ~75× cheaper, ~2× more reliable — and Dropbox’s engineering blog reports roughly halving their relevance-judge error after migrating the judge to a smaller model with optimized prompts. Vendor-reported, all of it. But vendors reporting production numbers is what practice signals look like.

The honest flip side: weights win somewhere. Millions of trajectories, genuine long-horizon credit assignment, a latency budget that can’t carry fat prompts — those push toward weight-RL, and nobody has independently replicated the GEPA-vs-GRPO result yet. One strong study is not a law. But for a small team running one production platform, the economics aren’t close, and they compound with one property weight-RL can’t offer: every config update is a diff. You can read it in review, audit it a month later, revert it in a minute, and carry it — with re-validation — across a model swap. Try any of that with a checkpoint.

One cost is real and worth naming: insights ride in the context window, and the window is never free — the pool competes with everything else the harness fights to keep small. That’s the other reason the lifecycle prunes hard. A lean pool isn’t just cleaner; it’s cheaper on every single call the agent ever makes.

The failure modes I’m designing against

The literature’s greatest gift to this design isn’t the encouragement. It’s the documented ways it fails.

Transfer failure. The most-measured problem in prompt optimization: gains on the cases you optimized against frequently don’t survive a different benchmark — or a different model backbone. Counters: the gate’s held-out goldens rotate, so the optimizer never sees tomorrow’s exam; and a model swap triggers re-validation of the entire pool, because an optimized config is a per-model artifact.

Context collapse and brevity bias. Named and measured by ACE: iterative full rewrites erode accumulated detail; summarization steadily drops the domain specifics that made an insight useful. Counters: deltas only, a structured pool instead of one blob, and no operation that touches more than it proposes.

Memory contamination. Stored errors compound — an agent that follows its own bad experience gets worse with use, and the memory survey is blunt that unbounded growth degrades agents. Counters: the importance lifecycle prunes whatever stops earning upvotes, and the gate keeps bad insights out of the pool in the first place. Curation at the door beats cleanup after.

Reward hacking. Give an optimizer one number and it will Goodhart it — confident evasion scores well on relevancy; brevity scores well on nearly everything. Counters: the score was plural from the start (the harness article’s eval section made that argument), canary tasks — goldens with locked answers that no legitimate improvement should move; if a proposal moves one, the proposal is lying — and a human holding the final merge.

One honesty note: the first three counters lean on measured findings. The regression-gate protocol itself — how often to rotate holdouts, how many canaries are enough — has thin literature behind it. That part is engineering judgment, and I’ve labeled it as such.

What transfers

Four principles, portable to any serious agent system:

1. The learning loop is assembly, not invention. If you built the harness — evals, trace-keyed feedback, memory, a lossless archive — the training setup already exists. You are one offline job away, not one research program away.

2. Make the reward speak. Scalar-only metrics starve the optimizer. Let the judge explain itself, and feed the explanation back into the loop.

3. Never let the optimizer grade its own homework. Held-out evals, rotated. Canaries. Re-validation on every model swap. Without these you are measuring overfitting and calling it improvement.

4. Improvements are diffs. Reviewable, auditable, revertible, portable. The moment an update can’t be read by a human, you’ve built the sci-fi version after all — just without the safeguards.

Which gives the through-line of these three articles its last clause. The model reasons. The harness makes it production-ready. The loop makes it better tomorrow than it was today.