Your AI Agent Learned Something. Should It Be Allowed to Remember It?

The agent makes a mistake.

It reviews the failed run and writes this lesson into memory:

When a deployment fails, retry with elevated permissions.

The lesson looks useful. It is short, specific, and derived from experience.

It is also dangerous.

The next task may fail for a completely different reason. The user may not have approved privilege escalation. The original failure may have been caused by a transient network error. Yet the rule now sits in durable state, waiting to influence every future deployment.

The agent did not merely remember something.

It deployed a behavior change.

That is the missing operational idea in most discussions of self-improving agents. We talk about memory as if it were a notebook and skills as if they were harmless documentation. But once an artifact is retrieved automatically and changes future decisions, it is part of the running system.

A new 2026 survey, Self-Improvements in Modern Agentic Systems, gives us a precise model. An agent is not only a foundation model. It is the model plus an operational scaffold of prompts, memory, tools, and control logic. Self-improvement is the process of obtaining and committing updates to the model or that scaffold.

The word “committing” matters.

If an agent changes its memory, prompt, tool policy, reusable skill, or own code, the next run is using a new agent version — even if nobody changed the model weights.

1 run

enough to propose a lesson, not prove it

0

persistent changes without rollback

N+1

agent version after a promoted memory or skill change

Useful learning already happens outside model weights

This is not hypothetical.

Reflexion lets an agent reflect on task feedback and stores the resulting text in an episodic memory buffer. Later trials retrieve those reflections so the agent can act differently without fine-tuning its weights.

ExpeL gathers experiences from training tasks, extracts natural-language insights, and recalls those insights during inference.

Voyager goes further. Its Minecraft agent accumulates an ever-growing library of executable skills. Environment feedback, execution errors, and self-verification improve programs that can later be retrieved and composed.

A Self-Improving Coding Agent moves the update target into the agent implementation itself. The system uses coding tools to edit its own code and reports a gain from 17% to 53% on a random SWE-bench Verified subset.

These systems persist different things:

• a reflection
• an extracted insight
• executable skill code
• an agent implementation
• a prompt or tool policy
• eventually, model parameters

The authority differs. A note about a user’s preferred date format is not equivalent to a skill that runs shell commands. A retrieved debugging heuristic is not equivalent to a permission rule. But all of them can influence later behavior.

The right question is not “does the agent have memory?”

It is:

What may be written?
Who or what supplied the evidence?
When is it retrieved?
What authority does it gain when retrieved?
How do we remove it if it is wrong?

Separate experience from policy

Most unsafe learning loops collapse four different artifacts into one bucket called memory.

Four layers of agent learning

Artifact

• Raw trace: what happened in one run
• Episode summary: a compact interpretation
• Candidate lesson: a proposed reusable rule
• Promoted policy or skill: affects future runs

Required treatment

• Append-only evidence with provenance and retention limits
• Marked as inference; linked back to the original trace
• Versioned patch with scope, test cases, and expiry
• Eval gate, approval boundary, monitoring, and rollback

A trace can usually be stored automatically. It records observable facts: inputs, tool calls, outputs, errors, and environment state.

A summary is already riskier. It selects what matters and may turn uncertainty into a confident explanation.

A lesson is a generalization. It claims that a pattern from one or more episodes should apply again.

A promoted skill or rule is executable policy. It changes what the system will do.

Do not let an agent move directly from failed run to promoted policy.

failure → reflection → permanent memory

Use a staged path:

Persistent change-control loop

learning becomes durable only after evidence survives the gate

1. 01
   capture trace
2. 02
   propose lesson
3. 03
   classify authority
4. 04
   run frozen evals
5. 05
   approve candidate
6. 06
   canary and monitor
7. 07
   promote or roll back

Every durable lesson needs a change manifest

The smallest useful unit is not a prose memory. It is a structured candidate change.

change_id: lesson-2026-0717-0042
target: skills/deployment-recovery.md
trigger:
  - run-184 failed after a registry timeout
proposed_rule: retry once after confirming the failure is transient
scope:
  task_types: [container-deployment]
  environments: [staging]
  excluded: [production, permission-errors]
evidence:
  supporting_runs: [run-184, run-191, run-205]
  counterexamples: [run-203]
verification:
  development_cases: [deploy-12, deploy-19]
  regression_cases: [permission-denied-03]
rollback:
  previous_version: skill-v17
  condition: any unauthorized retry or lower success rate
expiry: 2026-08-17
status: candidate

This does three important things.

First, it keeps the original evidence attached. Future reviewers can distinguish “the agent observed three registry timeouts” from “the agent believes retries are generally good.”

Second, it limits scope. A useful staging heuristic does not silently become a global production rule.

Third, it makes deletion possible. Natural-language memory systems are often optimized for adding and retrieving information. Safe systems also need invalidation, supersession, expiry, and rollback.

A candidate lesson must answer

• Which exact runs produced this proposal?
• Is the claim an observation, inference, preference, or executable rule?
• Where should it apply — and where must it not apply?
• What frozen case proves the change helps?
• Which regression would force rollback?
• Who is allowed to promote a change with this authority?

One successful run is weak evidence

An agent will often produce a plausible lesson after a failure. Plausibility is not generalization.

Suppose a coding agent fails because it did not read AGENTS.md. It proposes:

Always search the entire filesystem for instruction files before editing.

That rule may solve the recorded case and make every future run slower, noisier, and more likely to cross a privacy boundary.

The smallest durable rule might instead be:

Before editing within a repository, read the nearest applicable AGENTS.md files
from the repository root to the target directory.

You learn the difference by testing the proposal against multiple cases:

• the original failure
• similar failures
• ordinary successful tasks
• counterexamples where the rule should not activate
• adversarial inputs trying to turn untrusted text into durable policy
• a private holdout the proposing agent cannot see

The promotion question is not “did this lesson fix its triggering example?”

It is “does this change improve the task distribution without creating a new hard failure?”

promote =
  repeated_supporting_evidence
  AND private_holdout_gain
  AND zero_new_permission_or_safety_failures
  AND cost_within_budget
  AND rollback_ready

This is especially important for self-editing agents. The candidate must not be able to rewrite the eval, remove a failing case, lower the threshold, change its own permissions, and then declare itself improved.

The evaluator, approval boundary, and deployment mechanism live outside the candidate’s edit scope.

Memory can turn one injection into future behavior

Bad persistence is not only an accuracy problem.

Persistent memory creates a delayed security boundary. An agent may read an email, webpage, issue, repository file, or tool result containing adversarial instructions. If it saves those instructions as a lesson, the attacker no longer needs to be present in the next session.

From Untrusted Input to Trusted Memory identifies four memory-write channels and nine structural vulnerabilities. Its central operational finding is sharp: agents that write and retrieve memories more aggressively are more exploitable. Existing prompt-injection defenses do not automatically solve memory poisoning.

When Agents Remember Too Much demonstrates a two-stage attack: injection writes a hidden payload into memory; activation happens later when the poisoned memory is retrieved. The paper reports approximately 98% injection and 60% average activation in its evaluated systems.

Those numbers belong to specific benchmarks, agents, and threat models. They are not universal production failure rates. The architectural lesson is more durable:

Untrusted input must not become trusted future policy merely because
the agent summarized it in its own voice.

Provenance must survive summarization. A lesson derived from an external webpage remains externally derived. A tool echo does not make it trusted. Two memories copied from the same source are not independent corroboration.

For high-authority changes, require trusted evidence or human approval:

• permission and credential policies
• deployment and rollback procedures
• external communication rules
• security exceptions
• billing or spending behavior
• changes to the evaluator itself

Safe persistence boundaries

Do

• ✓ store source provenance with every proposed lesson
• ✓ separate untrusted observations from trusted operating rules
• ✓ require corroboration across independent runs or sources
• ✓ expire low-confidence memories and preserve rollback versions
• ✓ keep permissions and promotion policy outside self-edit scope

Avoid

• × promote one-run reflections directly into global memory
• × treat the agent own summary as a trusted source
• × allow retrieved text to expand tool authority
• × overwrite the previous rule without an audit trail
• × let the agent grade and deploy its own policy change

Build the minimum viable gate

Small teams do not need a research platform to make this safer.

Start with plain files and Git.

agent-state/
  traces/           append-only run evidence
  candidates/       proposed lessons and skill patches
  promoted/         current approved rules
  evals/            frozen regression and adversarial cases
  CHANGELOG.md      why each version was promoted or rolled back

Give each run an immutable ID. Require candidate changes to cite those IDs. Store prompts, skills, and memory policies as reviewable text or code. Diff them. Test them. Tag promoted versions.

Then implement three promotion classes:

class_1_low_authority:
  examples: formatting preference, project vocabulary
  gate: user-confirmed or repeated evidence

class_2_behavioral:
  examples: debugging heuristic, tool sequence, reusable skill
  gate: regression eval + private holdout + rollback

class_3_privileged:
  examples: permissions, external actions, deployment policy
  gate: explicit human approval; never self-promoted

Roll out behavioral changes as a canary. Send only a small task subset through the candidate version. Compare success, cost, latency, corrections, and hard failures with the current version. Promote only after the gain survives real traffic.

If a proposed lesson cannot be tested, do not silently turn it into policy. Keep it as a low-confidence note with limited retrieval or ask for human review.

“Remember everything” is not learning

More memory can feel like more intelligence. Often it is only more state.

A useful learning system must decide:

• what deserves retention
• what can be generalized
• what authority a retrieved item receives
• when old knowledge stops applying
• how conflicting lessons are resolved
• how a bad update is removed

Without those mechanisms, an agent does not accumulate wisdom. It accumulates configuration drift.

The safest default is asymmetric:

Easy to record evidence.
Hard to promote policy.
Easy to roll back.

That still allows agents to improve. Reflexions can become candidates. Repeated failures can become regression cases. Successful procedures can become skills. Better prompts and tool policies can replace weaker ones.

But the agent does not receive the authority to define “better,” rewrite the gate, and deploy itself in one motion.

The real architecture of a learning agent

A self-improving agent is not one loop.

It is two:

Task loop:
observe → reason → act → verify

Change-control loop:
collect evidence → propose patch → evaluate → approve → deploy → monitor → roll back

The first loop completes work.

The second changes the system that will complete future work.

Confusing them is how a temporary error becomes a permanent habit, a prompt injection becomes delayed control, or a narrow optimization becomes global policy.

So yes, let the agent learn.

Let it propose memories, skills, prompts, and even code changes. Let it connect failures across runs and generate hypotheses faster than a human operator could.

But require every durable behavioral change to arrive as a patch with provenance, scope, tests, authority classification, expiry, and rollback.

Your agent learned something.

That is a reason to evaluate the lesson — not permission to deploy it.

Sources

• Self-Improvements in Modern Agentic Systems: A Survey
• A Self-Improving Coding Agent
• Reflexion: Language Agents with Verbal Reinforcement Learning
• Voyager: An Open-Ended Embodied Agent with Large Language Models
• ExpeL: LLM Agents Are Experiential Learners
• From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agents
• When Agents Remember Too Much: Memory Poisoning Attacks on Large Language Model Agents
• Hidden in Memory: Sleeper Memory Poisoning in LLM Agents