In my previous post on LLM memory evolution through 2024-2025 we saw the shift from prompting to context engineering and use of techniques such as RAG, GraphRAG etc. Now lets examine this use case for memory when building more complex flows with LLMs to do a series of steps in an “agentic flow”
As we move from single-shot chatbots to workflows which span customers, internal users and systems these solutions tend to accumulate decisions over weeks or months. And that’s where most “agent demos” fall over – you want an employee that has a motivation to continually learn, improve and grow and be better at their job and this needs long term memory as a foundation.
The 2025 enterprise agent demos bring solutions which look smart in a single session, but then fail in real world scenarios because we re-discover the same constraints again and again as there is no durable, architected memory behind it
In 2026, that difference is the difference between a pilot and a platform capability.
1. Why enterprise agents need a memory stack
Pick any real enterprise scenario:
A claims assistant that needs to remember “what we agreed last time”, the evidence we used, and why an exception was approved. An integration design copilot that keeps proposing the same coupling mistakes because it can’t recall previous decisions, boundaries, and failure modes. A platform/SRE assistant that can answer “what is CPU”, but can’t answer “what changed between the last two incidents on this service”.
The tricky part is not whether the model can reason. The tricky part is whether the system can remember in a way that is:
fast enough for chat-like use structured enough for relationship-heavy questions governed enough for audit, security, and accountability cheap enough to operate at scale
That’s a memory stack problem, not a prompt problem.
2. From RAG to a layered memory architecture
Early enterprise deployments treated “RAG + vector database” as the full memory story. Chunk PDFs, embed, retrieve top-k, stuff it into the prompt, call it done.
That works for document Q&A. It breaks down when the work is episodic and long-running:
“What happened the last time we changed this integration?” “Which services tend to fail together after a deployment?” “What constraints did we agree with security and why?”
The mental model I use is the same one we already use in distributed systems:
cache vs source-of-truth hot vs warm vs cold storage bounded contexts and ownership read/write patterns that are disciplined, not magical
RAG becomes one layer. Not the whole architecture.
3. The four layers of a 2026 memory stack
Think of this like a reference diagram you can carry into design reviews.
3.1 Layer 1: Working memory (context window)
This is the immediate working set for this task, right now.
What lives here:
the last N exchanges (or a rolling summary) the active plan (“what I’m doing next”) tool outputs that matter right now a scratch-pad for intermediate structure (e.g., a draft architecture sketch)
Constraints:
token budget is real latency has to feel like chat “just add more context” eventually becomes a cost and quality problem
Design choices:
sliding window vs summarised window what is cached for the duration of a task vs re-fetched each step what gets dropped vs compacted
Short example: “design this integration”
keep the last 6–10 exchanges keep the current architecture sketch (components + interfaces + non-functionals) keep the active constraints list (security, residency, latency, cost) everything else is retrieved by policy, not by nostalgia
3.2 Layer 2: Episodic memory (tasks, cases, journeys)
Episodic memory groups interactions into meaningful units: a claim journey, an incident, a design engagement, a migration.
If you’ve worked with event-driven and microservice systems, this will feel natural. Episodes align to business concepts and bounded contexts, not arbitrary chunks.
Data model (minimum viable):
episode_id type (incident, claim, design, migration, release) participants (people/teams/agents) entities (customer, service, system, integration, product) timestamps (start/end, key events) artefacts (chat log pointers, diagrams, PRs, tickets, runbooks) summaries narrative summary (what happened) decision summary (what we decided and why) risk summary (what could bite us later)
Retrieval patterns:
“find previous episodes for this service/customer/domain” “show me the last two similar incidents and what fixed them” “what changed between the last two integration revisions”
Implementation hint:
treat the episode as an append-only log of events (tool calls, decisions, evidence) store the canonical record in a relational/document store add a secondary embedding index for recall and clustering keep “evidence pointers” first-class so you can audit later
3.3 Layer 3: Semantic / knowledge memory (facts and relationships)
Over time, episodes get distilled into more stable knowledge: entities, relationships, policies, and domain facts that change slowly.
This is where memory starts to look like what architects already care about:
services and capabilities as first-class entities dependencies and data flows as relationships ownership and boundaries as explicit structure policies and constraints as reusable “guard rails”
Contents:
entities (customers, products, systems, services, integrations) relationships (depends-on, publishes, consumes, owned-by, shares-data-with) constraints (SLAs, compliance rules, residency, cost limits, vendor constraints) known patterns and failure modes (the “how we do things here” memory)
Techniques:
entity-centric indexes: “everything about Service Y and Integration Z” graph-style retrieval when relationships matter semantic retrieval before similarity retrieval
Role:
shared memory across agents the layer that makes behaviour consistent across time and across teams
If your organisation has an API catalogue, capability model, domain boundaries, and event taxonomy, this is where those assets stop being documentation and start becoming operational memory.
3.4 Layer 4: Governance and observability memory
By 2026, governance is not a bolt-on. If you cannot answer “why did the agent do that?”, you don’t have a production system.
Data captured:
prompts and system instructions (versioned) retrieved items (what was injected, and from where) actions taken (tool calls, writes, approvals) outputs produced memory versions used (which knowledge snapshot, which policies)
Uses:
audit and compliance (“show me the basis of this decision”) post-incident analysis (“why did it approve this change?”) drift detection (behaviour shifts when memory or model changes) safety reviews (leakage, overreach, privilege misuse)
Implementation:
extend your existing observability stack (logs/traces/metrics) add AI-specific fields (episode_id, retrieval_set_id, policy_version, model_version) treat this as governance memory, not just debug logs
4. Cross-cutting concerns: capacity, latency, safety
A memory stack is a resource management problem. You balance capacity, latency, and safety rather than maximising any one.
4.1 Layer comparison table
Layer
Primary goal
Capacity strategy
Latency target
Safety strategy
Working
immediate task success
strict token budget + compaction
chat-speed
minimise sensitive carry-over
Episodic
continuity + audit
months of retention + summarisation
seconds (indexed)
role-based access + evidence pointers
Semantic
shared truth + consistency
small by design, distilled
fast reads
governance + schema ownership
Governance
accountability
append-only retention
not in hot path
immutable logs + versioning
4.2 Capacity
working memory: minutes/hours episodic memory: months (or longer with pruning/anonymisation) semantic memory: years (but curated) governance memory: retention by policy (often longer than you think)
Compression techniques:
summarisation at episode boundaries multi-resolution summaries (short/medium/deep) tiering (hot/warm/cold) with explicit eviction rules
4.3 Latency and cost
Different workflows have different SLAs:
chat-like assistance: sub-seconds to a few seconds complex planning: slower is acceptable if it is transparent overnight consolidation: batch-friendly
Optimisation levers:
precompute embeddings at ingestion cache per episode (don’t re-fetch the same evidence every turn) limit retrieval hops deterministically keep semantic memory small and structured
4.4 Safety and privacy
data minimisation: store what you need for future reasoning, not everything you can access control: not every agent sees every layer or every field redaction and residency: episodes often contain sensitive content separation of duties: approvals and writes should be policy-gated
5. How agents read and write the stack
The stack only matters if access patterns are disciplined. The pattern that keeps showing up is “read-then-write”.
Before acting:
query semantic memory for entities and policies retrieve relevant episodes (by entity, by recentness, by type) compose a focused working context within budget act with tools
After acting:
append events and evidence to the active episode update episode summaries update semantic memory only when durable knowledge changes emit governance records for audit
Background processes:
consolidate episodes into semantic facts prune/anonymise old episodes re-index when domain models change
Sequence diagram (PlantUML):@startuml actor User participant "Agent Orchestrator" as Orchestrator participant "Working Memory" as WM database "Episodic Store" as ES database "Semantic Store" as SS database "Governance Log" as GL participant "Tools" as Tools User -> Orchestrator: Request (goal + context) Orchestrator -> SS: Read entities + policies SS --> Orchestrator: Facts + constraints Orchestrator -> ES: Retrieve relevant episodes (entity/time/type) ES --> Orchestrator: Episode summaries + evidence pointers Orchestrator -> WM: Compose working context (budgeted) WM --> Orchestrator: Prompt-ready context Orchestrator -> Tools: Execute actions (read/write) Tools --> Orchestrator: Results Orchestrator -> ES: Append events + evidence to episode Orchestrator -> ES: Update episode summaries Orchestrator -> SS: Update durable facts (optional, policy-gated) Orchestrator -> GL: Record prompt/retrieval/actions/output (versioned) Orchestrator --> User: Response (with evidence and constraints) @enduml
6. Reference architecture: bringing it together
Walk-through example: “Propose a migration plan for System X over the next two quarters.”
Step 1: request arrives
the orchestrator routes to the right agent (migration planner) an episode is created (or resumed) for “Migration Q2–Q3 System X”
Step 2: semantic read
services and dependencies for System X known constraints (residency, cost caps, platform standards) current ownership and change windows
Step 3: episodic read
last migration attempt prior incidents that affected System X previous decisions and why they were made evidence pointers (tickets, PRs, runbooks, postmortems)
Step 4: working context assembly
tight context: constraints + selected episodes + key evidence snippets the plan is generated inside a bounded token budget
Step 5: write-back
the plan becomes an artefact in the episode decisions and risks are summarised durable knowledge is updated only where it truly changed governance memory records the full basis for later review
How this fits into broader enterprise architecture:
identity and access: governs which memory layers are visible and writable data platforms: host episodic and semantic stores, plus consolidation jobs integration platforms: act as tool surfaces and event sources for episodic memory observability platforms: host governance memory and drift detection
7. Design questions for architects starting now
Use this checklist in your next design review:
What are the natural episodes in this domain and who owns them? Which facts and relationships belong in a shared semantic layer instead of buried in prompts and logs? What are retention and latency targets per layer (working, episodic, semantic, governance)? How will you audit an agent’s decision three months after deployment? Which existing platforms can you reuse (API catalogues, event streams, logging, data stores) rather than rebuilding a parallel stack? What is the policy for updating semantic memory (who can write, how it is validated, how it is versioned)?
Next post: episodic memory patterns — how to design episode schemas, summaries, and indexes that actually work in enterprise systems.References (optional) - https://www.dataversity.net/articles/the-2026-enterprise-ai-horizon-from-models-to-meaning-and-the-shift-from-power-to-purpose/ - https://www.linkedin.com/pulse/real-enterprise-ai-architecture-2026-inclusioncloud-8erof - https://sambanova.ai/blog/ai-trends-2026-insights - https://www.ibm.com/think/news/ai-tech-trends-predictions-2026 - https://www.informationweek.com/machine-learning-ai/2026-enterprise-ai-predictions-fragmentation-commodification-and-the-agent-push-facing-cios - https://www.systemdesignhandbook.com/guides/llm-system-design/
Alok Mishra 