AI Is Now Stateful: What the Context Era Means for Builders

Stateless vs. stateful: the architectural divide

A stateless AI agent treats every session as its first session. The LLM receives a system prompt, the current query, and whatever conversation history the developer manually assembles. When the session ends, everything is discarded. The next session starts from a blank prompt.

A stateful AI agent maintains memory that persists across sessions. When a user returns after three weeks, the agent knows what was discussed, what preferences were expressed, what problems were resolved, and which approaches failed. It does not ask the same clarifying question it asked three weeks ago.

The difference is not just user experience. It is a fundamentally different architecture. Stateful agents require a memory store, a retrieval mechanism, a decay model, and a graph layer. Getting any one of those wrong produces a different class of failure — not a crash, but degraded relevance that compounds over time.

What changes architecturally

In a stateless system, the flow is: receive query → call LLM with system prompt + history → return response. History is managed by the client and grows linearly.

In a stateful system, the flow is: receive query → retrieve relevant memories from context engine → call LLM with system prompt + retrieved context → update memory store with interaction outcome → apply decay pass. History is managed by the context engine and stays bounded in size while growing in coverage.

The critical architectural shift is that memory retrieval is not the same as history retrieval. History retrieval returns a linear sequence of prior turns. Memory retrieval returns the most relevant, most recent, most important facts from an arbitrarily large history — ranked, decay-adjusted, and graph-traversed to surface connected context.

Stateless vs. stateful comparison

The memory loop in code

The stateful agent loop has four phases: retrieve, reason, store, decay. Here is the minimal implementation with Feather DB:

import feather_db as fdb
from your_embedder import embed  # any 768-dim embedder
from your_llm import call_llm

db = fdb.DB.open("agent.feather", dim=768)

def run_session(user_id: str, query: str) -> str:
    query_vec = embed(query)

    # Phase 1: Retrieve relevant memories
    # Decay and stickiness applied automatically on retrieval
    results = db.search(
        query_vec,
        k=12,
        half_life=30,       # memories halve in score every 30 days if not recalled
        time_weight=0.3,    # 30% recency weight, 70% semantic similarity
        filters={"user_id": user_id}
    )
    context_blocks = [r.payload["text"] for r in results]

    # Phase 2: Reason — call LLM with retrieved context
    system_prompt = "You are a helpful assistant with memory of prior sessions."
    context_str = "\n".join(context_blocks)
    response = call_llm(
        system=system_prompt,
        context=context_str,
        query=query
    )

    # Phase 3: Store the new interaction as a memory
    import uuid, time
    memory_id = str(uuid.uuid4())
    meta = fdb.Metadata(importance=0.8)
    meta.set_attribute("user_id", user_id)
    meta.set_attribute("text", f"Q: {query}\nA: {response}")
    meta.set_attribute("timestamp", str(time.time()))
    db.add(id=memory_id, vec=embed(f"{query} {response}"), meta=meta)

    # Phase 4: Link to prior related memories (graph edge)
    if results:
        top_related_id = results[0].id
        db.link(
            from_id=memory_id,
            to_id=top_related_id,
            rel_type="same_session",
            weight=0.8
        )

    return response

The retrieval call at line 12 does the heavy lifting. It does not return the 12 most similar chunks. It returns the 12 highest-scoring chunks after applying temporal decay (memories unused for 30 days score half as high) and recall stickiness (memories retrieved frequently resist that decay). The LLM sees a window of curated, time-relevant context — not a dump of everything ever said.

Graph traversal for causal context

The graph layer adds a capability that flat retrieval cannot provide: the ability to surface the context around a retrieved fact, not just the fact itself.

# Surface connected context from a retrieved memory
def get_context_chain(query: str, user_id: str) -> list[str]:
    query_vec = embed(query)

    # BFS traversal: start from top semantic matches,
    # then follow graph edges 2 hops deep
    chain = db.context_chain(
        query_vec,
        k=5,           # start from top 5 semantic matches
        hops=2,        # follow edges 2 hops
        filters={"user_id": user_id}
    )
    return [node.payload["text"] for node in chain]

If a user reports a billing issue, the top semantic match might be the prior conversation where they mentioned billing. Graph traversal at 2 hops surfaces the resolution that was attempted, the agent's response at the time, and any linked preference about communication style — all without requiring those facts to be semantically similar to the current query.

What to store and what to decay

Not everything belongs in the memory store. The discipline of stateful AI architecture is deciding what to remember.

Store with high importance (0.8–1.0): user preferences, confirmed facts, resolved issues, explicit instructions, key decisions. These should resist decay.

Store with medium importance (0.4–0.7): interaction summaries, intermediate reasoning steps, background context that provided useful grounding. These should decay at normal rate.

Do not store: LLM reasoning traces, intermediate tool outputs, conversational filler, low-confidence inferences. These add noise without signal.

The importance parameter at storage time determines the initial weight in the scoring formula. Combined with recall stickiness — which increases effective importance with each retrieval — the system naturally promotes what has proven useful and demotes what has not.

Retrieval speed as an architecture constraint

At 0.19ms p50 ANN latency on 500K vectors, Feather DB retrieval is fast enough to run synchronously in the agent loop without adding perceptible latency. This matters architecturally: if retrieval requires an async call to an external service with 50–200ms round-trip, the agent design changes — you add caching, you pre-fetch, you make tradeoffs that wouldn't exist with embedded retrieval.

The 5–6× faster cold-load performance in Feather DB v0.16 (persisted HNSW) means agents start with full context available immediately, even after service restarts. There is no warm-up period where retrieval quality is degraded because the index has not loaded.

The 97.2% recall@10 on 500K vectors means the retrieval quality is consistent at scale. An agent serving users with large memory stores — years of interaction history — retrieves with the same accuracy as an agent with small stores.

Frequently Asked Questions

What makes an AI agent stateful?

A stateful agent maintains a memory store that persists across sessions. At each interaction, it retrieves relevant memories, uses them to ground the LLM response, stores the new interaction, and updates memory scores. Statefulness requires a context engine — not just a vector database, but one with decay, stickiness, and graph traversal.

How much memory should an agent store per user?

There is no hard limit with a context engine — storage grows with user history, but retrieval cost stays constant because you always retrieve k memories regardless of total store size. In practice, high-importance memories accumulate for years while low-importance memories decay out of effective retrieval within weeks. The active working set stays bounded even as the total store grows.

How does decay prevent stale information from surfacing?

The decay formula applies an exponential half-life to memory age, adjusted by recall frequency (stickiness). A memory not retrieved in 30 days (at default half_life=30) scores half as much on the recency component. After 90 days without retrieval, it scores 12.5% of its initial recency score — below the threshold where it competes with fresh memories on the same topic.

What is the LongMemEval benchmark and why does it matter?

LongMemEval tests AI memory across simulated long-running conversations: recall of past facts, temporal reasoning (what happened before vs. after), preference tracking across contradictory statements, and knowledge updates when facts change. It is the closest available proxy for production agent memory quality. Feather DB with GPT-4o scores 0.693 vs. 0.640 for GPT-4o full-context — the same LLM performing better because the context it receives is better.

Can I use Feather DB with any LLM?

Yes. Feather DB is model-agnostic — it stores and retrieves embeddings, not model-specific formats. You can use any embedding model (text-embedding-3-small, all-MiniLM-L6-v2, nomic-embed-text, or custom) and any LLM for inference. The Gemini-2.5-Flash configuration achieves 0.657 on LongMemEval at $2.40 per full benchmark run, making it viable for cost-sensitive production deployments.