← Blog
    Multi-agentObservability

    By Alfie Lambert · · 7 min read

    The multi-agent knowledge problem: what nobody is checking for in agent drift


    There's a growing body of work on what happens when AI agents degrade in production. Models update, prompts get edited, data shifts under your feet. The observability community has built solid tooling for catching it. They call it agent drift.

    But there's a dimension of drift that almost nobody is building for. Not the kind where one agent's behaviour changes over time. The kind where multiple agents in the same organisation, running right now, give confidently different answers to the same question.

    This is the multi-agent knowledge problem. We think it's the bigger issue, and we think it's going to get worse before it gets better.

    The version of drift people are watching for

    The existing research on agent drift is real and worth taking seriously. It covers three things, broadly.

    Behavioural drift, where an agent gradually deviates from its intended specification across a session. It loses its tone, exceeds its authority, hallucinates actions it wasn't asked to take.

    State drift, where an agent's internal beliefs become misaligned with reality, even while its step-by-step reasoning still looks locally coherent.

    And temporal degradation: model updates, data shifts, and prompt modifications that erode performance over weeks and months.

    These are real problems. There's promising work on bounding drift at runtime and the observability stack to detect it is maturing fast.

    The catch is that all of this work focuses on individual agents deviating from their own spec. Even the multi-agent research, which correctly identifies that reliability degrades multiplicatively across handoffs, frames the problem as compounding errors in a chain. What it doesn't address is more fundamental: what happens when the agents were never aligned with each other in the first place.

    The version of drift nobody's checking for

    A scenario that comes up in almost every conversation we have with companies running multiple agents:

    A company has a customer-facing support agent, an internal operations agent, and a sales enablement agent. All three need to know the company's refund policy.

    The support agent was given the current policy in its system prompt three weeks ago. The operations agent pulls from a knowledge base that was last updated last quarter. The sales agent references a PDF that a product manager uploaded six months ago and forgot about.

    All three agents answer questions about refunds confidently. All three give slightly different answers. None of them know the others exist.

    This isn't a hallucination problem. Each agent is faithfully representing the knowledge it was given. The problem is that nobody gave them the same knowledge.

    We call this knowledge divergence: the structural inconsistency that emerges when multiple agents in the same organisation lack a shared, authoritative source of truth for the concepts they need to operate on.

    A quick distinction worth making. Not all divergence is bad. Specialisation is divergence by design. Your support agent should know things your sales agent doesn't. The problem is unintended divergence on shared concepts: pricing, policies, definitions, customer state, the things that should be identical across every agent that touches them.

    Why knowledge divergence happens

    It doesn't require anyone to make a mistake. It's the natural consequence of how multi-agent systems get built.

    Each agent gets its own context. Teams configure agents independently. System prompts are written per agent. Knowledge bases are set up per use case. Retrieval pipelines pull from different document sets. There's no mechanism ensuring Agent A and Agent B are working from the same definition of "enterprise tier" or "standard SLA".

    Knowledge updates don't propagate. When a policy changes, someone updates the support agent's prompt. They might remember to update the operations agent's knowledge base. They almost certainly don't update every source every agent references. The more agents you have, the more surfaces need updating, and the more likely it is that at least one gets missed.

    There's no shared schema. Traditional software systems solved this decades ago. If your CRM and your billing system need to agree on what a customer is, they reference the same database. Agents don't have this. Each one constructs its own understanding of business concepts from whatever unstructured context it was handed, the gap a proper shared schema would close.

    Nobody is testing for cross-agent consistency. Observability tools monitor individual agent performance. They check whether Agent A is responding accurately against its own baseline. They don't check whether Agent A and Agent B agree with each other. Every individual agent can perform well while the system as a whole is incoherent.

    A simple thought experiment

    Imagine you hire ten new employees on the same day. You give each of them a different version of the employee handbook. Some are from this year, some from last year, one from three years ago. Each employee reads their handbook carefully and follows it precisely.

    Now make it worse: these employees never talk to each other. When one of them handles a tricky case and learns something important ("actually, the refund window was extended to 60 days last month"), that knowledge stays with them. The other nine never hear about it. Each one keeps operating from their own outdated handbook, in their own silo.

    Every employee is doing exactly what they were told. Across the organisation, policies are being applied inconsistently. Customers get different answers depending on who they talk to. Knowledge gets discovered but never disseminated. Nobody is wrong individually. The system is broken collectively.

    This is what's happening inside most multi-agent deployments today. The agents aren't failing. The knowledge layer that would keep them aligned is missing.

    Why this gets worse at scale

    With one agent, knowledge divergence isn't a problem. You configure it, test it, maintain it. Whatever you put in the prompt or the retrieval pipeline is the source of truth.

    With three agents, inconsistencies are annoying but manageable. Someone notices, someone fixes it.

    With ten or twenty agents serving different parts of the business, maintained by different teams, pulling from different knowledge sources, you have a systemic problem. Inconsistencies multiply. They compound across handoffs, where one agent's output becomes another agent's input. A subtly wrong assumption in Agent A becomes a confidently wrong answer in Agent C, two handoffs later.

    The failure mode is silent. Unlike a crashed server or a timeout error, knowledge divergence doesn't trigger alerts. The agents keep running. They keep responding. They just quietly disagree with each other. Nobody knows until a customer notices, an audit catches it, or something goes visibly wrong downstream.

    What the current stack doesn't solve

    The agentic infrastructure has matured rapidly. Orchestration frameworks like LangChain and CrewAI handle coordination. Observability platforms like Langfuse and Helicone monitor performance and cost. Memory systems give agents persistence across sessions. There's good work on runtime enforcement to keep individual agents within spec.

    These are real advances. None of them address knowledge consistency across agents.

    Layer
    Orchestration
    What it does
    Tells agents what to do, in what order
    What it doesn't do
    Doesn't ensure they share the same facts
    Layer
    Observability
    What it does
    Tells you how each agent performed
    What it doesn't do
    Doesn't compare agents against each other
    Layer
    Memory
    What it does
    Lets agents remember across sessions
    What it doesn't do
    Each agent's memory is its own silo
    Layer
    Runtime enforcement
    What it does
    Keeps individual agents within bounds
    What it doesn't do
    Doesn't define those bounds collectively
    Layer
    RAG / vector DBs
    What it does
    Retrieves relevant chunks per query
    What it doesn't do
    Each agent indexes and interprets independently

    You can have perfectly coordinated, perfectly monitored, perfectly remembered agents that still give contradictory answers because they're drawing from different sources of truth. This is exactly why RAG on its own doesn't fix it, and why each agent's memory ends up as its own silo without shared memory across the system.

    This is the gap. Not in how agents behave. In what they know, and whether they agree.

    How to know if you have a knowledge divergence problem

    If you're running multiple agents, or planning to, four questions worth asking:

    Can you enumerate every source of knowledge each agent relies on? Not just the primary knowledge base. System prompts, uploaded documents, hardcoded context, every retrieval source. If you can't list them, you can't keep them consistent.

    When a business rule changes, how many places need updating? Count the surfaces. If the answer is "one per agent, plus the knowledge base, plus a documents folder", you have a propagation problem.

    Have you ever tested whether two agents give the same answer to the same question? Try it now. Ask your support agent and your sales agent the same thing about pricing, refunds, or SLA. The results are usually surprising.

    When agents hand off to each other, does the receiving agent validate what it's been told? In most systems, Agent B trusts Agent A completely. If Agent A's knowledge was off, that error now lives in Agent B's context as fact.

    If three or more of those questions made you uncomfortable, you have a knowledge divergence problem. It's just not visible yet.

    Where this is heading

    The companies that solve multi-agent knowledge consistency early will have a real advantage as deployments scale. The ones that don't will spend the next few years debugging mysterious failures that no single agent's logs can explain.

    The answer isn't better monitoring of individual agents, though that matters. It's a shared knowledge layer: a single, structured, authoritative source of business truth that every agent in your organisation references. Not a static document repository. Not a vector database that each agent queries independently with its own interpretation. Something that actively maintains consistency across every agent that touches it, and lets them learn from each other rather than in isolation.

    That's what we're building at Oiya: treating cross-agent knowledge management as first-class infrastructure rather than a side effect of prompts and retrieval. If you're deploying multiple agents and any of this sounds familiar, we'd like to talk.

    FAQ

    What is the multi-agent knowledge problem?

    The multi-agent knowledge problem is the structural inconsistency that emerges when multiple AI agents in the same organisation operate from different sources of truth for shared business concepts. Each agent answers confidently. Their answers don't match.

    How is knowledge divergence different from agent drift?

    Agent drift describes a single agent's behaviour degrading over time, usually because of model updates, prompt edits, or data shifts. Knowledge divergence is a multi-agent problem: the agents aren't degrading individually, they were never aligned with each other to start with.

    Doesn't RAG solve this?

    No. Retrieval-augmented generation gives each agent access to a knowledge base, but each agent indexes, retrieves, and interprets independently. Two agents querying the same vector store with different prompts can pull different chunks and reach different conclusions. RAG retrieves; it doesn't enforce consistency.

    Don't orchestration frameworks solve this?

    Orchestration frameworks like LangChain and CrewAI coordinate which agent does what and in what order. They don't ensure those agents share the same understanding of the business concepts they're acting on.

    Don't observability tools catch this?

    Observability tools monitor each agent against its own baseline. They don't compare agents against each other. Two agents can both score well on individual evaluations while contradicting each other in production.

    What does "knowledge divergence" actually look like in production?

    The most common symptoms: customers getting different answers from different channels, internal teams citing different versions of the same policy, audit failures where the documented rule and the agent's behaviour don't match, and silent compounding of errors through agent handoffs.

    Oiya is building the structured knowledge layer for multi-agent AI systems. We're working with a small group of design partners to solve this problem in production. If you're running three or more agents and want to compare notes, book a 30-minute call.

    Related