Inside XLR8 AI's Memory Agent: How We Give Every Client a Memory
The problem we were solving
Here's a simple way to think about it: if you upload one document to Claude and ask it a question, it does a great job. Upload a thousand documents about the same client — every call transcript, every Slack message, every file they've sent you, spanning months — and ask the same kind of question, and it starts to break down. Not because the model got worse, but because there's a limit to how much any AI can hold in view at once and still reason about it well.
That's a real problem for a client-services business, because the most useful information about a client rarely lives in a spreadsheet. It lives in a kickoff call where someone mentions their real priority almost in passing. It lives in a Slack thread where a client explains, in their own words, what they actually mean by "sounding on-brand." It lives in a comment on call #7 that contradicts something said on call #2, which matters more than either call on its own — and by the time you're managing dozens of clients, that's not a thousand documents, it's an amount no single AI conversation can just hold in context.
That gap is exactly what breaks the model XLR8 AI depends on. Our business relies on using AI and internal software so a small customer success team can properly serve a large number of clients — the kind of client-to-headcount ratio a traditional agency couldn't sustain. But every part of that — generating the right search queries for a client, revising them as the client gives feedback, writing content that actually reflects what a client has told us — depends on being able to reliably pull up what that specific client has said, whenever it's needed. Without a dedicated way to store and retrieve that, AI can only work at the scale of "one document, one question." We needed it to work at the scale of "every conversation we've ever had with this client."
The solution: XLR8 AI's memory agent
The fix is what we call the memory agent — one instance per client. There was no way around building this ourselves: every client needs their own memory, and there wasn't an off-the-shelf tool that could do this reliably at the scale we operate at. So we built our own AI memory layer, in-house.
The idea, in plain terms: instead of a flat pile of transcripts and messages that only a human (or a single, limited AI conversation) can dig through, the memory agent organizes everything a client has ever told us into a knowledge graph — a structured web of facts and how they relate to each other. That's what makes it possible to ask something specific and get a real, sourced answer back, instead of doing a keyword search and hoping for the best. Someone on our team (or an AI agent acting on their behalf) can ask a client's memory agent things like:
"What queries are important to this client?"
"What has this client told us about their blog post preferences?"
"What's this client's overall goal for this engagement?"
And get back the actual, specific thing the client said — pulled from a real meeting, a real Slack message, or a document they sent — instead of a generic answer or a "go check the call recording" shrug.
How it actually works
A few mechanics worth being specific about, since the point of this piece is to go past the pitch and into the system itself.
What feeds it. Each client's memory agent draws on three sources: call recordings (captured through Fathom), Slack activity across internal and external channels, and any documents uploaded for that client. This isn't a one-time snapshot — it keeps growing as clients keep giving us feedback on calls, in Slack, and through new documents, which is exactly what lets us keep revising a client's search queries and content strategy over time instead of setting them once and letting them go stale.
How it searches. It doesn't rely on just one method. It supports semantic search (meaning-based — finds things that are conceptually related even if the wording is different), lexical search (matching on the actual words used), and structural relationship search — this last one is where the knowledge graph does its job, surfacing how different facts connect to each other rather than treating every mention as an isolated data point. Which method is the right one to reach for, for a given question, is still active engineering work rather than a solved problem.
What it's built on. We didn't build the underlying framework entirely from scratch. It's built on top of Cognee, an open-source knowledge-graph memory framework for AI agents — and, worth a shoutout, also an XLR8 AI client. What's ours end to end: the client-specific ingestion pipeline across all three sources above, and exposing all of it as a "remember" tool/API that our own platform backend and our MCP layer can call directly — so any workflow we build can query a client's memory as easily as calling any other function.
One memory per client, kept separate. Each client's memory graph lives in its own dedicated database — a deliberate design choice, since a shared store would raise real questions about information ever crossing between clients.
How it gets used. Because this is accessible programmatically, it's not limited to a human typing a question into a search box. This is already live, internal infrastructure: our query-generation workflow already pulls from a client's memory agent to decide which search queries matter most, and we're extending the same access into content generation next, so content actually reflects what a specific client has told us rather than generic best practices. Any of our team, or any AI agent we build, can be given access to a client's memory the same way you'd give a program access to a database — a Slack bot called Wolfie is the internal-facing version of this: tag it in a client's channel and ask something like "what did this client say about X in our first call?" and it queries that client's memory agent directly and answers in the thread, no digging through old transcripts required.
Why this matters (beyond being a neat feature)
It would be easy to build a bare-minimum version of AI-search optimization: generic queries, generic content, applied the same way to every client. Plenty of agencies do exactly that. The bet XLR8 AI is making is the opposite — that being genuinely useful to a client means the queries we target and the content we write have to reflect what that specific client has actually told us, revised as they keep giving us feedback, at every client we work with, without needing to scale headcount at the same rate. That only works with real infrastructure behind it, which is what the memory agent actually is: the system that makes AI usable at the scale a real client-services business needs, instead of the scale of a single document and a single question.
Today, the concrete uses are picking the right target queries for a client's AI-search strategy — the same fan-out query targeting we've written about before — and generating content shaped by what that client has told us. But the more interesting question, one that's genuinely still open internally, is how far this extends. A few directions already on the table:
Product development: surfacing patterns in what clients are asking for or requesting as features, pulled directly from real conversations instead of relying on someone remembering to log it.
Marketing and sales: understanding, in aggregate, the kinds of questions and concerns clients actually raise — again sourced from what was really said, not anecdote.
Workflow design: given there are multiple ways to search a client's memory (semantic, lexical, structural), figuring out the best way to route different kinds of questions to the right search method is still active engineering work, not a finished system.
That last point is worth being upfront about: this isn't a "solved and shipped, nothing left to do" story. The system works today and is in active use for query generation, with content generation next in line, but a lot of the most interesting use cases are still being figured out.
The takeaway
The underlying idea is simple even if the engineering isn't: any AI is only as useful as what it can actually hold in view when you ask it something, and for a business built on serving many clients well, that ceiling shows up fast. Building a dedicated memory layer — one that keeps growing with every call, every Slack message, and every document, kept separate per client — is what lets that ceiling move. It's also, plainly, more work than treating every client the same way. That extra work is the point: it's what lets a small team be both accurate and able to scale, instead of picking one or the other.
FAQ
What is XLR8 AI's memory agent, in one sentence?
It's a per-client system that takes unstructured information — call transcripts, Slack messages, and uploaded documents — and organizes it into a structured, searchable knowledge graph, so specific things a client has said can be found and reused instead of getting buried or lost past the point any single AI conversation could hold.
Where does it get its information from?
Three sources: call recordings (captured through Fathom), Slack activity across internal and client-facing channels, and any documents uploaded for that client. It keeps growing as clients keep giving feedback.
Isn't this just what Claude or ChatGPT already do?
At small scale, yes — upload one document to Claude and ask a question, and it does a great job. The gap shows up at scale: once you're dealing with hundreds of calls, messages, and documents per client across many clients, a single AI conversation can't hold all of that in view at once. The memory agent is the layer that makes that scale possible, by organizing everything into a knowledge graph that's queried on demand rather than dumped into one conversation.
Did XLR8 AI build this entirely in-house?
Not the underlying framework — it's built on top of Cognee, an open-source AI memory framework that's also an XLR8 AI client. What XLR8 built on top of it is the ingestion pipeline across calls, Slack, and documents, per-client isolation, and the internal "remember" tool/API that exposes it all to XLR8's own platform and MCP layer. There wasn't an existing tool that solved memory at the scale we needed, so the application layer had to be ours.
How is one client's data kept separate from another's?
Each client has their own dedicated database — there's no shared store that information could leak across.
Who actually uses this — clients, or XLR8's team?
Right now it's internal infrastructure. It already powers our query-generation workflow, is being extended into content generation next, and any team member or AI agent we build can be given access the same way a program is given access to a database. The Wolfie Slack bot is the internal-facing way team members query it directly.
