A support agent has ninety seconds and a customer who is already annoyed. Searching for the answer during the call is the problem, not the solution. This is a multi-agent assistant that does the reading beforehand — customer value, segment, warranty position and issue history enriched ahead of time, so five specialist agents can answer a live transcript while the conversation is still happening.
An LLM-powered enrichment pipeline turns raw customer records and unstructured email into a queryable profile — lifetime value, segment, warranty position, and an issue history threaded across touchpoints.
A live transcript triggers a chain of five specialist agents over vector search and function calls — and a judged evaluation loop that took the system from 33% to 84% accuracy before it went anywhere near production.
Retrieval during a live conversation is a latency budget you cannot afford. So the reading moves earlier.
Manufacturing support runs on context the agent does not have in front of them. A customer calls about a delivery delay. Whether that call should end in an apology, a credit, a warranty claim or an upsell depends on things scattered across systems: what the customer is worth, what they have already complained about this quarter, whether the unit is still in warranty, and whether the last three touchpoints were about the same unresolved issue or three unrelated ones.
Assembling that picture manually takes longer than the customer will wait. So it does not get assembled. The agent answers the question in front of them, the commercial opportunity goes unnoticed, and the fourth call about the same freight classification problem gets treated as if it were the first.
The conventional fix — retrieve everything at call time — runs straight into the latency wall. The answer has to arrive inside the conversation, not after it.
An LLM-powered data enrichment pipeline runs ahead of any conversation, reading customer raw data and unstructured customer email and writing an enriched customer record back to the lakehouse. What the agents query at call time is not the source systems — it is a profile that has already been built.
The enrichment produces the judgements that would otherwise have to be made live: lifetime value, customer segment classification, and an interaction history threaded across touchpoints, so a delivery delay in March and an invoice dispute in May resolve into one narrative rather than two unrelated records. Sentiment is tracked across that thread, which turns satisfaction from a survey question into a trajectory you can see moving.
The expensive reasoning happens before anyone picks up. The live call is a lookup.
Enriching in advance is what makes the recommendation possible at all. Classifying a customer segment or reconstructing an issue thread is a reasoning task measured in seconds; doing it while somebody waits on the line is not an option. Moving it upstream converts an impossible latency budget into a solved one, and has the useful side effect of making the enriched profile available to every other system that wants it.
A single prompt cannot do this job. Five narrow ones, each with its own tool, can — but only after they have been measured honestly.
The assistant takes the transcript of an ongoing conversation and passes it down a chain of specialist agents. Each has a single responsibility and a single mechanism — two over vector search, one function call against a system of record, and two generation steps. Narrow agents are easier to evaluate, easier to fix, and fail in ways you can locate.
Two retrievals, one function call, two generation steps — one recommendation.
What comes back to the support agent is not a search result. It is a recommended response and a recommended action, informed by the enriched profile: whether this customer's segment and lifetime value justify a credit, whether the unit is in warranty, whether this is the fourth contact about one unresolved freight classification issue, and whether the sentiment across that thread is recovering or deteriorating.
“Narrow agents fail in ways you can locate.”
The first working prototype answered correctly 33% of the time. That number is the most useful thing in this case study, because it is what an unevaluated multi-agent system actually looks like before anyone measures it — and it is roughly where a great many of them ship.
A ground-truth set of 50 question-and-answer pairs was generated to evaluate against. Then two distinct rounds of judging: first LLM-as-judge, which is fast and cheap and catches the structural failures — wrong retrieval, missing warranty check, malformed response. That loop took accuracy to 56%.
The second loop swapped in subject-matter experts as judges. SMEs catch what an LLM judge cannot: answers that are well-formed and plausible and wrong in a way only someone who knows the product line would notice. That loop took the system to 84%, and that is the number it deployed on.
Two judges, because they catch different classes of error.
The agent architecture did not change much between 33% and 84%. The evaluation harness did. Most of the distance was covered by generating a ground-truth set, judging against it cheaply, then judging against it expensively — in that order. An LLM judge on its own would have stalled near the middle; SMEs from the start would have burned expert time on errors a model could have caught for pennies.
The assistant is served through a Databricks App, so the interface, the agents and the enriched data sit inside one governance boundary rather than being stitched across a separate application tier. Every call is traced through MLflow, which means a recommendation that looks wrong can be opened up and read — which agent retrieved what, which tool was called, what the model saw.
The platform surface behind the enrichment pipeline and the agent system.
Two decisions carry this build. Move the reasoning before the call, so the live conversation only ever performs a lookup. And judge the system honestly before deploying it, twice, with two different kinds of judge. Neither is exotic. Both are the difference between a demo at 33% and a production system at 84%.