Databricks Partner Reference Architecture

The Arithmetic Stays in Code

A retirement advisor that answers “when can I access my pension?” across four national systems — for a third of a cent. The model plans the question and writes the answer. It never does the maths. Every tax calculation runs as a governed Unity Catalog function, because a hallucinated withdrawal limit is not a wrong answer, it is a regulatory event.

SectorFinancial services — pension & retirement funds
PlatformDatabricks Agent Framework
ModelsClaude Opus 4.1 · Claude Sonnet 4
PatternAgentic advisory with deterministic tools
$0.003
cost per member query, end to end
18
Unity Catalog functions doing the regulated maths
4
national pension systems, each with authentic rules
8
pipeline phases, traced individually
2
Claude models — one advises, one judges
Chapter 01 / 02

What the Model Does, and What It Must Not

Language models are excellent at understanding a pension question and dangerous at answering one. The architecture is built around that distinction.

The Problem

A member turns 58 and wants to know when she can access her pension. It is a simple question with an answer that depends on her balance, her contribution history, her retirement age under the relevant national scheme, and the tax treatment of each withdrawal option available to her.

Today that question goes to a helpline. She waits, gets a callback scheduled days out, and eventually reaches an advisor who reads from a brochure that does not account for her situation. A simple question takes a week. Six months later, if a dispute arises, there is no record of what she was told.

At fifty thousand members and ten to twelve dollars per call interaction, the routine questions alone run past six million dollars a year — and advisors charging a hundred and fifty an hour spend their time on queries a calculator could answer.

Before
  • 25 minutes on hold, callback three days out
  • Generic brochure advice, not member-specific
  • No record of what advice was given, or on which rules
  • Advisors at $150/hour answering routine questions
  • Costs scale linearly with membership
  • Volume spikes absorbed with overtime, or not at all
The Architecture

Two things happen, and they are kept apart

The intelligence layer runs on Claude Opus 4.1 behind the Databricks Agent Framework. It reads the member’s question, works out what is actually being asked, decides which calculations the answer requires and which member data to retrieve, then synthesises the result into something a person can act on.

What it does not do is arithmetic. Every tax calculation, withdrawal limit and benefit projection executes as a Unity Catalog function — code, versioned and governed, implementing the actual regulatory logic. Eighteen of them, country by country.

This is the load-bearing decision. A model that remembers roughly what the concessional contribution cap was is useless in a regulated advisory setting. Put the rule in a function and the number is either right or it is a bug someone can find. The model orchestrates; the code computes.

MEMBER Question natural language INTELLIGENCE LAYER Claude Opus 4.1 anonymise PII plan the query select the tools results back TOOL LAYER — CODE, NOT MEMORY 18 Unity Catalog functions tax calculators limits & projections ANSWER Synthesis cited personalised PII restored The model never holds a number it did not receive from a function.

Planning and language on one side of the line, regulated arithmetic on the other.

“A hallucinated withdrawal limit is not a wrong answer. It is a regulatory event.”
Privacy

The model works on an anonymised member

Member names are anonymised before anything reaches the LLM and restored afterwards for the personalised response. The model reasons about a case; it does not need to know whose case it is. The personalisation the member experiences is reassembled on the way out, after the reasoning is done.

Coverage

Four national systems, real rules

Each geography carries its own functions implementing that country’s actual regulatory logic — not a generic calculator with a currency symbol swapped.

Australia
Superannuation
ATO tax calculator, Age Pension impact, balance projection
United States
401(k) & Social Security
IRS 401(k) tax, Social Security benefits, RMD calculator
United Kingdom
Workplace & State
HMRC tax calculator, State Pension check, pension drawdown
India
EPF & NPS
EPF tax, NPS benefits, EPS pension, corpus projection
Chapter 02 / 02

A Second Model, Reading the First

In regulated advice, generating the answer is half the job. Being able to show that it was checked, cited and recorded is the other half.

Validation

Claude Sonnet 4 judges before the member sees anything

Every response passes through an independent validation step before delivery. Claude Sonnet 4 reads the generated answer against the retrieved data and the regulatory context, checking accuracy and adherence rather than fluency. It is a different model from the one that wrote the answer, which is the point — a model reviewing its own reasoning tends to agree with itself.

Validation runs in one of three modes: LLM judge, deterministic rule checks, or a hybrid of both. The choice is a configuration decision, so a fund with a stricter compliance posture can dial toward deterministic checking without a rebuild.

The pipeline

Eight phases, each one traced

The agentic pipeline is not a single call. It is eight phases, and each is instrumented individually for duration and cost, which is what makes a slow or expensive query diagnosable rather than mysterious.

Layer 01
Intelligence
Agent Framework with Claude Opus 4.1 — privacy protection, query planning, tool orchestration, synthesis
Layer 02
Tools
18 country-specific Unity Catalog functions for tax, benefits and projections
Layer 03
Validation
Claude Sonnet 4 as judge — accuracy, quality and regulatory adherence
Layer 04
Governance
MLflow tracing and Unity Catalog — audit trail, cost tracking, citation registry
Economics

A third of a cent, itemised

Cost is tracked per query and broken down between the main model and the judge, in real time. Not an estimate at the end of the month — a number attached to the individual answer, which is what makes the running cost of a two-model architecture defensible to whoever signs for it.

Human advisory
$10–12
per call interaction, with advisors at $150/hour and a week’s turnaround on a simple question. At 50,000 members, over $6M a year on routine queries.
Agentic advisory
$0.003
per query, main model plus judge, answered instantly and available around the clock. Volume spikes cost what volume costs, with no added headcount.
Why the cost number matters more than it looks

A three-thousand-fold cost difference does not just save money — it changes which questions are worth answering. At ten dollars a call, a fund discourages members from asking. At a third of a cent, a member can ask six follow-up questions at two in the morning and the economics do not care. That is a different service, not a cheaper one.

Audit

Every interaction reconstructable

MLflow captures traces across the whole pipeline; Unity Catalog governs the functions, the member profile tables and the citation registry. Regulatory references are attached to responses automatically rather than added by an advisor who remembers to.

The dispute six months later — the one that had no record under the old model — now has the question, the data retrieved, the functions called, the rules cited, the validation verdict and the cost. That is the difference between advice you gave and advice you can prove you gave.

Reference

Every Component, in One Place

The platform and model surface behind the advisor.

Component
Role in this build
Claude Opus 4.1
The main agent. Interprets the member’s question, plans which calculations are required, selects and orchestrates tools, and synthesises the final personalised response.
Claude Sonnet 4
The judge. Independently validates every response for accuracy and regulatory adherence before delivery, in LLM, deterministic or hybrid mode.
Databricks Agent Framework
Orchestrates the eight-phase pipeline, binds the Unity Catalog functions as callable tools, and handles the privacy anonymisation step.
Unity Catalog Functions
Eighteen SQL functions carrying the regulated arithmetic — country-specific tax calculators, benefit checkers and projections. Governed, versioned, and the only place a number is computed.
Unity Catalog
Governance across member profile tables, the functions themselves, and the citation registry. One permission model over data, tools and audit.
MLflow
Experiment tracking and trace logging across all eight phases, with per-phase duration and cost metrics, making the pipeline diagnosable and the audit trail complete.
ai_classify
Topic classification on incoming queries, routing the question toward the right country logic and calculation set.
Databricks Apps
Hosts the advisor interface inside the platform, so the UI shares a governance boundary with the models, functions and member data.
Outcomes

What It Changes

Cost to serve
$0.003 a query against $10–12 a call, with routine questions deflected from advisors charging $150 an hour.
Personalisation
Answers computed against the member’s own balance, age and contribution history under their own national scheme — not a brochure.
Compliance
Every interaction logged with its citations, validation verdict and the functions it called. Reconstructable months later.
Scale
Instant responses around the clock, with volume spikes absorbed at marginal cost rather than through overtime.
The reusable part

Nothing here is specific to pensions. Any regulated advisory domain has the same shape: a question that needs understanding, a calculation that must not be improvised, a regulator who will ask what rule you applied, and a cost per interaction that determines how freely customers are allowed to ask. Wealth management, insurance comparison, loan eligibility and investment research all fit the pattern without changing the architecture — only the functions in the tool layer.