Two production builds on the Databricks Lakehouse Platform — one unifying a multi-branch services organization's CRM, email, and voice data into a single customer view, the other turning a pile of scanned visa documents into structured, queryable fact tables. Same catalog, same medallion discipline, two very different problems.
Unity Catalog, Delta Lake, Auto Loader, Serverless Compute, Databricks Asset Bundles, and the AI Gateway, stitched into a 360° customer view across three business systems.
A confidence-routed Haiku → Sonnet pipeline on the Databricks AI Gateway, replacing a brittle Lambda/SQS design for classifying visa and enrolment documents at scale.
A CRM, an email server, and a VoIP phone system, each with their own version of the truth about a customer — consolidated into one governed, serverless Databricks lakehouse.
The client runs a national network of branch offices helping students and skilled migrants through visa lodgement, education admissions, and compliance. Every one of those journeys generates data in three disconnected places: a SQL Server CRM behind an operational RDS instance, a mail server handling thousands of counsellor↔applicant threads a day, and a 3CX VoIP system recording every call.
None of the three could see the others. A counsellor could tell you a lead's visa status. They could not tell you, in the same screen, whether that lead had emailed twice this week asking about a delayed document, or called the office sounding frustrated. Reporting meant a one-off SQL export; there was no analytics surface on top of the operational database at all.
The platform ships as four independently deployed Databricks Asset Bundles — CRM, Email, Audio, and a Cost Intelligence bundle — each owning its own bronze → silver → gold schema, its own job schedule, and its own CI/CD path through GitHub Actions. They deploy on their own timelines but read across each other's gold layers at query time, which is how a single Lead 360 view assembles CRM, email, and call data without any one bundle owning the whole picture.
Three source systems, one medallion pattern, one 360° bridge layer joining CRM leads to email senders and call records.
| Databricks component | Role in this build |
|---|---|
| Unity Catalog | Single governance plane across two workspaces (dev/prod). Every job runs as a service principal, not a person — no human credential ever touches the pipeline. Grants are managed as code via migration SQL, cross-catalog reads (e.g. CRM gold reading emails_gold.dim_person) are permissioned explicitly per bundle. |
| Delta Lake | Storage format for all three tiers. Bronze tables are registered as external Delta tables over Fivetran's CDC output; silver and gold are Databricks-managed. Liquid Clustering (CLUSTER BY) replaces manual partitioning on the highest-traffic dimensions (company_id, lead_id for CRM; call_id for audio). |
| Auto Loader | Streams newly landed email JSON and call-transcript JSON from S3 into bronze with schema inference and mergeSchema: true, so new upstream fields are picked up without a manual DDL change. |
| Serverless Compute | All four bundles run exclusively on serverless Jobs and SQL Warehouses — no managed or interactive clusters in any production path. Zero idle spend, but it also rules out .cache()/.persist() and custom pip installs, which shaped how the transforms were written (stdlib + PySpark builtins only). |
| Databricks Asset Bundles | Every job, schedule, permission, and file is defined as code and deployed with databricks bundle deploy through GitHub Actions. Dev and prod run the identical bundle, parameterized by target — no environment-specific branches of the code. |
| Databricks Workflows | Orchestrates the task DAGs — a single CRM job fans out 40 silver tasks in parallel, then 45 gold tasks, then facts and 360° bridges, with per-task retry and dependency-aware pausing on upstream failure. |
AI Gateway (ai_query()) | Email gold calls databricks-claude-haiku-4-5 directly from Spark SQL to summarize *.edu.au correspondence for institution-relationship tracking; the audio pipeline calls the same model via the Foundation Model API to score sentiment, flag critical calls, and classify call direction. |
| System Tables | The cost-intelligence bundle reads system.billing.usage, system.billing.list_prices, and system.lakeflow.jobs directly — no external cost-monitoring tool — to build a daily-refreshed spend dashboard by job, warehouse, and identity. |
Every slowly-changing dimension — leads, companies, products, users, people — carries a five-column SCD Type 2 pattern: a SHA-2 hash of the business key detects a real change, the old row closes with _effective_to, and a new current row opens. Re-running the pipeline with no upstream changes writes nothing, which makes the whole system safely idempotent.
"An empty bridge table silently passing is worse than a loud failure — analysts build dashboards assuming the table is populated."
That's the reasoning behind one deliberate design choice: the cross-bundle 360° join fails the task outright, in both dev and prod, if the email bundle's gold layer isn't there yet — rather than silently building an empty bridge. Below that, a quarantine pattern catches rows that would otherwise violate a NOT NULL contract downstream: they're routed to a paired _rejects table instead of being dropped or crashing the run, so the pipeline stays green while nothing gets lost.
Because the platform is serverless end to end, every job run is a metered line item in system.billing.usage — which turned the cost-dashboard bundle into a genuine optimization tool rather than just a reporting exercise. Auditing real run history against schedule surfaced two concrete findings:
The same billing-driven analysis is now standing engineering practice: before proposing a schedule or a warehouse, look at what the last N runs actually cost against what they actually changed.
Visa and enrolment documents arrive in every format a CRM upload widget allows. A confidence-routed, two-model pipeline on the Databricks AI Gateway turns them into structured Delta tables — with a human always one click away when the model isn't sure.
Every visa lodgement and every education admission is backed by paperwork: passports, bank statements, academic transcripts, offer letters, compliance forms. Counsellors upload it all through the CRM into a shared S3 bucket, in whatever format it arrives — a scanned PDF, a phone photo, a Word document, an Excel bank statement export.
The prior design — a Lambda/SQS/DynamoDB pipeline — worked, but every new document type meant new glue code, and there was no visibility into what a failed extraction actually cost or why it failed. The brief was to replace it without losing anything on cost, while gaining structured, per-document-type warehouse tables and a real audit trail.
Every non-image format is normalized first — Word documents render to PDF via headless LibreOffice, then every page becomes a PNG — so a single vision-capable model interface handles the entire format matrix. From there, the pipeline is a two-model cascade behind the Databricks AI Gateway:
~93% of documents are read once by Haiku 4.5; the ~7% below a 70% confidence threshold get a 300 DPI re-render and a second pass from Sonnet 4.6 before anything is written.
| Databricks component | Role in this build |
|---|---|
| AI Gateway | Hosts two governed endpoints — a primary databricks-claude-haiku-4-5 route and a databricks-claude-sonnet-4-6 fallback — so the document images never leave the Databricks security perimeter in ap-southeast-2 to reach a model provider directly, and switching the underlying model is a UI config change, not a code deploy. |
| Inference Tables | Auto-log every gateway call — tokens in, tokens out, model used, latency — straight into Unity Catalog. That gave the pipeline per-document cost attribution the prior Lambda design never had. |
| Auto Loader | Ingests normalized pages as they land, replacing the DynamoDB-based dedup layer from the previous architecture with a Delta-native dedup table. |
| Delta Lake | 32 per-document-type bronze tables (passport, bank statement, transcript, offer letter, and so on) plus a dedicated document_audit_log table that captures every failure with its stage, error type, and a 3-hour pre-signed S3 URL for a human to pull the original file. |
| Delta Live Tables | Runs the gold transform hourly, building gold_client_profile, gold_document_status, and gold_compliance_readiness declaratively on top of the 32 bronze tables. |
| Unity Catalog | Governs who can query document_audit_log and generate a fresh pre-signed download link — the audit query itself filters out anything already reprocessed, so a resolved failure stops generating new links automatically. |
A document that fails at any stage — normalization, the LLM call, a rate limit, a confidence score that never clears 70% even after the Sonnet pass — lands in document_audit_log with enough detail for a staff member to act on it: the failure stage, the error type, the model that was tried, and a working link to the original file. A manual reprocessing job lets that person key in the correct fields directly against the target bronze table, closing the loop without an engineer in the room.
Multi-page documents are handled the same way a person would read them: batches of up to 20 pages go to the model together with context ("page batch 2 of 3 of the same document"), fields are merged across batches, and the reported confidence is the average across all of them — so a 40-page bank statement doesn't get judged on its first page alone.
The full platform surface spanning both builds — governance, storage, compute, orchestration, and AI, all inside one lakehouse.
CLUSTER BY on the highest-traffic join keys (company_id, lead_id, call_id) so Databricks SQL prunes files automatically, without hand-managed partitioning.ai_query()) and the Foundation Model API — model swaps are a config change, and every call stays inside the workspace's cloud perimeter.system.billing.* and system.lakeflow.jobs power a daily cost-intelligence dashboard built entirely on platform-native metadata — no third-party FinOps tool.