When the Answer Is in a Scan Your Search Engine Can't Read: Multi-Modal Clinical RAG on Databricks

A senior orthodontist at a large dental practice sees a new patient presenting with Class III malocclusion, significant crowding, and narrow arches. She knows she has seen a case like this before — possibly several. She knows the treatment outcomes are somewhere in the practice's clinical records. X-rays, treatment notes, progress images, final results. Years of it.

But 'somewhere in the records' is the problem. The images are in one system. The clinical notes are in another. The treatment outcomes are in a spreadsheet someone built three years ago. The only way to find a comparable case is to remember it — or to spend an hour manually searching through folders that were never designed to talk to each other.

This is the state of clinical knowledge management at most medtech organisations in 2026. Not because the data does not exist. Because it exists in silos — images here, text there, outcomes elsewhere — and no search system has been able to bridge them.

Multi-modal clinical RAG on Databricks changes that. Not by organising the folders differently. By embedding everything — images and text — into a single unified vector space where a query in any modality can retrieve results in any other modality.

The Problem: Years of Clinical Data That Cannot Communicate With Itself

Medtech organisations accumulate extraordinary volumes of clinical data over time: patient imaging (X-rays, CBCT scans, intraoral photos, progress images), clinical notes and treatment plans, outcome records, published case studies, annotated clinical gallery entries. The challenge is not data volume. It is data fragmentation:

• Images are in PACS systems, cloud storage, or local drives — accessible to imaging specialists, inaccessible to queries that start with text.
• Clinical notes are in EHR systems — structured around patient records, not clinical similarity or treatment patterns.
• Case gallery data is often manually curated — valuable for learning, but not searchable by clinical indication or morphological similarity.
• Outcomes data lives in spreadsheets or reporting tools — disconnected from the case records that produced them.

The result: a clinician looking for evidence to support a treatment decision — "show me cases like this one, with these indications, and what worked" — is either relying on memory, asking a colleague, or spending time they do not have manually searching systems that were not designed for clinical similarity retrieval.

This is not a search quality problem. It is an architecture problem. And it requires an architectural solution.

The Solution: A Unified Vector Space for Clinical Knowledge

The Databricks HLS multi-modal RAG application solves the architecture problem at its root. Rather than connecting siloed systems with better metadata or smarter keyword search, it takes a fundamentally different approach: embed everything into the same vector space, regardless of modality.

When a clinical image and a clinical text description of the same condition are embedded into the same vector space, they end up close to each other — because they represent the same clinical reality. A text query about 'Class I crowding narrow arches' finds not just text records with those words, but images that represent that clinical presentation.

The Align Dental Assistant application demonstrates all four. A clinician can type 'Class I, Crowding, Narrow Arches' and retrieve both comparable case notes and the actual clinical images from those cases - or upload a scan and retrieve the most similar cases from the gallery complete with treatment notes and outcomes, without writing a single word.

The Technical Architecture: How It Is Built on Databricks

The application is built on a Databricks-native Medallion Architecture — the Bronze / Silver / Gold data pipeline pattern — augmented with a vector store and a RAG-based conversational layer.

The Embedding Model: BiomedCLIP

The foundation of the unified vector space is BiomedCLIP — a biomedical adaptation of the CLIP (Contrastive Language-Image Pre-training) model, trained specifically on biomedical image-text pairs from the scientific literature. Unlike general-purpose CLIP models, BiomedCLIP understands the relationship between clinical images and clinical language — connecting an intraoral photograph to the clinical terminology that describes what is visible in it.

The Medallion Pipeline on Databricks

Bronze layer — raw ingestion. Clinical images (JPEG, DICOM, PNG), clinical text (case notes, treatment plans, gallery descriptions), and metadata are ingested via Databricks Auto Loader into Delta Lake. No transformation — just reliable, ACID-compliant storage with full version history.

Silver layer — processing and embedding. BiomedCLIP processes every clinical record — images and text — and generates embedding vectors. Images are resized and normalised before embedding. Text is chunked and cleaned. Embeddings are stored alongside original records in Delta Lake, with Unity Catalog governing access to clinical records.

Gold layer — vector index and retrieval-ready data. Embeddings are indexed into Databricks Vector Search for approximate nearest-neighbour retrieval. Clinical metadata is structured and enriched for filtering — enabling queries like 'find visually similar cases where the treatment included clear aligner therapy and the outcome was rated excellent.'

RAG layer — conversational clinical intelligence. A RAG-based chat interface sits on top of the vector index. When a clinician performs a search, the retrieval results become the context for the LLM. The chat assistant can then answer follow-up questions about the retrieved cases — grounded in specific clinical records retrieved, not in general medical knowledge.

What Cross-Modal Search Changes for Clinical Decision-Making

Why Databricks HLS Is the Right Platform for Clinical RAG

• HIPAA-ready data governance via Unity Catalog. Every clinical record — image or text — is governed by Unity Catalog from the moment it enters the pipeline. Access control is defined at the data asset level. Audit logs of every data access event are maintained automatically.
• Delta Lake for clinical data integrity. ACID transactions ensure clinical records are never partially written or inconsistently updated. Time-travel capability means the state of any clinical record at any point in time can be reconstructed — critical for regulatory compliance and clinical audit.
• MLflow for model lineage and reproducibility. The BiomedCLIP embedding model version, embedding parameters, and indexing configuration are tracked in MLflow — ensuring the vector index is reproducible and every change is versioned and documented.
• Databricks Vector Search for scalable retrieval. As the clinical gallery grows from thousands to hundreds of thousands of cases, Databricks Vector Search scales without requiring changes to the application layer. Retrieval latency stays sub-second regardless of index size.
• Self-contained data residency. Clinical data does not leave the Databricks environment. The embedding model runs within the Databricks compute cluster. The RAG layer processes only retrieved context — not raw clinical records — limiting the data surface area of any external model API call.

The Broader Application: Where Multi-Modal Clinical RAG Goes Beyond Dentistry

How Reckonsys Would Approach a Multi-Modal Clinical RAG Deploymen

Reckonsys is a Gen AI boutique and Databricks partner. Multi-modal RAG systems — combining embedding pipelines, vector store architecture, retrieval design, and LLM-grounded conversational layers — are a core area of our Gen AI engineering capability. Clinical deployments on Databricks HLS, with the governance and compliance requirements they carry, are exactly the type of technically complex, high-stakes engagement we are built to approach carefully.

Here is how we think about structuring a deployment of this type:

1. Data Audit and Modality Mapping First

Before any embedding model is selected or any pipeline designed, the first priority is understanding what clinical data exists, in what formats, in what systems, and what the access and compliance constraints are. A deployment for a dental practice with 50,000 case images and structured clinical notes has a different architecture than one for a hospital network with DICOM imaging infrastructure and FHIR-compliant EHR data. The data landscape determines the architecture — not the other way around.

2. Embedding Model Selection Is a Clinical Decision, Not Just a Technical One

BiomedCLIP is the right starting point for most biomedical imaging use cases. For highly specialised imaging modalities — retinal imaging, histopathology slides, specific radiological types — domain-specific fine-tuning on the organisation's own image-text pairs may be warranted. The embedding model determines the quality of cross-modal retrieval, and that quality directly affects the clinical utility of the system.

3. Governance Before Retrieval

Unity Catalog access control, data residency configuration, and audit logging need to be designed before the first embedding is computed. In a clinical setting, access control is not a post-launch configuration — it is the constraint that determines what data can be indexed, who can query it, and under what circumstances. HIPAA compliance (for US deployments) and DPDP Act compliance (for Indian deployments) are architectural inputs, not compliance reviews.

4. RAG Grounding as a Patient Safety Requirement

The RAG layer must be explicitly bounded by retrieved context. System prompts must instruct the LLM to answer only from retrieved records, to cite the specific cases its answers are drawn from, and to explicitly state when retrieved context is insufficient to answer a question. This is a clinical safety requirement for any system used in treatment planning or clinical decision support — not a user experience consideration.

5. Measuring Success in Clinical Utility, Not Retrieval Metrics

The right measure for a clinical RAG deployment is not retrieval precision on a benchmark dataset. It is whether clinicians find comparable cases they would not have found otherwise, whether the system reduces time between patient presentation and treatment plan formulation, and whether clinicians trust the RAG layer's answers as grounded in actual clinical records.

Conclusion: The Clinical Knowledge That Has Always Existed, Now Accessible

The orthodontist at the beginning of this piece has the answer she needs somewhere in her organisation's clinical records. The comparable cases exist. The treatment outcomes are documented. The images are stored. The knowledge is there.

What has been missing is not data. It is the architecture to make that data queryable across modalities — to allow a clinical image to find its textual equivalent, and a textual description to find its visual match, in a unified space that treats both as equally valid starting points for a clinical query.

Multi-modal clinical RAG on Databricks HLS provides that architecture. BiomedCLIP provides the unified embedding space. The Medallion pipeline provides the governance and data integrity. The RAG layer provides the conversational interface that makes retrieved knowledge useful at the point of clinical decision-making.

Reckonsys is the Gen AI partner positioned to bring this architecture to medtech organisations — from data audit through embedding pipeline design, vector store configuration, RAG layer implementation, and the compliance infrastructure that makes a clinical AI system trustworthy.

The knowledge has always been there. The architecture to access it is now available.