• 4 min read •By Tiago Marques

Building Practical Kimball-Style Modeling on Databricks

Kimball-style dimensional modeling remains the industry standard for one simple reason: it works. While the underlying technology has shifted from legacy warehouses to the Lakehouse, the need for a clean star schema hasn’t changed.

However, on Databricks, the hard part isn’t ingestion or compute, it’s the modeling glue. The platform gives you the tools to move data, but the mechanics of surrogate keys, SCD logic, and orchestration order are still left entirely to you. Usually, this means every team ends up re-inventing the same boilerplate code from scratch. And, in a way, teams end up writing tooling to perform integration duties that were achieved through simple GUI-based tools previously, creating a large surface for all sorts of errors to happen.

That recurring led me to create the databricks_kimball_framework. It’s a small, open-source project designed to make dimensional modeling repeatable, letting you focus on the data logic instead of the MERGE syntax.

Why Kimball Modeling Is Difficult on Databricks

Here are the parts that kept biting me:

1. Implementing SCD logic consistently

Delta Lake’s MERGE command is powerful, but implementing SCD Type 1 and 2 patterns correctly—with natural keys, surrogate keys, versioning, effective dates, and deduplication requires a LOT of boilerplate. Most organizations end up reinventing variations of the same code.

2. Handling incremental change feeds

Databricks advocates incremental ingestion, whether through Auto Loader or Change Data Feed (CDF). But turning raw changes into structured dimension updates is not something the platform abstracts for you. That layer still falls to engineers to design and maintain.

3. Managing dependencies between facts and dimensions

Fact tables rely on up-to-date dimension rows. Ensuring that dimensions load first, dealing with early-arriving facts, and maintaining referential consistency are all warehouse fundamentals, but Databricks does not provide modeling-aware orchestration.

4. Schema evolution and physical optimization

Even with Delta Lake’s schema evolution features, determining when and how to propagate field changes and how to tune clustering/compaction is a manual effort.

None of this is “Databricks is bad.” It’s a general-purpose platform. But it does mean classic Kimball modeling takes real engineering effort to implement repeatedly and safely.

Why Auto Loader and Declarative Pipelines Weren’t Enough

Auto Loader is excellent at file discovery and incremental ingestion, but it does not attempt to handle SCD logic, surrogate keys, or dimensional structure. It addresses the ingestion layer, not the modeling layer.

Databricks has also offered declarative pipeline mechanics in various forms (e.g., Lakeflow Spark Declarative Pipelines, materialized views, AUTO CDC). These features are useful, but their capabilities and visibility have shifted over time, and they are not designed as dimensional modeling frameworks. They don’t provide constructs like facts, dimensions, natural-to-surrogate key resolution, or early-arriving fact handling.

So if you want structured, repeatable Kimball patterns, you’re still largely building that layer yourself.

What the Framework Provides

The databricks_kimball_framework offers a simple, declarative way to define dimensional models in Databricks:

  • YAML-based definitions for facts and dimensions
  • Built-in SCD logic (1, 2, 4 and 6)
  • Surrogate key creation and natural-key matching
  • Optional placeholder generation for early-arriving facts
  • Incremental pipelines using CDF and watermarks
  • Optional clustering/optimization settings for Delta tables
  • Audit fields for transparency and debugging

The focus is intentionally narrow: provide consistent, repeatable patterns for Kimball-style modeling without requiring engineers to write and maintain large amounts of repetitive MERGE or orchestration code.

Architecture Overview

Kimball Framework Architecture

The diagram above illustrates the flow from raw data sources through the framework’s dimension and fact processing, ultimately producing a clean star schema in your Delta Lake tables.

How It Compares to dbt

dbt shines as a SQL transformation framework with strong testing, documentation, and community support. Its approach is flexible and well suited to many analytical workloads.

My framework differs in its focus:

  • It provides native SCD handling, whereas dbt requires custom macros or packages.
  • It manages surrogate keys and historical versions, which dbt leaves to user-defined SQL.
  • It is designed specifically for Delta Lake and Databricks incremental processing.

For many teams, dbt and this framework can complement each other. dbt is excellent for wide transformation logic; the framework focuses specifically on warehouse modeling semantics. You might use both—this framework for the SCD/surrogate key mechanics, and dbt for the broader transformation DAG.

Try It Out

The project is available on GitHub:

👉 github.com/accordionjourneyman/databricks_kimball_framework

It is still in beta quality. Not all scenarios have been tested, and bugs are likely to surface as more organizations apply it to different datasets and pipelines. I welcome feedback, issue reports, and contributions from data engineers who are interested in improving dimensional modeling on the lakehouse.

If you’re working with Databricks and you still value clear, well-structured dimensional models, try the framework, and let me know both where it helps and where it breaks.

Tiago Marques
Written by

Tiago Marques

Principal Data Analytics Engineer specializing in cloud data platforms, Databricks, Kimball dimensional modeling, and reliable BI architectures.