Note: “AI rumination” is not a formally recognized construct in clinical psychology or psychiatry. Here, the term is used informally to describe a pattern of rumination and reassurance-seeking that can be amplified by AI-generated output. It is not a diagnostic label.

Why I Call It AI Rumination

There’s a familiar mental pattern where a thought doesn’t resolve. You go over it again, slightly differently each time, expecting that the next pass will unlock something new.

In psychology, this is called rumination: a repetitive, difficult-to-disengage form of thinking that doesn’t converge on an outcome.

Large language models interact with this pattern in a particular way. They don’t close the loop—they expand it. Every prompt generates more plausible explanations and angles that feel like progress without actually resolving anything. The result is a feedback loop that increases cognitive load and prolongs indecision.

I am not naming AI rumination as a clinical category, but rather as a shorthand for a pattern that becomes easier to understand once you’ve experienced it.

When the Loop Took Over

The Bandmate

A former bandmate stopped appearing at rehearsals for months—no explanation, no messages, just silence.

I kept replaying past conversations, trying to figure out what had gone wrong. The question became a loop: “What did I do wrong? Nothing, right?” Instead of letting the uncertainty sit, I kept feeding it into an AI chatbot, looking for reassurance that I hadn’t missed something obvious.

When I finally confronted him, the response was abrupt. I ended the band and cut ties on the spot. The lack of communication had felt like the central issue, and in the moment, I felt justified in reacting strongly.

With distance, though, I can see how much rumination had amplified the intensity of my response. There was no clear alignment between that reaction and any rational or strategic goal for preserving the three years we had invested in the band.

Later, I learned he had been planning to leave town and simply hadn’t communicated it. With that context, my reaction feels disproportionate in hindsight. The band could likely have continued with a replacement, and the friendship might have been preserved. People move on from projects all the time.

The Architecture Decision

In a previous team, I respectfuly disagreed with the engineering manager about a core data modeling approach, that would shape years of data pipelines and reports to come.

I had (and still have) a strong belief that the Ralph Kimball methodology represents one of the most robust approaches to data warehousing. With that framing, I turned to an LLM and asked it to take on the role of a “grumpy Kimball purist” and argue against the alternative.

It did exactly that. The argument was clean, confident, and hard to refute. I revisited it multiple times, refining prompts and strengthening the case. I reinforced my prior knowledge of Kimball practices, and my assumptions that it was the absolute best fit.

What I didn’t account for in AI prompting were the surrounding constraints: the team’s skill set, management direction, and my colleague’s experience managing different systems and tools. The solution that moved forward wasn’t what I would have chosen. It wasn’t theoretically optimal, but it was good enough to meet most of the company’s reporting needs.

There’s also a structural reality in team environments: not every decision is yours to make. Influence has boundaries. If you’re not part of the final decision-making process, responsibility doesn’t fully sit with you either. That doesn’t mean disengaging or withholding input. It means recognizing where your role ends. If things don’t work out, your understanding doesn’t disappear, but neither does the distribution of accountability.

Looking back, the issue was that I used AI to ruminate on a position that had already formed, rather than to test its limits or understand the trade-offs more fully, and I ended up wasting time in the process. I lost count of how many times I revisited the approach, repeatedly stress-testing it in my head and with prompts, even guesstimating the amount of money potentially wasted on storage and compute.

In this particular case, the worst outcome of my rumination was definitely cynicism. I found myself internally expecting the alternative to fail, as if that would validate my original view or prove my worth. In that frame, it becomes easy to mentally split approaches into “how we do things here” versus “the right way”, even when reality is more constrained and less absolute.

Seeing the Loop in Action

The mechanisms behind it are well understood and some individuals may be more prone than others.

• Intolerance of uncertainty: ambiguity is uncomfortable, so the mind keeps searching for resolution
• Reassurance-seeking: asking for confirmation even when no new information is available
• Cognitive biases: especially confirmation bias and the illusion of explanatory depth
• Metacognitive beliefs: the assumption that continued analysis will eventually produce certainty

With AI, most questions return an answer that is coherent, structured, and convincing. Over time, it becomes harder to recognize when you’re actually learning versus just circling the same idea from different angles.

You spend more time reviewing generated arguments than moving forward. Decisions stretch longer than they should. Confidence starts to come from how persuasive an answer sounds, rather than how well it reflects reality.

When nothing feels resolved, the instinct is to try again: rephrase the prompt, explore another angle, generate a stronger argument. The loop repeats.

That pattern doesn’t always stay contained. It can show up in team discussions as well—bringing increasingly refined questions or critiques about decisions that have already been explored. The intention is usually to improve outcomes, but the effect is often different. When repeated without new information or clear alternatives, constant questioning tends to create friction without changing direction. Conversations slow down, alignment becomes harder, and the signal-to-noise ratio drops.

Over time, this can start to affect trust. What feels like careful analysis on one side can be experienced as churn on the other. Teams may become less responsive, not necessarily because the questions are invalid, but because they don’t appear to lead anywhere actionable.

Reflection vs. Rumination

Not all repeated thinking is a problem.

Reflection is usually anchored to something: evidence, a decision, a next step. It tends to move toward closure.

Rumination feels different. It repeats without resolving. The question shifts slightly each time, but the underlying uncertainty stays intact.

The system tends to align with the framing you give it. Not perfectly, but enough that it reinforces the direction you’re already leaning toward.

This is functionally similar to reassurance loops seen in anxiety-related patterns. The act of asking provides temporary relief, but it doesn’t resolve the underlying uncertainty.

The important distinction is that this is a behavioral pattern, not a clinical condition. The similarity is structural, not diagnostic.

Interrupting the Loop

In practice, a few things tend to help:

• Deciding in advance how long you’ll explore a question before acting
• Writing down what would actually change your mind before asking the model
• Bringing another person into the conversation, especially when you notice you’re looking for agreement rather than information

The shift is subtle but important: from generating more interpretations to testing reality.

On Adversarial Prompting

Using AI to explore worst-case scenarios can be useful in a controlled context.

But when used as a personal check, it tends to amplify negativity, giving disproportionate weight to unlikely negative outcomes.

Over time, that can skew perception in a way that reinforces the loop rather than breaking it.

Where AI Actually Helps

None of this means AI-driven iteration is inherently problematic.

Used well, it can:

• expand the solution space
• accelerate learning
• generate hypotheses that can be tested

When It Becomes a Problem

There isn’t a clean boundary, but certain signals tend to show up together:

• You keep revisiting the same question without getting closer to a decision
• The process increases tension instead of reducing it
• You delay acting because you feel like you’re “almost there”

At that point, the issue is usually difficulty tolerating the uncertainty that remains.

Like most cognitive patterns, this one is easier to recognize in hindsight than in the moment.

Parting Thought

AI reflects what you bring into it. When used deliberately, it can sharpen thinking. When used to resolve uncertainty that can’t be resolved through more analysis, it tends to do the opposite.

Recognizing that difference is less about discipline and more about awareness. Once you see the rumination pattern, it becomes easier to step out of it.

I say this as someone who lived that lifestyle. A decade ago I devoured every article on the Asian Efficiency blog, cycled through countless note-taking apps, installed every Pomodoro timer and distraction-free writing screen I could find. If there was a system that promised to optimize how I worked or thought, I tried it.

Structure is a tool, not a cage

I used Obsidian for years and built an extensive PARA-style system for almost everything: journaling, monthly self-reviews, tracking progress on personal projects, structuring my engineering work, even scripting music practice sessions. I wouldn’t start any task without a plan or attend a meeting without pre-structuring my thoughts. I was getting things done, but I wasn’t enjoying the process—and I began to question why.

I even built a personal dashboard that pulled data from my watch, my practice sessions, and my activity logs across work and side projects—a single pane of glass for my own productivity. It felt like the ultimate system. Then I added a metric based on total hours worked per week across everything: my job, side projects, and music development. I was averaging well over 70 hours. The WHO and ILO consider 55 hours per week a serious health hazard, linked to a 35% higher risk of stroke and 17% higher risk of heart disease. That number was the wake-up call I needed to stop living inside the cage I’d built for myself.

The irony wasn’t lost on me. The system I’d built to support thinking had quietly become a prerequisite for it. If the structure wasn’t there, neither was the thinking. In a way it short-circuited my ability to just sit, reflect, and explore ideas freely.

When the system becomes the driver

Over time I noticed something deeper: the more I structured my vault, the more I constrained my intuition and intrinsic motivation. What was meant to support reflection turned it into a prerequisite-driven process: a pipeline of templates and reviews that had to be “fed” before any real thought could happen.

That’s the risk I rarely see acknowledged in the productivity space. People talk about the benefits of structure endlessly, but almost nobody talks about what happens when the structure starts driving you instead of the other way around. I’m fairly certain I’m not the only one who ended up there.

The joy of forgetting

When I started taking notes more sparingly, something unexpected happened. The anxiety of losing ideas was replaced by the joy of rediscovering them. Not every thought deserves to be captured immediately. Some need time to mature, to be forgotten and remembered again, reshaped by whatever you’ve lived through since.

Letting go of compulsive capture helped me relearn how to remember. It made me less dogmatic in how I process information and more inclined to think through problems only when they’re genuinely ready to be worked through, not because a template demanded it.

There’s a name for this in psychology: incubation. Letting ideas rest before consciously revisiting them often leads to richer, more original thinking. The note-taking orthodoxy, with its emphasis on capturing everything immediately, can actively work against that process.

Scarcity isn’t what it used to be

There’s a broader context worth acknowledging. A decade ago, the value proposition of dense personal knowledge bases was obvious: information was scattered, retrieval was hard, and the person who had it written down won. That scarcity justified elaborate systems.

Today, that equation has shifted. LLMs and modern search have made retrieval near-trivial for most technical knowledge. Dense notes on APIs, frameworks, and configuration details are redundant. What retains value is insight, synthesis, and perspective: the things that come from thinking, not from filing.

A rigid vault optimized for information scarcity can actually become a liability in an era of information abundance. The hours spent maintaining it would be better spent developing the kind of judgment and creative thinking that no retrieval system can replicate.

Systems should serve thinking, not replace it

None of this is an argument against note-taking, it’s a tool I still use, just more selectively. The argument is against letting any system dictate the process of thought itself.

Use structure for what it genuinely supports: capturing fleeting ideas that would otherwise be lost, planning work that benefits from a checklist, logging progress that future-you will thank you for. But don’t let the system become so elaborate that maintaining it crowds out the unstructured reflection where real insight happens.

Overengineering productivity can undermine the very goals it’s meant to serve. That’s not a warning I read somewhere. It’s something I lived.

For artists who don’t rely on arranging tracks to earn a living, AI tools like Suno are a fascinating playground. With a click, you can hear dozens of variations of an idea that previously existed only as a hum or sketch. For casual users, it opens a doorway to music creation—but this accessibility comes with a trade-off: when polished tracks can be generated in seconds, the “product” itself becomes less scarce.

The Shift: Arrangement to Melody

Through experimenting with Suno, I noticed something surprising: melodies and harmony have become more important than arrangement.

Historically, arrangement was a bottleneck. In professional DAWs like Ableton or Logic, building a professional arrangement requires enormous skill—choosing instruments, layering sounds, designing transitions, and structuring sections. Many great melodies never reached their potential simply because the production couldn’t match the idea. A skilled professional could easily charge 1,000–5,000 euros for even a relatively simple arrangement, putting polished production out of reach for many creators.

AI flips this scarcity. Tools like Suno or Udio make arrangement cheap and abundant. You can try dozens of instrumentation styles in minutes. When arrangement is infinite, the melody and harmony carry the emotional signal.

Some producers have realized this and are learning to use AI tools to enhance their business and satisfy customers. But it’s only a matter of time before AI production becomes so widespread that clients will bypass producers entirely. This abundance of polished tracks increases the pressure on professional artists, who now face a world where technical execution is no longer a barrier to entry—success increasingly depends on originality, emotional impact, and the ability to connect with listeners.

The Counterpoint: World Music as the Antidote

In many ways, arrangements have already been “solved” through centuries of human music-making. World music—from West African polyrhythms to Indian classical ragas, flamenco guitar, or Georgian polyphony—offers styles refined generation after generation, with emotional textures, dynamic progressions, and cultural resonance that remain compelling.

AI can emulate these styles convincingly, but if it leads to uniformity or boredom in mainstream music, it may be a signal to rediscover the richness of world traditions. Exploring these musical lineages provides fresh inspiration and reminds us that arrangement is not just technical—it carries culture, identity, and human memory.

Reviving the Graveyard

Many of us have a “graveyard” of musical sketches—unfinished fragments, voice memos, or DAW projects that never saw the light of day.

In the age of generative AI, these sketches feel more like melodic seeds than failures. Where turning a sketch into a full track once required hours of arrangement and mixing, AI now provides the ecosystem for those seeds to grow almost instantly. By feeding a melodic concept into Suno, I can explore dozens of styles, discovering which arrangement makes the melody truly come alive. My archive no longer feels like a graveyard—it’s a vault of Idea DNA, waiting for the right context to flourish.

The Economics of Abundance

The flipside of AI abundance is exposure scarcity. Today, with AI making polished music more accessible than ever, the barrier to production disappears—but competition for human attention intensifies.

The supply of music has exploded while the attention of listeners remains finite, reducing the commercial scarcity of polished tracks. For professional musicians, this intensifies the emotional and economic pressure to stand out, as skillful production alone is no longer enough. The real currency in this landscape is context, connection, and live experience.

Two Realities for Artists

This abundance affects artists differently. For those creating music as a means of expression and purpose, external validation is secondary. We compose to explore ideas, process emotions, and find meaning—so AI abundance doesn’t threaten our sense of purpose.

For professional musicians whose livelihood depends on exposure, streams, and commercial success, the landscape is far more precarious. The pressure to “stand out” in a world of infinite supply is real, and the scarcity of attention and original ideas can create both emotional and financial strain.

Music is About Context and Connection

Despite the rise of AI-generated music, the human element remains irreplaceable. Music is not just notes and chords—it’s about context, emotion, and human connection. Live performance, collaboration, and the shared experience of music will never disappear. AI may help generate tracks, but it cannot replicate the energy of a live crowd or the intimacy of a song sung for someone who needs it.

It usually starts with an “AI-first Monday.” Your CEO watched a YouTube video over the weekend about how autonomous AI agents are going to replace entire engineering departments. Now, every new feature must be an “agent,” every process must be an “A2A (Agent-to-Agent) workflow,” and deterministic code is suddenly viewed as legacy software.

I fell for it, too.

I spent several weeks building a system called “Git Diff RAG” in my spare time: an over-engineered, multi-agent code reviewer designed to automate pull request (PR) reviews end-to-end. It was a massive, sprawling architecture of hallucinating text generators. And I built it directly into the trap of LLM Sycophancy.

When you ask an LLM, “How should I design a multi-agent RAG system for code review?” it enthusiastically helps you build it. It suggests architectures, orchestrators, and vector databases. It never stops to ask the most important question: “Do you actually need this?”

To be clear: having some automated review system: even a flawed one: is better than having none. It is vastly superior to the “LGTM” rubber-stamp culture or the non-existent coding hygiene that plagues so many teams. The goal of this project was learning and sharing those findings, and I certainly learned a lot. But we can do much better than building over-engineered systems out of probabilities.

One antidote to this sycophancy trap? Adversarial Prompting. Try to roast your own idea using AI. If I had asked a model to act as a grumpy veteran coder and ruthlessly deconstruct my A2A plan, I probably would have saved myself weeks of over-engineering. Since this was a personal project, I could afford that luxury: though my wife, family, and music teachers might disagree. But what happens when you build these sprawling systems on company time?

2. The Over-Engineered System (What I Built)

On paper, “Git Diff RAG” was a masterpiece of modern AI hype.

I ingested Git diffs, ran them through a RAG layer filled with vector search hits, architecture trees, and build tool output. I built a parliament of AI “personas”: an Architect agent, a Reviewer agent, a Security expert, a Frontend expert. The idea was that these agents would debate the diff, reach a consensus, produce revision artifacts, update the specs, and rewrite the documentation.

When you combine multiple steps of analysis, modification, and documentation without accountability gates, you institutionalize subtle defects. If an AI agent proposes a change, another AI agent approves it, and a third AI agent writes the spec to match the change… who is responsible when production breaks?

And that assumes the system acts logically. The worst footgun I experienced was due to a lack of guardrails. Early on, I ran the agent against a “vibe-coded” job search assistant I was actively using for real applications. My adversarial prompt had instructed the AI to ensure the code was “GitHub-ready.” The agent reviewed the code, concluded the idea wasn’t valuable enough to exist, and slipped an obscure PowerShell deletion command into its suggestions. I blindly accepted it, instantly wiping out my entire project.

Interestingly, when I asked the AI how to prevent this, the sycophancy kicked back in. Instead of telling me to pause and reconsider why an autonomous code reviewer had unrestricted write access to a local file system in the first place, it happily suggested continuing with complex CLI SDKs, configuring read-only flags, and mounting virtual file systems. It focused entirely on engineering a solution to the problem instead of asking: “Wait, what was the original premise here?”

Another weird idea suggested by the LLMs was an “intelligent prompt selection module.” The theory was that the system could dynamically compose the exact right prompt at runtime based on the code context. I ended up generating hundreds of prompt snippets for this system, and at some point, I was completely lost in my own architecture.

I didn’t build an AI code reviewer. I built a castle of AI prompts.

3. Accountability: The Line Between Recommenders and Cancer Diagnoses

AI is inherently probabilistic. That is perfectly fine for tasks with rapid feedback loops and low stakes: like A/B testing ad copy, building recommender systems, or even some high-frequency micro-trading. The cost of being wrong is effectively zero.

But when an autonomous system blindly merges a data pipeline change that drops a production table, or worse, is diagnosing a patient with cancer, the math completely changes.

Code review is not just about catching typos; it is an accountability gate. It is a governance checkpoint where an engineer says, “I authorize this change.” Treating code review like a generative task meant to be fully automated erases the vital boundary between critique, decision, and institutional authorization. High-consequence errors require a deterministic, auditable system.

4. Lost in the Middle: The Physics of Context Pollution

Why did all those “expert” agents produce such shallow, noisy, and often hallucinated reviews? Because my agents were shallow precisely because I was drowning them in context.

I threw architecture diagrams, Jira tickets, repository metadata, and full build logs into the prompt alongside the diff. But LLMs do not get smarter with more context; they get smarter with higher signal density.

This phenomenon is documented in the Stanford/Berkeley paper “Lost in the Middle: How Language Models Use Long Contexts”. The researchers found that LLMs heavily index on information placed at the very beginning or the very end of a prompt. If you bury the actual important detail (the code diff) in the middle of a massive RAG blizzard of irrelevant vector search hits, the model’s attentional blind spot takes over. My over-engineered architecture was effectively an expensive way to blind the model to the very code it was supposed to review.

In review systems, precision beats volume. An LLM that highlights a single critical SQL performance bottleneck with certainty is far more valuable than one that generates a checklist of twenty “best practice” hallucinations because it got distracted by unrelated documentation. Every marginally relevant file you add acts as camouflage for the bug you’re actually trying to catch.

5. The Antidote: Deterministic Workflows over Emergent Chaos

After two months of drowning in emergent chaos, I reached for the axe. I killed the agents, purged the vector database, and burned the RAG layer to the ground. I deleted the entire repository and started from zero.

I was about to give up, convinced that my time spent on a worthless idea could not be reclaimed. Then I realized I had learned something valuable: the limitations of LLMs and the importance of deterministic workflows. Determined to do better, I decided to build the simplest thing that could possibly work using a procedural approach. I tested it relentlessly with adversarial prompting until I felt confident I wouldn’t repeat the same mistakes.

I replaced the whole architecture with a simple .review folder full of plain text prompts and a deterministic pipeline, that you can add on a per-project basis.

Instead of independent agents talking to each other, the pipeline became a sequence of operations that outputs verifiable, timestamped artifacts. Inside the .review directory, we added a runs/ folder. Every time a review is triggered, it creates a timestamped subfolder (e.g., runs/2026-03-16_10-15-30/) that stores the exact intermediate file produced by each step of the LLM timeline.

This means you can audit exactly what the model saw and what it output at every phase:

1. Scan: Identify risk areas based on the diff (e.g., does this touch auth? does this alter a core schema?).
2. Review: Produce claims about the code, but only with solid evidence. If a claim lacked a code snippet or a quoted spec, it was ignored.
3. Refine: Remove weak or duplicate findings.
4. Summarize: Output actionable conclusions for a human engineer to review.

The core principle: Keep the workflow deterministic. Use LLMs for semantic extraction and analysis within strictly bounded steps, but require human accountability gates for any state changes. And most importantly, keep the receipts.

6. Practical Application: A .review Folder for Data Engineering

How does this actually look in practice? It looks like a folder of highly specific, role-based prompts that are invoked by a simple wrapper script (like a GitHub Action or a local Python script) based on the files touched in the PR.

Here are simplistic examples of how you segment the reasoning instead of relying on one massive, confused agent. It starts with a Meta Prompt: essentially the system instructions for your orchestrator. That defines the workflow and enforces artifact persistence.

meta.prompt

You are executing a deterministic, multi-stage pull request review. You will adopt three subsequent personas to analyze the provided diff.

IMPORTANT SAFETY RULE:
- You have read-only access to the repository.
- You may NOT modify any files outside the `.review` folder.
- All outputs, intermediate files, and artifacts must be written only to `.review/runs/{{timestamp}}/`.

Workflow:
1. Load `architect.prompt`. Perform structural analysis. Save output to `architect_findings.md`.
2. Load `reviewer.prompt`. Perform code-level review on the output of step 1. Save output to `reviewer_findings.md`.
3. Load `verifier.prompt`. Audit all previous findings against the diff evidence. Save output to `verified_findings.md`.

Once all three files are written to the runs folder, output a final `summary.md` containing only the surviving, verified findings for human review. Do NOT approve or reject the code yourself.

architect.prompt

You are analyzing a data engineering pull request. Your goal is structural analysis.
- Identify if this change affects DAG dependency trees.
- State whether the new data transformations maintain table idempotency.
- Highlight any breaking schema changes (dropped columns, altered types).
- Provide evidence for your claims from the provided diff. 
Do not approve or reject the code. Only state your findings.

reviewer.prompt

You are conducting a code-level review on a SQL/dbt pull request.
- Focus on SQL performance: Look for missing partition filters, expensive JOINs, or window function abuse.
- Identify potential data quality issues: Are there protections against NULL values or fan-outs?
- Check test coverage: Are there `not_null` and `unique` tests applied to newly created fundamental columns?
Quote the exact lines of code from the diff that support your findings. If you have no findings, output "NO_FINDINGS".

verifier.prompt

You are an adversarial verifier. 
Review the output of the previous analysis alongside the original diff. 
For every finding reported:
1. Does the evidence in the diff match the claim?
2. If the answer is no, mark the finding as "HALLUCINATION" and remove it.
You are the final filter before a human sees this review.

7. Conclusion: The Real Value of AI

After building a complex AI review system that completely spiraled out of control, the best solution was embarrassingly simple: deterministic workflows, structured prompts, and curated evidence—closer to the scientific method than the hype.

One simple question could have saved me: before greenlighting a multi-agent RAG swarm, I should have asked, “What is the simplest deterministic workflow that solves this?” If you can’t answer that first, adding AI agents will only get you to the wrong destination faster.

I had chased the hype, eager to learn multi-agent systems as a supposed silver bullet. In the end, it’s often wiser to let the dust settle and see which design patterns actually stand the test of time. Remember: you won’t be fired for skipping the latest AI trends.

After listening to the Éthiopiques recordings on repeat countless times, I eventually explored the Jerusalem EP. There I felt a different kind of connection with the music, and I immediately began to imagine the possibility of arranging some of these pieces for my accordion.

Technically speaking, the writing in these works seemed particularly suited to the instrument. The left hand often takes on a more rhythmic role rather than serving purely as counterpoint. There is less repetition and less sense of teleological deferral, and the musical narrative feels stronger, while still carrying Emahoy’s distinctive sense of rubato. It felt closer to a classical language, though I would not claim expertise in saying that. These are not necessarily better or worse compositions; rather, they seemed to resonate naturally with my own musical language and with the possibilities offered by the standard-bass accordion, the instrument I play.

From that moment, I began working on arrangements of two pieces from the EP: “Movement from Rainbow Sonata” and “Jerusalem.”

For two weeks, night after night, I listened to the pieces and wrote the scores note by note. Then came the challenge of making them work on the accordion. At times I had to make difficult decisions: whether to remain faithful to the score and to the spirit of certain moments, or to allow myself to drift away from it so the music could live more naturally on the accordion.

Since I do not consider myself a virtuoso musician and like to always have other people’s opinions, I brought these pieces to my accordion lessons and worked on them week after week. Gradually, the arrangement was refined through interpretative guidance: adjusting dynamics and using accordion-specific elements such as register switches to shape the color of the music. The goal was to evoke Emahoy’s spirit in a way that felt true to the connection I experienced in her music.

I took the time to let the pieces and the interpretation mature. From beginning to end, the process took nearly a year before the final video was completed. In fact, it could have happened a few months earlier, but I had been hoping to record in a monastery near my home. Unfortunately, that permission never came.

In the end, I realized that you do not need a grand or sacred place to celebrate someone’s legacy. All you need is something humble and sincere. That is what I tried to do here, grounded in my own effort, sensibility, and limitations.

You can watch the final cover below:

The sheet music for this arrangement is also available now on the Emahoy Music Publisher store.

In Part 1, we established that specification-driven development excels at generating physical infrastructure. You can write a configuration file, and a deterministic parser can reliably generate staging models, apply basic transformations, and deploy tests.

But extending this same basic configuration approach to core business logic reveals its limitations.

The “Semantic Gap” is the fundamental difference between specifying a physical constraint and defining business reality. Flat configuration files cannot capture the interwoven complexity of dimensional modeling—a challenge mature data teams address using dedicated Semantic Layers.

Physical Contracts vs. Semantic Reality

To understand the semantic gap, we must first look at what deterministic tooling already does well. Modern data teams use strict data contracts (e.g., schema.yml in dbt or native data platform contracts) to enforce the physical structure of a table:

# A physical data contract
models:
  - name: fct_transactions
    columns:
      - name: transaction_id
        data_type: integer
        constraints:
          - type: not_null
          - type: unique
      - name: customer_id
        data_type: integer
        tests:
          - relationships:
              to: ref('dim_customers')
              field: customer_id
      - name: amount_usd
        data_type: numeric(18,2)

This ensures column types are correct, nulls are prevented, and referential integrity holds. If the contract is violated, the pipeline fails. This is deterministic, automatable governance at the infrastructure layer.

The Metadata Layer: Catalogs and Information Schemas

There is also an incredibly valuable intermediate layer between raw physical tables and complex semantic query engines: the Data Catalog.

By querying a modern data warehouse’s information_schema (or using tools like dbt docs, DataHub, and Alation), engineering teams can automatically extract referential relationships, primary keys, and constraints. Adding table- and column-level comments directly to the underlying SQL views or Python scripts means the Catalog can expose an always-up-to-date, highly visible version of the data model to the whole company.

But there is a critical limit here: a data catalog provides discoverability, but it doesn’t quantify logic.

It informs a human analyst that amount_usd is a number, but it doesn’t define the mathematical rules that prevent a BI query from returning incorrect results.

The Nuances of Semantic Modeling

A true dimensional model—the kind that analysts can explore safely without generating silent errors—must encapsulate complex business logic. Here is what a simple physical spec fails to articulate:

1. What Is the Grain of This Fact Table?

Grain is the atomic level of your fact table. It answers the question: What does one row represent?

Consider this definition:

“One row per transaction line item, per product, per day. Natural key: (product_id, transaction_id_line). Surrogate key: revenue_line_sk.”

Aggregating this table by product_category without respecting its grain can produce subtle, catastrophic errors. Analysts must understand the correct joins and cardinalities to avoid fan-outs or double-counting, which flat specifications cannot enforce.

2. How Do You Handle Slowly Changing Dimensions (SCDs)?

Imagine a product gets recategorized from “Electronics” to “Accessories” on March 15th. Does historical revenue from February shift to Accessories, or does it stay in Electronics?

“Historical transactions use the category as it existed at the time of purchase (Type 2 SCD). Only transactions after March 15th use the new category.”

SCDs dictate how history is written. A physical spec knows that a category_id exists. But a semantic model must understand the temporal relationship between the fact event and the dimension state. If a system doesn’t understand that a join must include an active_daterange qualifier, the history of the data warehouse rewrites itself every time a source system updates a record.

3. Is ‘amount_usd’ Fully Additivity?

Additivity determines what aggregations are mathematically valid:

If an analyst writes SELECT SUM(unit_price) because a basic spec didn’t prevent it, the database will return a meaningless result. Defining additivity upfront prevents invalid aggregations and protects analysts from silent errors.

4. What Are the Conformed Dimensions?

“This fact table conforms to dim_product (Enterprise v2.0) and dim_date. All domain-specific facts must join through these conformed dimensions to ensure cross-departmental reporting matches.”

If the marketing team builds a fct_campaign_ROI table and creates their own local dim_product, their reports will inevitably disagree with the finance team’s reports. Specifying the physical columns doesn’t solve this; the organization needs an architectural constraint—a conformed dimension strategy—to ensure everyone is measuring the same entities.

5. What Is the Business Definition of “Revenue”?

“Revenue = settled_amount + pending_adjustments, excluding refunds issued > 30 days post-purchase. Do NOT include internal transfers.”

A column named amount_usd hides edge cases that matter to accounting and compliance. Generic documentation often fails to capture the conditional logic required for business accuracy.

Governance requires a translation layer between the raw database and the business user. Enterprise BI tools like Cognos or BusinessObjects provided this, but their rigidity eventually drove teams toward the Modern Data Stack. We replaced heavy, GUI-driven tools with version-controlled SQL, but in the process, we lost the semantic abstraction itself.

Data teams are now re-establishing these guardrails using code-first Semantic Layers like Cube (a headless BI platform) or dbt Semantic Layer.

The Reporting Tool “Solution”

Modern BI tools like Power BI or Tableau offer sophisticated semantic layers, but embedding logic directly into visualization tools creates vendor lock-in and decentralized silos. Business rules become inaccessible to downstream consumers like data scientists using Python or analysts querying via other platforms.

A centralized, code-first Semantic Layer addresses this by establishing the abstraction below the BI tool and above the database, serving as a governed source of truth for all consumers.

These tools succeed because they don’t just specify physical columns—they introduce domain-specific abstractions designed expressly for semantic concepts. Let’s look at how a mature semantic layer handles additivity and join paths:

// A Semantic Layer configuration (e.g., Cube)
cube(`transactions`, {
  sql_table: `public.fct_transactions`,

  joins: {
    products: {
      relationship: `many_to_one`, // Prevents fan-out errors
      sql: `${CUBE}.product_id = ${products.product_id}`,
    },
  },

  measures: {
    revenue: {
      sql: `amount_usd`,
      type: `sum`,           // Explicitly fully additive
    },
    avg_price: {
      sql: `price`,
      type: `avg`,           // System prevents invalid SUMs
    },
  },
});

Here, the abstraction understands the business rules:

• Aggregation type is codified: The system structurally prevents SUM(price).
• Joins declare cardinality: many_to_one tells the generation engine how to handle the grain correctly during a join.

Mapping the Control Boundaries: Spec vs. Code

To return to the original promise of spec-driven development—where does it actually fit? What does the spec control, and what does the code control?

In a mature, correctly layered data architecture, you use different tools to govern different phases of execution:

1. The Physical Spec (Data Contracts / schema.yml)
   • Controls: Database DDL. It enforces what shapes the data takes (types, nullability, unique keys, hard referential integrity).
   • Goal: Fails the pipeline if bad data arrives.
2. The Execution Code (SQL / Python)
   • Controls: The actual data transformations. It takes raw data and shapes it into dimensional models, handling edge cases and data cleansing.
   • The Micro-Spec (Inline Comments): Code isn’t just execution; it’s also the most accurate source of business logic. A one-line comment (-- Exclude test accounts prefix 'QA_') explains why a filter exists. If written using a strict convention (e.g., Javadoc style, dbt doc blocks, or Sphinx-SQL), these comments can be programmatically extracted by a CI/CD process to automatically populate your data catalog. This ensures the granular “why” stays perfectly synced with the execution code.
   • Goal: Populates the physical tables with correct historical state and embeds the micro-reasoning alongside the logic.
3. The Metadata Spec (Data Catalogs / information_schema)
   • Controls: Discoverability and human understanding. It surfaces descriptions, owner assignments, and extracted relationships.
   • Goal: Tells a human analyst what the table means.
4. The Semantic Spec (Cube / dbt Semantic Layer)
   • Controls: Runtime BI query generation. It maps exactly how a BI tool is allowed to aggregate or join facts to dimensions dynamically.
   • Goal: Prevents analysts from generating technically correct SQL that returns mathematically wrong numbers.

When a team tries to use a flat “Physical Spec” to govern “Semantic Rules” (e.g., feeding a schema.yml to an LLM and hoping it generates valid BI logic), they are using the wrong abstraction for the layer.

The Semantic Layer as a Service

Rather than forcing AI to rediscover business logic from raw tables, we can expose the semantic model as a structured service. Agents gain access to human-authored definitions—metrics, grain specifications, and join paths—that are unavailable in standard database catalogs.

Tools like dbt-mcp or Cube’s REST/GraphQL APIs enable this by providing structured endpoints for semantic metadata. This context allows an agent to understand that revenue requires specific aggregation rules, preventing common errors like double-counting during joins.

The architecture looks like this:

By interacting with the semantic model, consumers leverage pre-defined logic rather than attempting to derive it from first principles. The choice of protocol (MCP, REST, GraphQL) is incidental; the value lies in the human-curated semantic asset.

Federated Semantic Contracts

While this pattern is distinct from Data Mesh, it shares the core principle of self-serve consumption. Downstream consumers—whether AI agents, BI tools, or analysts—interact with a governed interface rather than writing arbitrary SQL against raw tables. The interface enforces domain rules without requiring consumers to manage the physical implementation.

This federated semantic contract pattern works whether the semantic layer is centralized or domain-owned. In both cases, the contract surface is the semantic model, and the interaction happens through a structured interface.

The Verdict

Use deterministic specifications like Data Contracts to govern infrastructure and physical pipelines. Use a semantic layer to govern business logic and provide a consistent API for both humans and machines.

By separating the governance of the shape of data from the governance of its meaning, you build a system that is both technically reliable and business-accurate.

To see how to capture the “why” behind these human decisions and automate the review of your documentation, check out BI Docs: Automated Documentation for Data Platforms.

This post is part of the series Spec-Driven Development in BI: A Pragmatic Series.

At its core, spec-driven development in Business Intelligence is a simple idea: instead of writing hundreds of SQL files and test scripts by hand, you write a configuration file (a specification) and let automation generate the code for you. Write the spec once, generate the pipelines everywhere.

When executed well, it makes data pipelines faster to build, more maintainable, and remarkably consistent.

But here is the reality: specs work brilliantly for some parts of your analytics stack and create massive friction for others.

The key insight—and the one that dictates whether automation helps or hurts—is understanding the difference between infrastructure and exploration.

The Essential Distinction: Infrastructure vs. Exploratory

Before anything else, you need to answer this question about any given piece of analytics work:

Is this infrastructure (stable, widely-reused, high-impact) or exploratory (fast iteration, low risk, transient)?

Your answer determines everything.

Infrastructure Characteristics

• Changes quarterly or less
• Used by many downstream dashboards or teams
• High impact if broken (CFO’s scorecard goes dark, 50 people can’t do their jobs)
• Small group maintains it (data engineers, not analysts)
• Risk of failure is critical (breaking changes cascade across reporting)

Examples: conformed dimensions, core fact tables, shared KPI definitions.

Exploratory Characteristics

• Changes weekly or daily
• Used by one person or team
• Lower blast radius if wrong (one dashboard needs a fix, easier to revert)
• Many people touch it (analysts, business users)
• Risk of failure is deceptive (wrong metric → wrong decision, often compounding silently for weeks before anyone notices)

Examples: one-off dashboards, quick analyses, metrics under active discussion.

The Honest Big-Tech Lesson

Large-scale data organizations do use spec-driven approaches. Airbnb’s Minerva centralizes metric definitions into a single semantic layer that all dashboards consume. Uber’s uMetric enforces metric governance so teams can’t redefine “trip” differently. LinkedIn’s Unified Metrics Platform standardizes how metrics are computed, stored, and served across products.

The common pattern across all of them is separating tiers while enforcing governed exploration:

1. Infrastructure layer (specs, strict governance) – Data engineers maintain pipelines, schemas, and conformed dimensions using deterministic specs (e.g., schema.yml, data contracts).
2. Semantic layer (governed metrics) – A unified, tool-agnostic layer (like Cube or dbt Semantic Layer) where metrics are defined explicitly.
3. Exploratory layer (flexible, but within guardrails) – Analysts iterate and build one-off dashboards, but they do so by querying the semantic layer, not by writing rogue, ad-hoc SQL that re-invents metric definition logic.

The spec discipline lives in the infrastructure and semantic layers. The exploratory layer stays flexible, but never bypasses the governed metrics.

Promotion Pathways: When Exploration Becomes Infrastructure

The infrastructure/exploratory split isn’t static. Today’s one-off dashboard becomes tomorrow’s CFO scorecard. You need a deliberate promotion pathway—a process for hardening an exploratory artifact into governed infrastructure:

1. Trigger: The dashboard is referenced by more than one team, is cited in a recurring meeting, or has survived two quarterly planning cycles without being retired.
2. Promotion checklist: The analyst (or sponsoring team) opens a PR that moves the logic into the governed layer: defines the metrics in the semantic layer, adds data quality tests, writes a DDR documenting the grain and business rules, and links the dashboard to the conformed dimensions.
3. Governance change: Once promoted, changes to the underlying model require code review and CI validation—the same bar as any other infrastructure artifact.

Without an explicit promotion process, you end up with shadow infrastructure: exploratory work that is infrastructure in practice but has none of the governance. This is how the “two dashboards, two answers” problem starts.

Where Specs Actually Shine

✅ Boilerplate Generation (The Staging Layer)

The staging layer (stg_*) is where you pull raw data from source databases and apply only mechanical transformations:

• Rename columns: CUST_ID → customer_id
• Cast types: VARCHAR → STRING
• Add UTC timestamps
• Apply basic null handling

A tool or LLM can generate 50 perfect staging models in seconds by reading the source schema. No complex business logic. No grain mismatches.

Why it works: This is syntax, not semantics.

✅ Metric Definition (The Semantic Layer)

You cannot write a universal spec that generates Snowflake SQL, Power BI DAX, and LookML simultaneously—those paradigms are incompatible. But you can create a single, version-controlled metric definition that multiple tools consume.

Here is what this looks like using dbt Semantic Layer / MetricFlow—not a homegrown schema, but the actual syntax a mature tool uses:

# dbt Semantic Layer — MetricFlow metric definition
semantic_models:
  - name: transactions
    defaults:
      agg_time_dimension: transaction_date
    entities:
      - name: transaction
        type: primary
        expr: transaction_id
      - name: customer
        type: foreign
        expr: customer_id
    dimensions:
      - name: transaction_date
        type: time
        type_params:
          time_granularity: day
    measures:
      - name: transaction_amount
        agg: sum
        expr: amount_usd

metrics:
  - name: daily_revenue
    description: Sum of settled transactions, per day, excluding internal accounts
    type: simple
    type_params:
      measure: transaction_amount
    filter: |
      {{ Dimension('transaction__account_type') }} != 'internal'

MetricFlow renders this into the correct SQL dialect for your warehouse. Different BI tools consume it. The key difference from a generic physical spec: MetricFlow’s schema enforces entities, grain, and aggregation type—it won’t let you build a metric that silently double-counts.

Why it works: You prevent silent metric drift. You don’t necessarily eliminate the “Finance says $10k, Sales says $12k” discrepancy—often, those are legitimately different metrics from different bounded contexts (e.g., booked vs. recognized revenue). But a semantic spec forces them to have distinct, governed definitions (sales_bookings vs finance_recognized_revenue) rather than colliding as ambiguous revenue columns.

✅ Test Automation (Data Contracts)

If a stakeholder says “Transaction ID must be unique,” a script can automatically generate the data quality test:

constraints:
  - column: transaction_id
    test: unique
    critical: true

You’re not writing complex logic—you’re translating explicit data-quality rules into deterministic test code.

Why it works: Automation for explicit contracts, not implicit semantics.

✅ Preventing Scope Creep: Infrastructure Specs

Apply specs ruthlessly to:

• Conformed dimensions (dim_product, dim_date, dim_customer) – change quarterly
• Core fact tables (fct_transactions, fct_events) – spine of the warehouse
• Shared KPI definitions (revenue, churn, LTV) – 50+ downstream dependencies

Don’t use rigid infrastructure specs for:

• One-off dashboards
• Exploratory analysis (as long as it queries the semantic layer)
• Quick iterations while stakeholders pivot
• Local analyst work

The Golden Rule

Use automation for syntax, not for semantics.

Automate:

• Boilerplate (stg_ models, naming conventions, casts)
• Test generation (from explicit constraints)
• Metric definitions (version-controlled, tool-agnostic)
• Staging scaffolding (from source schemas)

Do NOT automate:

• Grain decisions (what’s a “daily revenue” row?)
• SCD logic (product category changes)
• Dimensional modeling (conformed dimensions)
• Complex business rules (how to calculate adjusted revenue)

Those require conversation. Code generation can draft them; human review is mandatory.

Where This Sits in the Ecosystem

If you’re reading this in 2026, you should know: the data contracts movement already solves many of these problems with deterministic tooling.

Where spec-driven development fits

Spec-driven dev is a lightweight bridge for teams that aren’t ready for full data contract infrastructure. If your org:

• Doesn’t use an orchestration and testing framework natively yet
• Has no schema registry or contract framework
• Needs a low-friction starting point

If you are just modernizing a legacy platform, then writing infrastructure-level specs and generating staging models from them is a pragmatic first step. But understand the ceiling: as your maturity grows, you’ll likely replace custom specs with native data contracts and a semantic layer tool.

Scope honesty

This approach is SQL-ecosystem specific. The physical specifications described here generate SQL. They don’t generate DAX, LookML, or anything outside the SQL paradigm. If your BI stack spans multiple paradigms, specs help within each one but don’t bridge them.

Orchestrating Spec-Driven Code Generation: Determinism is Mandatory

If you want to automate the workflow (spec → code → orchestration pipeline), you must rely on deterministic AST parsers and templating engines, not probabilistic LLMs.

Using an LLM to hallucinate SQL for core infrastructure based on a physical specification is an architectural anti-pattern. It injects non-determinism, fragility, and silent semantic failures into a tier that requires absolute predictability.

Instead, use tools designed for safe, programmatic scaffolding:

1. Jinja Macros: Many tools allow you to write macros that loop through a graph object and dynamically generate SQL for staging layers.
2. Code generators: Use established packages or scripts to generate baseline models deterministically from source schemas.
3. Alternative dialects (e.g., PRQL): If SQL is too verbose for metric definitions, compile simpler declarative languages into SQL deterministically.

Here is a simple example of a Python script using Jinja2 to generate a staging model from a spec:

# Minimal example of deterministic code generation (No LLMs)
import yaml
from jinja2 import Template

def generate_staging_model(spec_file: str):
    with open(spec_file, 'r') as f:
        spec = yaml.safe_load(f)

    # A strict Jinja template guarantees output structure
    sql_template = Template("""
    WITH source AS (
        SELECT * FROM {{ "{{ source('" + source_name + "', '" + table_name + "') }}" }}
    ),
    renamed AS (
        SELECT
        {%- for col in columns %}
            cast({{ col.name }} as {{ col.cast_type }}) as {{ col.rename | default(col.name) }}{% if not loop.last %},{% endif %}
        {%- endfor %}
        FROM source
    )
    SELECT * FROM renamed
    """)

    return sql_template.render(
        source_name=spec['source']['database'],
        table_name=spec['source']['table'],
        columns=spec['columns']
    )

# The output is 100% predictable: same spec = same SQL, every time.

The workflow is straightforward and safe:

1. Define your spec (YAML, JSON, or any structured format)
2. Pass it through a deterministic templating engine
3. Write the generated code to your models/ folder

This works well for mechanical scaffolding. It shouldn’t be used for complex logic involving grain, SCDs, or conformed dimensions—topics we’ll cover in Part 2.

Optional: Architectural Guidance Files

If you want to formalize architectural standards that your LLM or code generator should follow, you can create a lightweight guidance file (e.g., skills.md). This is not required, but it helps enforce consistency:

# Architectural Standards for Data Models

## Naming Conventions
- Staging models: `stg_` prefix
- Facts: `fct_` prefix 
- Dimensions: `dim_` prefix
- All files: snake_case

## Column Standards
- Primary keys: `{model_name}_id` (e.g., `customer_id`, `transaction_id`)
- Created/updated timestamps: `created_at`, `updated_at` (UTC)
- Boolean flags: `is_` prefix (e.g., `is_active`, `is_deleted`)
- Amounts: always `decimal(18,2)` or higher precision

## Testing Rules
- All primary keys: `not_null` + `unique` tests
- All foreign keys: `relationships` tests
- All fact amounts: `not_null` (no NULL amounts)
- Fact tables: at least one test on the grain (e.g., deduplication check)

## Documentation
- Every model must have a description block
- Every column must have a description block
- Fact tables must document their grain explicitly

Important Caveat

This guidance file helps when humans review generated code, but it doesn’t enforce anything automatically. An LLM will sometimes ignore it. Your real enforcement layer is:

• CI linting (fail the build if fct_ tables don’t match the naming pattern)
• Data quality tests (mandatory tests that fail if violated)
• Code review (humans check that generated code follows the standards)

The guidance file is a reference, not a guarantee.

What Goes Wrong

The post assumes:

“Specs will make everything faster and more consistent.”

But when you apply specs universally, you get:

• Year 1: Great! Specs for critical KPIs. Adoption and enthusiasm.
• Year 2: Marketing wants a one-off Customer Lifetime Value dashboard. Do you mandate a spec PR?
• Year 3: Half your dashboards have specs, half don’t. Specs feel like bureaucracy for the explorer-analysts because the platform’s Developer Experience (DX) is poor.
• Year 4: Nobody remembers why specs exist. Adoption dies. You’re back to hand-written SQL with extra configuration files gathering dust.

The drift happens because you created a false dichotomy: rigorous infrastructure on one side, and unsupported ad-hoc SQL on the other. Good data engineering provides tools (like a robust semantic layer) that make doing the right thing easier than doing the wrong thing. Exploration shouldn’t mean abandoning guardrails; it should mean exploring through the semantic layer.

The Framework for Decision-Making

Before you commit a BI artifact to spec-driven development, ask:

1. Is this infrastructure or exploration?

   • Infrastructure → specs are worth the overhead
   • Exploratory → let it stay flexible

2. Will this change frequently?

   • Rarely → specs pay off (you reuse them)
   • Frequently → specs create friction

3. How many teams/dashboards depend on this?

   • Many → specs reduce coordination overhead
   • One or two → not worth the complexity

4. Is this a rule, or still being negotiated?

   • Settled rule → spec it
   • Still being figured out → explore via the semantic layer’s flexible interfaces, hardening into a spec only when the logic stabilizes

Next Steps

If you’re starting small, this is the roadmap:

1. Identify your infrastructure layer – which tables/metrics are stable, widely-used, and rarely change?
2. Apply specs to the staging layer – this is the easiest win and requires almost no semantic modeling.
3. Define shared KPIs as specs – this prevents the “two dashboards, two answers” problem. This is the second-highest ROI.
4. Empower analyst work with semantic layers – Don’t mandate rigid infrastructure specs for exploratory analysis, but give them a semantic layer that makes exploring governed metrics easier than writing ad-hoc SQL.
5. Expose the semantic layer to AI agents – Once your semantic layer is governing your metrics, AI agents can query it directly through structured interfaces like MCP, REST APIs, or GraphQL endpoints (e.g., dbt-mcp, Cube’s REST API). The agents generate staging models, trace lineage, and run tests—all while respecting the grain and business rules you defined. The protocol is plumbing; the semantic contract is what matters.

Next: Part 2: The Semantic Gap: Specifying Business Reality

This post is part of the series Spec-Driven Development in BI: A Pragmatic Series.

Over the years, I’ve watched critical knowledge accumulate in the worst possible places: email threads, meeting transcripts, Slack messages, and in the heads of people who eventually leave. Thresholds that determine how revenue gets bucketed. Cutoff dates baked into SQL queries. Business rules that exist because someone in Finance asked for them in 2022 and nobody wrote it down.

When I decided to leave my last role, I didn’t want to be that person who walks out with half the platform’s context in their head. So I built a system to capture it, organize it, and keep it honest after I was gone.

I call it BI Docs. It’s not a product. It’s a pattern: a way to build and maintain up-to-date documentation for data platforms using MkDocs and a lightweight AI review layer. The stack I used was Databricks and Power BI, but the approach is portable to any combination of ETL and reporting tools.

Phase 0: Building the Baseline

The hardest part of any documentation effort isn’t the tooling. It’s the initial excavation.

When I started, the “documentation” for our data platform was scattered across hundreds of emails, dozens of Jira tickets, a few architecture powerpoints that were two years out of date, and a SharePoint site that nobody visited. The first job was to gather all of that into a single place and make sense of it.

This is where AI proved genuinely useful. Not as a writer, but as a librarian. I used markitdown to convert documents from all the usual formats (Word, PDF, PowerPoint, HTML emails) into clean Markdown that an LLM could parse. The workflow looked like this:

1. Collect everything. Emails from stakeholders containing requirements. Jira tickets describing feature requests and bug fixes. Architecture diagrams. Meeting notes. SharePoint pages. Database schemas.
2. Convert to Markdown. Using markitdown, I transformed these documents into a uniform format. This is the critical step. LLMs work best when they can read plain text, and Markdown is a practical intermediate format that both humans and machines handle well.
3. Feed and filter. I used GitHub Copilot to help me extract the actual facts from the converted documents: business rules, column definitions, threshold values, timeline decisions. The AI accelerated the extraction, but I reviewed and curated every single page. There is no shortcut here. AI will confidently present irrelevant or wrong information next to useful facts, and only someone who understands the domain can tell them apart.

The output of Phase 0 was a structured MkDocs repository containing everything the team needed to understand the platform. The AI didn’t write the documentation. I wrote it, using AI as a high-speed research assistant.

The Ground Truth: Data Decision Records (DDRs)

Before we jump into the tooling, we have to talk about format. You cannot just dump a 30-page requirements document into a repository and expect an automated system (or a human) to parse it effectively when code changes.

The documentation needs to be specific, isolated, and structured. To achieve this, I used Data Decision Records (DDRs).

A DDR is a lightweight markdown file committed to Git that captures one specific architectural or business logic decision. It’s an adaptation of Michael Nygard’s Architecture Decision Records (ADRs), but tailored for data platforms.

When a junior analyst looks at a strangely complex UNION ALL in a fact table and thinks, “I can optimize this,” the DDR is the single file that explains exactly why that join exists (e.g., a recurrent accounting restatement process mandated by Finance).

A DDR has three core sections:

1. Context: The problem and the environment (e.g., “Sales targets arrive on the 5th, but Finance closes on the 3rd.”)
2. Decision: The exact logic implemented to solve it.
3. Consequences: The known downstream impacts of this decision.

By forcing documentation into this highly granular DDR format, the AI review layer (which we’ll cover shortly) has a much easier time mapping a specific Git diff to a specific business rule.

In practice, I recommend storing DDRs under docs/requirements/ddrs/ with a filename convention like 20260223-fiscal-year-cutoff.md. That keeps them easy to sort chronologically while still being specific enough to reference from reviews.

Here’s a minimal DDR template:

# DDR-20260223: Fiscal Year Cutoff Logic

## Context
Finance closes the month on the 3rd, but sales targets arrive on the 5th.

## Decision
Use the previous month's targets until the current month's targets are loaded.

## Consequences
Dashboards show a two-day lag at the start of each month.
Any change to this timing must be reflected in the Services dashboard docs.

Getting Started with MkDocs

If you haven’t used MkDocs before, it’s a static site generator designed specifically for project documentation. You write Markdown files, organize them in folders, and MkDocs builds a clean, searchable website. Combined with the Material for MkDocs theme, the result looks professional with almost zero effort.

Installation

MkDocs runs on Python. To set up a new project:

# Install MkDocs and the Material theme
pip install mkdocs mkdocs-material

# Create a new documentation project
mkdocs new bi-docs
cd bi-docs

# Start the local dev server
mkdocs serve

This gives you a live preview at http://127.0.0.1:8000 that updates every time you save a file.

Configuration

The mkdocs.yml file at the root of your project controls everything. Here’s a starting point:

site_name: BI Platform Documentation
theme:
  name: material
  palette:
    scheme: default
    primary: indigo
    accent: teal
  features:
    - navigation.sections
    - navigation.expand
    - search.suggest
    - search.highlight
    - content.code.copy

plugins:
  - search

markdown_extensions:
  - admonitions
  - pymdownx.details
  - pymdownx.superfences
  - pymdownx.tabbed:
      alternate_style: true
  - tables

nav:
  - Home: index.md
  - Data Sources: data_sources/index.md
  - Data Models: data_models/index.md
  - Requirements: requirements/index.md
  - Timelines: timelines/index.md
  - Glossary: glossary/index.md
  - Dashboards: dashboards/index.md
  - Architecture: architecture/index.md
  - Review: review/index.md

Building for Production

When you’re ready to publish:

# Build the static site
mkdocs build

# The output goes to the /site directory
# Deploy to GitHub Pages, Azure Static Web Apps, or any static host
mkdocs gh-deploy  # shortcut for GitHub Pages

The Repository Structure

The folder hierarchy should reflect two things: the logical structure of your data platform and the team structure of your organization. Different sections will have different audiences. A business analyst cares about glossaries and dashboard definitions; a project manager cares about requirements and timelines.

Crucially, do not duplicate your column-level definitions. Modern data engineering dictates that documentation lives close to the code (e.g., in table comments, a schema registry, or a data catalog). BI Docs and MkDocs should be reserved for high-level concepts: business requirements, Data Decision Records (DDRs), architecture standards, and timelines. Do not create a fact_sales.md in MkDocs that just lists columns, as you guarantee that documentation will rot.

Here’s the structure I settled on:

bi-docs/
├── mkdocs.yml
└── docs/
    ├── index.md                    # Landing page with platform overview
    │
    ├── data_sources/               # Where the data comes from
    │   ├── index.md                # Summary of all source systems
    │   ├── crm_system.md           # CRM ingestion details
    │   ├── erp_financials.md       # ERP feed documentation
    │   └── third_party_api.md      # External API contracts
    │
    ├── data_models/                # Architectural modeling decisions (NOT column specs)
    │   ├── index.md                # Modeling conventions and standards
    │   └── architecture_notes.md   # Broad architectural decisions
    │
    ├── requirements/               # Why the data looks the way it does
    │   ├── index.md                # How to read and contribute requirements
    │   ├── req_fiscal_year.md      # Fiscal year cutoff logic and its history
    │   └── req_product_flags.md    # Product categorization rules
    │
    ├── timelines/                  # When decisions were made
    │   ├── index.md                # How the timeline works
    │   └── 2024_decisions.md       # Chronological log of platform decisions
    │
    ├── glossary/                   # What the terms mean
    │   ├── index.md                # Glossary conventions
    │   └── business_terms.md       # Revenue, Net Sales, Active Customer, etc.
    │
    ├── dashboards/                 # What the business sees
    │   ├── index.md                # Dashboard inventory
    │   ├── executive_dashboard.md  # Measures, filters, upstream dependencies
    │   └── regional_sales.md       # Regional breakdown report
    │
    ├── architecture/               # How the platform is built
    │   ├── index.md                # Architecture overview
    │   ├── etl_patterns.md         # Standard ETL patterns (incremental, full)
    │   └── platform_norms.md       # Naming conventions, branching strategy
    │
    └── review/                     # Where the AI review system lives
        ├── index.md                # How the review process works
        ├── meta.md                 # The orchestration prompt
        ├── architect.md            # The "Architect" persona prompt
        ├── reviewer.md             # The "Reviewer" persona prompt
        └── runs/                   # Review run history
            └── 2026-02-23T10-00/
                └── review.md       # Findings from this review run

A Note on Audiences

Not everyone needs to read everything. When I structured the docs, I kept this mapping in mind:

The Automated Layer: Keeping Docs Honest

Phase 0 gets you a comprehensive documentation baseline. That’s valuable on its own, but documentation rots. The moment someone changes a column name in Databricks or updates a DAX measure in Power BI without touching the docs, the baseline starts to drift.

The automated layer is designed to catch that drift. The idea is straightforward: when someone makes a change to the ETL code (Databricks notebooks, SQL files, PySpark scripts) or the reporting layer (Power BI semantic model files via Git integration), the system compares the change against the existing documentation and flags mismatches.

It doesn’t fix the code. It doesn’t rewrite the docs autonomously. It raises a flag and says: “This change affects documented behavior. Here’s what you should review.” The developer always makes the final call.

The Architecture

Here’s the high-level flow:

The diagram shows a simple sequence: trigger, context assembly, architect pass, reviewer pass, and a final timestamped report. If you present this pattern internally, keep that caption with the image so the flow is still understandable in screen readers, exports, and chat previews.

The system supports two entry points. Git Diff Mode computes the diff between the current branch and main, reacting to code changes. Priority Mode takes a plain-language focus area (e.g., “validate docs against the Services dashboard”) and proactively audits a specific part of the platform.

Regardless of the entry point, the pipeline is the same:

1. The Context Builder assembles a context bundle: the trigger (diff or priority text), the full source code from the target repos (Databricks, Power BI), and the relevant documentation from the MkDocs repository (data models, requirements, dashboards).
2. The Architect (architect.md) receives the full context bundle. It scopes the analysis by identifying which code files and which documentation pages are affected, and provides impact cues. Its output is appended to the context bundle.
3. The Reviewer (reviewer.md) receives the enriched context bundle—everything the architect identified, plus the original code and docs. It evaluates the findings, assigns severity, cites evidence, and writes suggested actions.
4. The final output is written to a timestamped review.md under .review/runs/.

There are multiple ways to trigger this: Git hooks running locally, CI/CD pipeline steps on pull requests, or scheduled jobs. For documentation reviews, I’d recommend keeping a human in the loop—have the system generate the review, but let the engineer decide what to act on. Automated gates that block merges based on AI findings introduce friction that teams tend to disable after the first false positive.

If you want a lightweight CI starting point, a pull request workflow can be as simple as this:

name: bi-docs-review

on:
  pull_request:

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Generate BI Docs review
        run: |
          python scripts/run_bi_docs_review.py \
            --mode git \
            --base main \
            --output .review/runs/${{ github.event.pull_request.updated_at }}/review.md

      - name: Upload review artifact
        uses: actions/upload-artifact@v4
        with:
          name: bi-docs-review
          path: .review/runs/

The exact runner script will depend on your setup, but the shape should stay the same: compute the diff, build the context, run the two prompts, and persist the review as an artifact instead of silently discarding it.

The Meta-Prompt: A Two-Agent Review System

Rather than a single monolithic prompt, the review system uses two separate agent personas that run sequentially. One identifies the affected area, the other evaluates the impact.

Setting Up the .review/ Folder

The review system lives inside your BI Docs repository:

.review/
├── meta.md          # Orchestration: which prompts to run and in what order
├── architect.md     # Agent 1: Identifies affected documentation
├── reviewer.md      # Agent 2: Evaluates impact and writes findings
└── runs/            # Output directory for review results
    └── .gitkeep

meta.md — The Orchestrator

This file defines the review pipeline. It tells whatever tool you’re using (Copilot CLI, a custom script, a CI/CD action) what to do:

# BI Docs Review Pipeline

## Usage

Execute this prompt with one of the following:

- `execute meta.md with git diff`
  Runs a change-driven review. The pipeline computes the diff
  between the current branch and main, then identifies which
  docs are affected.

- `execute meta.md with priority: <description>`
  Runs a target-driven review. For example:
  `execute meta.md with priority: validate docs against the Services dashboard`
  `execute meta.md with priority: audit the fiscal year cutoff logic`

## Inputs
- `target_repos`: List of repository paths to monitor
- `docs_repo`: Path to the BI Docs MkDocs repository

## Steps — Git Diff

1. **Identify Changes**
   - For each target repo, compute `git diff HEAD main`
   - Filter to relevant file types: `.py`, `.sql`, `.dax`, `.bim`, `.tmdl`
   - Execute `architect.md` with the diff as context

2. **Evaluate Impact**
   - Take the output from Step 1 (list of affected doc pages)
   - Load the content of those doc pages from the MkDocs repo
   - Execute `reviewer.md` with the diff + doc content

3. **Write Results**
   - Save the reviewer's output to `runs/<timestamp>/review.md`
   - List affected files and flag severity (info, warning, critical)

## Steps — Priority

1. **Scope the Priority**
   - Execute `architect.md` with the priority text as context
   - The architect identifies which codebase files and which
     documentation pages are relevant to the priority,
     and provides impact cues for the reviewer

2. **Evaluate Alignment**
   - Load the identified code files and doc pages
   - Execute `reviewer.md` with the code + doc content
     and the original priority as context

3. **Write Results**
   - Save the reviewer's output to `runs/<timestamp>/review.md`
   - Include the priority text as a header for traceability

The two modes serve different purposes. The git mode is reactive—it runs after a change and catches drift. The priority mode is proactive—you use it when you want to validate a specific area of your platform against the documentation, for example before a stakeholder presentation or after onboarding a new team member who will own a particular dashboard.

architect.md — The Identifier

This agent adapts its behavior based on the mode. In git mode, it maps code changes to documentation. In priority mode, it scopes the review by identifying which parts of the codebase and which docs are relevant to the stated priority.

# Role: Data Platform Architect

You are reviewing a data platform against its BI documentation.

## Mode: Git (Change-Driven)

When provided with a git diff:
- Identify which documentation files are potentially affected.
- Look for changes to column names, data types, table structures,
  join conditions, filter logic, and hardcoded values.
- Map each change to the most specific documentation page.
  For example, a change to the `dim_customer` table maps to
  `docs/data_models/dim_customer.md`.
- If a change introduces or modifies a business rule (e.g., a
  fiscal year threshold), map it to the relevant requirement
  in `docs/requirements/`.
- If a backend change affects columns used by a Power BI measure,
  flag the relevant dashboard page in `docs/dashboards/`.

## Mode: Priority (Target-Driven)

When provided with a priority (e.g., "validate docs against
the Services dashboard"):
- Identify which codebase files are relevant to this priority.
  For example, if the priority mentions a dashboard, find the
  ETL notebooks that feed its upstream tables and the semantic
  model files that define its measures.
- Identify which documentation pages cover this priority.
  Pull from `docs/dashboards/`, `docs/data_models/`,
  `docs/requirements/`, or any other relevant section.
- Provide impact cues to the reviewer: which code-to-doc
  relationships are most likely to have drifted for this
  specific area.

## Output Format
Return a list of:
- **Affected code files** (with a one-line explanation)
- **Affected documentation pages** (with a one-line explanation)
- **Impact cues** for the reviewer (what to look for)

reviewer.md — The Evaluator

This agent receives the diff, the affected doc pages, and the current documentation content. It writes the actual findings:

# Role: Senior Data Engineering Reviewer

You are conducting a documentation impact review.

## Your Task
Given a code change and the current state of the affected
documentation, evaluate whether the docs need to be updated.

## Constraints
- Do NOT summarize the code. Identify specific deviations between
  the code change and what the documentation currently states.
- Check for these patterns:
  1. **Grain changes**: Does this change alter the documented
     row-level granularity of a table?
  2. **Hardcoded values**: Does this change introduce a date,
     threshold, or constant that should be documented or
     externalized to config?
  3. **Cross-layer coupling**: Does a Databricks change require
     a corresponding update in the Power BI semantic model
     or vice versa?
  4. **Naming mismatches**: Has a column or measure been renamed
     in code but not in the docs?

## Output Format
For each finding:
- **Severity**: info | warning | critical
- **Location**: The file and line range in the code
- **Documentation Page**: The affected .md file
- **Finding**: What the deviation is
- **Suggested Action**: What the developer should do

Example Review Output

Here’s what a review run produces. This file gets saved to .review/runs/2026-02-23T10-00/review.md:

# BI Docs Review — 2026-02-23T10:00

**Repos reviewed:** databricks-etl (HEAD: abc1234 vs main: def5678)
**Files changed:** 3 | **Findings:** 2

---

## Finding 1

- **Severity:** critical
- **Location:** `notebooks/silver/sales_transform.py` L45-47
- **Documentation Page:** `docs/requirements/req_fiscal_year.md`
- **Finding:** The fiscal year cutoff date was changed from
  `2024-07-01` to `2025-07-01` directly in the notebook.
  The current documentation states the fiscal year boundary
  is `2024-07-01`. Additionally, this value is hardcoded
  rather than pulled from configuration.
- **Suggested Action:**
  1. Update `req_fiscal_year.md` with the new date and the
     rationale for the change.
  2. Consider moving the cutoff date to a config file or
     Databricks widget parameter to avoid future hardcoding.

---

## Finding 2

- **Severity:** warning
- **Location:** `notebooks/gold/fact_sales.py` L112
- **Documentation Page:** `docs/data_models/fact_sales.md`
- **Finding:** A new column `discount_flag` (BOOLEAN) was added
  to the `fact_sales` Gold table. This column is not yet
  documented in the data model page, and no corresponding
  DAX measure exists in the Power BI semantic model.
- **Suggested Action:**
  1. Add `discount_flag` to `docs/data_models/fact_sales.md`
     with its definition and business meaning.
  2. Coordinate with the BI team to determine if a dashboard
     measure is needed.

What This Isn’t

There are a few things this system does not do.

It’s not a compiler or an AST parser. For teams working in highly structured environments with strict schema registries, a deterministic validation layer (like a custom linter or schema contract tool) will always be more reliable than an LLM. BI Docs is designed for environments where that kind of tooling doesn’t exist yet and may never exist—where the data platform is a collection of notebooks, SQL files, and Power BI datasets held together by institutional memory.

It’s not a knowledge graph. The folder structure is flat and simple on purpose. Knowledge graphs are powerful, but they require significant ongoing maintenance. A set of Markdown files in Git is something any engineer can update with a text editor. If nobody maintains it, at least the content is still readable. A neglected knowledge graph is just a broken database.

It’s not a replacement for an architect. The LLM catches pattern-level deviations, but it doesn’t understand your business. It can flag that a column type changed, but it can’t tell you whether that change is the right call. The human always decides.

!!! warning “EDIT: A Reality Check on LLMs in Data Engineering”

In my [spec-driven BI series](/blog/spec-driven-bi-part1-when-it-works), I argue that using LLMs to *generate* production SQL is an anti-pattern because it introduces non-determinism. I must acknowledge the inherent risk here: using AI to evaluate cross-language semantic impact (Databricks to DAX) relies on the same probabilistic mechanisms. If an LLM can hallucinate a `SELECT` statement, it can absolutely hallucinate an evaluation. BI Docs mitigates this by keeping the AI strictly in a *review and flagging* capacity—never blocking CI pipelines and never writing production code. You are trading a generation hallucination risk for an evaluation hallucination risk. Consequently, alert fatigue is a real danger. The system must remain a lightweight assistant, not an absolute governor.

Tools like [dbt-mcp](https://github.com/dbt-labs/dbt-mcp) (open-sourced in April 2025, GA since October 2025) demonstrate that AI agents can query metric definitions, trace lineage, and run dbt commands through structured interfaces—operating *through* the semantic layer rather than around it. The specific protocol (MCP, REST, GraphQL) matters less than the principle: AI is most useful when it consumes governed, human-authored structures, not when it generates them from scratch.

## Related: Spec-Driven Development

BI Docs enforces documentation-as-code by catching drift between your codebase and your Git-versioned Data Decision Records (DDRs). 

In a separate two-part series, I explore **spec-driven development**, which explores the inverse flow: generating physical code from specifications, and why those physical specs fail to capture the semantic reality that DDRs are designed to protect.

- [Part 1: When Specs Work](spec-driven-bi-part1-when-it-works.md) — The infrastructure vs. exploratory distinction
- [Part 2: The Semantic Gap](spec-driven-bi-part2-semantic-gap.md) — The limits of specification and the need for Semantic Layers

Closing Thoughts

The best documentation system is one that survives your departure. Not because it’s automated, but because it’s simple enough that anyone can contribute to it, and grounded enough that people actually trust it.

If you’re a data engineer preparing for a transition, or just tired of answering the same questions about why a threshold is set to 0.15 or why the fiscal year starts in July, start with the evidence. Dig through the emails. Read the old Jira tickets. Talk to the people who were there when the decisions were made. Write it down in Markdown. Put it in Git.

The automation layer is nice. It helps prevent drift. But the real value is the baseline you build in Phase 0. That collection of facts, timelines, and documented decisions is the asset. Everything else is tooling to protect it.

Start small. One repository. One mkdocs.yml. One folder called timelines/ with a single Markdown file logging every decision you can remember. A simple chronological ledger preserves more context than you’d expect, and your successor will thank you for it.

We obsess over technique, repertoire, interpretation. We optimize practice schedules and lesson structures. But we rarely talk about the physical capacity required to execute all of that without breaking down.

The problem stays, invisible, until it’s too late.

The physical load

Practice is physical work. Not in the dramatic, sweaty way (sometimes it is), but in the slow, accumulative way that causes damage over time.

Static load. Holding a 10–12 kg instrument in a fixed position for many hours is demanding. Your muscles aren’t moving much, but they’re firing constantly just to keep the instrument stable.

Repetition. The same fingering patterns, the same bellows movements, the same postural asymmetries—session after session, year after year. Micro-trauma accumulates.

Asymmetry. The accordion sits to the left. The right hand stretches out. The left arm pulls. The shoulders rotate. This isn’t neutral. Over time, the body adapts, but not always in healthy ways. The imbalance pulls your spine out of alignment and creates tension patterns that eventually lead to injury.

Micro-stress accumulation. No single session destroys your back. But a thousand sessions with poor setup and no recovery strategy? That adds up.

Among acoustic instruments, the accordion is one of the most physically demanding: 9–15 kg of weight, constant bellows resistance, straps transferring load to your trapezius and upper back, and asymmetric sitting posture with limited movement.

This isn’t to scare anyone. Plenty of accordionists play for 50 years without major issues. But they usually have two things in common: good setup habits and some form of physical conditioning.

My wake-up call

Ten years ago, I developed severe sciatic back pain that changed how I viewed my instrument. The pain was intense enough that I had to wear a support belt to be able to play the accordion standing up or walking. At its worst, it would temporarily paralyze my leg.

I give fewer concerts and have plenty of time to recover nowadays, but it was a harsh lesson. My approach was not sustainable.

Now, at 37, I treat my body like an athlete would.

Building the capacity

I started wondering: what if the problem wasn’t just posture, but capacity?

I remember watching an interview on YouTube with a pianist named Lubomyr Melnyk. He developed a style he calls “Continuous Music”, which consists of maintaining a dense wall of sound through rapid arpeggio runs up and down the keyboard, holding the world record of over 19 notes per second on each hand. He explained that he couldn’t achieve this just by practicing harder. He had to treat it like a martial art. He described transforming his hands to be “soft and water-like”, fundamentally altering his physiology to handle the physical stress. He didn’t just learn to play fast, he engineered a body that could play fast for hours on end.

Interestingly, the accordion is already continuous music by design, so we don’t need to emulate Melnyk’s technique. I’m not aiming for 19 notes per second, nor am I interested in martial arts, but I realized that the same idea of handling physical stress applies to us accordionists: we need to build the physical capacity to handle the demands of the instrument.

I had been lifting weights on and off for years, but never with structured intent for my instrument. I decided to get serious about it. Not bodybuilding serious—just consistent, focused training on the muscles that actually support playing. Here are a few examples:

• Pullups: Counteract the “hunch.” We spend hours rounding forward over the keyboard; pullups strengthen the muscles that pull your shoulders back and down.
• Deadlifts: The ultimate builder for the posterior chain. They teach you to hinge at the hips and keep a flat spine under load—exactly what you need to hold the accordion without slumping.
• Crunches: Not for a six-pack, but for deep stability. A strong anterior core protects the lower back from the weight of the instrument.
• Butterfly (Chest Flys): The accordion builds lopsided strength. Chest flys help balance the permanent tension of the bellows arm against the static right side.
• Shoulder Press: Strengthening the deltoids helps carry the strap load directly, taking pressure off the sensitive neck muscles.

My current routine is non-negotiable:

• Gym 4x/week: following a strict split.
• Mandatory sessions per week: One dedicated to Back & Biceps, another for Chest & Triceps, and another for Legs & Shoulders.
• Cardio: Running 1-2x/week for general endurance.

I’m mainly doing this to keep my body apt for playing the accordion. Targeted weight training also helped me correct asymmetrical chest development from years of bellows work. By isolating sides and ensuring balanced load, I stopped the accordion from reshaping my body in harmful ways.

Beyond the gym, I’ve also restructured how I practice. Instead of marathon sessions, I work in focused drills with breaks every 30 minutes, or 60 minutes at most if I’m in some kind of flow state. It forces me to stay intentional and gives my muscles time to recover between efforts. I strech my back and shoulders by hanging from a pullup bar for a few minutes after each session.

The payoff

Core stability. I notice a massive difference in instrument stability. When your core is solid, the accordion doesn’t pull you around; you control it.

Endurance. I have far more endurance for long concerts. The accordion doesn’t feel lighter—but holding it does.

Better posture retention. I used to start a session sitting properly, then slowly collapse as I got tired. Now I can maintain my setup for the whole session. My body has the reserves to stay aligned.

Improved breath control. A stable, strong core gives you better control over bellows pressure. I wasn’t expecting strength training to affect my sound, but it did.

Control. Demanding techniques, especially bellow shakes, are much easier. When your primary muscles are strong, you don’t need to recruit tension from your neck, shoulders or forearms to generate force. You stay relaxed because you have power in reserve.

Skill × Capacity = Sustainable Performance

This series started with a chair. It ended with a realization: sustainable playing isn’t just about what you practice, but whether your body can handle what you’re asking of it.

This post is the final part of the series Why Accordion Posture Is Not Optional.

My teacher took one look at the setup and repeated the same advice she had given me before. She wasn’t angry in a shouting way—but she was firm. The kind of firm that makes you realize you’ve been doing something wrong without quite understanding what.

“You can’t practice on that”, she said, pointing at the chair. “No wheels, stable base. You should use a proper stool.” Then she looked at the sheet music flat on my desk. “And please use a music stand. You’re constantly looking down at the desk and it affects you more than you think.”

I remember thinking to myself, the first time she told me this: ok, you’re right about the music stand, but the chair? Really?

I’d been so focused on fingering, bellows control, phrasing. The chair felt like furniture. Background. Surely what mattered was what I did with my hands. I even had bought that particular chair because it had retractable armrests!

But she repeated the same advice the next lesson. And the second time she pointed it out, I knew I was being lazy, and finding excuses, basically.

Given that my lessons are currently virtual, and since I use my computer for all sorts of music-related tasks (controlling the metronome, adjusting scores, logging my practice sessions, etc.), I had bought a pedal to switch music pages on my screen. My office is very small, so even though I have a music stand, I deliberately don’t incorporate it into my setup, because I don’t have space near the desk. But out of laziness, I didn’t even plug that pedal in, and kept putting the sheets flat on my desk. I sometimes use the music stand when I practice outside of my office, but that’s not the case during lessons.

Secondly, I juggle constantly between office work, and picking up the accordion as soon as I have some free time. This means constantly switching between sitting on an office chair and an accordion stool is not necessarily practical. My excuse was immediately dismissed by my teacher, because I don’t switch contexts for at least one hour, the time of the lesson, so if I really cared, I would have prepared in advance.

The pattern I didn’t see

A few years prior, I studied with another teacher—Liliana Aparício—who had just finished her master’s research on exactly this topic: posture, pain, and perceived effort in accordion students. Her thesis studied 21 young accordionists and found clear connections between playing setup, asymmetric body load, and musculoskeletal pain.

The findings weren’t surprising to her. She’d seen the pattern for years. The accordion is heavy, asymmetric, and you hold the same position for an hour at a time. If you add an unstable seat to that, your spine has to compensate for every micro-wobble on top of managing the instrument.

That’s when my current teacher’s reaction started to make sense. She wasn’t being picky. She was trying to stop me from building a habit that would hurt me later.

Most accordion teachers have either dealt with back pain themselves or watched students develop it. They know the damage builds up slowly—you don’t notice anything for months, maybe years, and then one day you can’t ignore it anymore. That’s why they react before you feel anything wrong. By the time you feel it, the habit is deep.

It’s not only about your back

My teacher made another point that stuck with me: posture isn’t only about pain prevention—it directly affects your technique.

When you bend forward, your elbow and wrist automatically adjust to compensate. Your hand ends up wherever it lands, rather than where you deliberately place it. You stop paying attention to how your fingers approach the keyboard because your whole arm has already made the decision for you.

Leaning backwards on a reclinable chair is even worse. It feels comfortable, but that’s the trap—it enables a kind of body laziness where your core disengages entirely. You’re not sitting anymore; you’re sinking. And your arms have to reach forward awkwardly to find the keys.

Sitting upright forces you to stay conscious of your hand position. It’s harder at first, but that’s the point—the difficulty is the feedback that tells you you’re doing it right.

I ended up using a simple wooden stool that was sitting under my wife’s vanity desk. No back support, no wheels, no tilt—just a flat, stable surface at the right height. Sometimes the solution is already in your house; you don’t need a shopping spree. It feels awkward—I have to actually sit properly, engaging my core, keeping my spine aligned. It’s too early to report results, but at least I’m not fighting the chair anymore.

Regarding the page turning pedal, I still need to get used to it. I need to find a comfortable workflow that makes me more productive using the computer screen, without having to fight it. Also, I can use the music stand when I practice outside of my office, and I can certainly do that more often, namely on weekends.

Morale of the story: no excuses. Just make things work, and protect your body.

That lesson pushed me to change more than my chair—it changed how I thought about training my body. More on that in Part 2.

This post is part of the series Why Accordion Posture Is Not Optional.

Reference: Aparício, L. (2014). Postura, dor e perceção de esforço na aprendizagem do acordeão [Master’s thesis]. Available at Academia.edu.

Most practice advice assumes you have unlimited time and a quiet calendar. I don’t.

Between my data engineering job, family, kids, gigs, and teaching, I needed something that keeps practice focused without turning it into another job. A typical week isn’t “practice every day for two hours”, it’s more like one good day, two chaotic ones, then rushing to prepare the next concert. The system has to survive that.

What I ended up building in Obsidian is simple. Lesson notes become a plan, the plan becomes one daily checkbox, and each session leaves a short trail I can pick up tomorrow.

My method involves three steps:

1. Capturing feedback as early as possible, because I’ll forget half of it by tomorrow. During each lesson, I take as many notes as I can. Then I turn them into drills. These become my baseline until the next lesson.
2. Before each practice session, I make a short plan, because I can’t realistically go through every single drill I have, so I prioritize smaller chunks of work.
3. After practice, I log what actually happened. The whole point is to keep the “what should I do next?” question out of the practice room.

Capturing feedback as early as possible and writing the drills

Each lesson gets its own note. Instead of writing vague feedback, I turn teacher comments into explicit, testable drills—bar ranges, tempo targets, musical goals. Those drills become the raw material for the next practice plan.

Planning the practice session

The day before, I create a short Practice Plan note with:

• How much time I have
• One or two priority pieces
• The specific drills I’ll work on
• What I expect to achieve

A plan might look like 120 minutes across two pieces, drilling left-hand fingering changes, rhythmic control in 6/8, and pushing a triplet passage from 70 to 80 bpm.

This isn’t a diary. It’s just a decision made in advance so I don’t spend the first 15 minutes negotiating with myself—or the whole afternoon procrastinating.

Logging the practice session

After practicing, I fill in a Practice Note. I keep the format consistent:

• Total minutes
• Which pieces I worked on
• Drills with section, tempo, and result
• Observations and issues to carry forward

One recent session logged three pieces and 180 minutes of work, noting which passages were clean at target tempo and which still needed attention. That detail makes tomorrow’s session faster because I’m not guessing where I left off.

The real value, though, is the self-reflection. Over time you start to notice patterns, and your internal assessment gets sharper. That makes the whole loop tighter.

Where AI helps

Because my notes follow a consistent structure, I can use AI to handle two annoying parts:

• Drafting the plan from recent lessons and logs (e.g., “Bars 199–202 LH fingering at 50→60 bpm”)
• Summarizing the last session so I can pick up where I left off

It keeps the system lightweight, even when I only have 35 minutes and my head is already full.

Wrapping up

I’ve been using this for three months now. Some weeks I still skip days, or show up less prepared for the next lesson than I would have liked. But when I sit down to practice, I know exactly what to work on. And when life gets messy, at least I have a note telling me where I left off. Lastly, there’s a thin line between a practice system and a productivity system, and I’m trying to stay on the self-tolerant side of it, because life happens, and that’s okay.

Not toxic workplaces. Not bad people. Just systems where responsibility quietly disappears.

In a lot of places I’ve worked, decisions get made somewhere upstream while execution gets pushed downstream. Your direct manager coordinates work but doesn’t really own the direction. The priorities, the trade-offs, the long-term bets—those are made elsewhere. When things don’t improve, nobody’s explicitly responsible. When things go wrong, it’s “the system.”

People keep delivering, reliably and professionally, while quietly disengaging from trying to fix how things actually work.

The art of the explanation

Scarcity mindset usually doesn’t show up as “we don’t have budget.” It shows up as explanation.

Technical debt piles up because “it’s not prioritized.” Architectural problems wait because “a migration is coming.” Feedback is nonexistent because “users aren’t complaining.” Underutilization is normal because “this is how things are.”

You can feel it in meetings. Someone brings up a recurring problem. Everyone agrees it’s real. Then the next ten minutes are spent building the case for why it can’t be touched right now. A ticket gets logged, a status gets updated, and the product stays exactly the same.

The implicit rule becomes: if something doesn’t change, it’s structural. If something fails, it’s procedural. Nobody owns the outcome.

What often gets framed as “focusing on low-hanging fruit” is really a preference for work that carries no personal risk. Cosmetic improvements, small tweaks, isolated fixes—they look like progress while everyone gets to say “I followed the process.” What’s missing is someone stepping forward to say: this is the direction, and if it fails, I own it.

When responsibility keeps getting redirected to “the system,” meaningful change never actually starts.

The slow drain

This kind of environment rarely causes burnout through overload. It causes it through loss of agency.

When every constraint is framed as systemic, initiative stops mattering. Expertise becomes decorative. Growth becomes accidental. People don’t leave because they’re exhausted—they leave because nothing they do can meaningfully change anything.

I’ve watched this happen to good engineers who just… stopped trying. Not because they gave up, but because the system trained them to. Why push for a better solution when the answer is always “we’d love to, but”?

Careers don’t stall because of one bad decision. They stall in places where excuses are safer than ownership, where responsibility is always someone else’s problem, where the system is always to blame and nobody ever is.

If you’re in that kind of environment and you recognize it early, that’s not pessimism. That’s just paying attention.

However, on Databricks, the hard part isn’t ingestion or compute, it’s the modeling glue. The platform gives you the tools to move data, but the mechanics of surrogate keys, SCD logic, and orchestration order are still left entirely to you. Usually, this means every team ends up re-inventing the same boilerplate code from scratch. And, in a way, teams end up writing tooling to perform integration duties that were achieved through simple GUI-based tools previously, creating a large surface for all sorts of errors to happen.

That recurring led me to create the databricks_kimball_framework. It’s a small, open-source project designed to make dimensional modeling repeatable, letting you focus on the data logic instead of the MERGE syntax.

Why Kimball Modeling Is Difficult on Databricks

Here are the parts that kept biting me:

1. Implementing SCD logic consistently

Delta Lake’s MERGE command is powerful, but implementing SCD Type 1 and 2 patterns correctly—with natural keys, surrogate keys, versioning, effective dates, and deduplication requires a LOT of boilerplate. Most organizations end up reinventing variations of the same code.

2. Handling incremental change feeds

Databricks advocates incremental ingestion, whether through Auto Loader or Change Data Feed (CDF). But turning raw changes into structured dimension updates is not something the platform abstracts for you. That layer still falls to engineers to design and maintain.

3. Managing dependencies between facts and dimensions

Fact tables rely on up-to-date dimension rows. Ensuring that dimensions load first, dealing with early-arriving facts, and maintaining referential consistency are all warehouse fundamentals, but Databricks does not provide modeling-aware orchestration.

4. Schema evolution and physical optimization

Even with Delta Lake’s schema evolution features, determining when and how to propagate field changes and how to tune clustering/compaction is a manual effort.

None of this is “Databricks is bad.” It’s a general-purpose platform. But it does mean classic Kimball modeling takes real engineering effort to implement repeatedly and safely.

Why Auto Loader and Declarative Pipelines Weren’t Enough

Auto Loader is excellent at file discovery and incremental ingestion, but it does not attempt to handle SCD logic, surrogate keys, or dimensional structure. It addresses the ingestion layer, not the modeling layer.

Databricks has also offered declarative pipeline mechanics in various forms (e.g., Lakeflow Spark Declarative Pipelines, materialized views, AUTO CDC). These features are useful, but their capabilities and visibility have shifted over time, and they are not designed as dimensional modeling frameworks. They don’t provide constructs like facts, dimensions, natural-to-surrogate key resolution, or early-arriving fact handling.

So if you want structured, repeatable Kimball patterns, you’re still largely building that layer yourself.

What the Framework Provides

The databricks_kimball_framework offers a simple, declarative way to define dimensional models in Databricks:

• YAML-based definitions for facts and dimensions
• Built-in SCD logic (1, 2, 4 and 6)
• Surrogate key creation and natural-key matching
• Optional placeholder generation for early-arriving facts
• Incremental pipelines using CDF and watermarks
• Optional clustering/optimization settings for Delta tables
• Audit fields for transparency and debugging

The focus is intentionally narrow: provide consistent, repeatable patterns for Kimball-style modeling without requiring engineers to write and maintain large amounts of repetitive MERGE or orchestration code.

Architecture Overview

The diagram above illustrates the flow from raw data sources through the framework’s dimension and fact processing, ultimately producing a clean star schema in your Delta Lake tables.

How It Compares to dbt

dbt shines as a SQL transformation framework with strong testing, documentation, and community support. Its approach is flexible and well suited to many analytical workloads.

My framework differs in its focus:

• It provides native SCD handling, whereas dbt requires custom macros or packages.
• It manages surrogate keys and historical versions, which dbt leaves to user-defined SQL.
• It is designed specifically for Delta Lake and Databricks incremental processing.

For many teams, dbt and this framework can complement each other. dbt is excellent for wide transformation logic; the framework focuses specifically on warehouse modeling semantics. You might use both—this framework for the SCD/surrogate key mechanics, and dbt for the broader transformation DAG.

Try It Out

The project is available on GitHub:

👉 github.com/accordionjourneyman/databricks_kimball_framework

It is still in beta quality. Not all scenarios have been tested, and bugs are likely to surface as more organizations apply it to different datasets and pipelines. I welcome feedback, issue reports, and contributions from data engineers who are interested in improving dimensional modeling on the lakehouse.

If you’re working with Databricks and you still value clear, well-structured dimensional models, try the framework, and let me know both where it helps and where it breaks.