[{"data":1,"prerenderedAt":177},["ShallowReactive",2],{"$f2g09nhdgacggi":3,"$f3ddtnpk8sx54l":139},[4,23,37,50,61,72,82,95,106,118,130],{"id":5,"slug":6,"title":7,"date":8,"tags":9,"image":17,"content":18,"excerpt":19,"section":20,"series":21,"series_part":21,"lang":22},1825,"kimball-pyspark-in-a-dbt-world","Finishing What I Started: Kimball on PySpark in a dbt World","2026-08-30T00:00:00.000Z",[10,11,12,13,14,15,16],"Data Engineering","Databricks","Kimball","dbt","Data Warehousing","Open Source","Data Modeling","\u002Fimg\u002Fblog-kimball-header.png","\u003Cp>Back in December, I wrote about \u003Ca href=\"\u002Fblog\u002Fdatabricks-kimball-framework\u002F\">building a Kimball-style modeling framework for Databricks\u003C\u002Fa>. At the time, it was a modest v0.1.1 beta: fourteen Python modules that parsed YAML configurations and generated basic Delta Lake MERGE statements for SCD Type 1 and Type 2 tables.\u003C\u002Fp>\n\u003Cp>Almost immediately, colleagues and data engineers asked the obvious question:\u003C\u002Fp>\n\u003Cblockquote>\n\u003Cp>\u003Cem>“The entire data engineering world has settled on dbt for modeling. Why spend your nights building a custom PySpark framework for the same thing?”\u003C\u002Fem>\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Cp>It was a fair question. dbt is the industry standard for analytics engineering. It has an immense ecosystem, great documentation, and millions of users. Writing a bespoke dimensional framework in Python can easily look like classic NIH (Not Invented Here) syndrome.\u003C\u002Fp>\n\u003Cp>Yet over the past nine months, I kept going. The project evolved from a proof-of-concept script into a hardened Gold-layer loading engine, now at \u003Cstrong>v0.3.0\u003C\u002Fstrong>. Along the way, I also built a private dbt-native experiment (\u003Ccode>databricks_kimball_framework_dbt\u003C\u002Fcode>) to test parity, and started tinkering locally with automated linting (\u003Ccode>kimball_lint\u003C\u002Fcode>) and visual authoring (\u003Ccode>SDL\u003C\u002Fcode>).\u003C\u002Fp>\n\u003Cp>Here is why I chose to finish the PySpark framework, what changed between the early prototype and the current codebase, and what working across both stacks taught me about where dimensional complexity actually lives.\u003C\u002Fp>\n\u003Chr>\n\u003Ch2>1. What the PySpark Framework Actually Grew Into\u003C\u002Fh2>\n\u003Cp>The original December version was essentially a boilerplate generator. If you had a clean source table and wanted to MERGE it into an SCD2 dimension without writing 40 lines of repetitive SQL, it worked.\u003C\u002Fp>\n\u003Cp>Production pipelines, however, do not stay in happy-path territory for long. Real lakehouse workloads bring out-of-order records, late-arriving dimensions, schema drifts, conflicting writers, and long cluster start-up times. As I ran into these walls, the framework evolved significantly:\u003C\u002Fp>\n\u003Ch3>A Strictly Spark-Free Control Plane\u003C\u002Fh3>\n\u003Cp>In early iterations, even basic configuration validation required spinning up a local PySpark session. That meant waiting 10–15 seconds for a JVM to initialize just to find out someone made a typo in a YAML field.\u003C\u002Fp>\n\u003Cp>In v0.3.0, the entire control plane is decoupled from Spark:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>\u003Ccode>kimball validate\u003C\u002Fcode>\u003C\u002Fstrong>: Validates YAML configs against strict Pydantic schemas instantly on a laptop.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>\u003Ccode>kimball compile\u003C\u002Fcode>\u003C\u002Fstrong>: Resolves dependencies, builds a directed acyclic graph (DAG), and detects circular references or duplicate target writers before touching any data.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>\u003Ccode>kimball plan\u003C\u002Fcode>\u003C\u002Fstrong>: Compares your local configuration against a deployed \u003Ccode>manifest.json\u003C\u002Fcode>. It classifies changes (metadata update, validation rule, backfill required, or breaking change) and reports every transitively affected downstream table.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Because this layer has zero Spark dependencies, it runs in CI in milliseconds.\u003C\u002Fp>\n\u003Cpre>\u003Ccode>YAML Configs ──► kimball compile ──► DAG Validation ──► manifest.json\n                                        │\n                                        └──► Databricks Asset Bundle (DAB)\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch3>Databricks Asset Bundles (DABs) for Parallel Execution\u003C\u002Fh3>\n\u003Cp>A recurring challenge with PySpark is orchestration. Running multiple threads in a single Python process often leads to SparkSession contention and subtle race conditions on target tables.\u003C\u002Fp>\n\u003Cp>Instead of writing custom thread pools, the compiler now emits native Databricks Asset Bundle job definitions (\u003Ccode>kimball compile --bundle-output generated-job.yml\u003C\u002Fcode>). The DAG levels are compiled directly into parallel Databricks Workflow tasks. Independent dimensions load concurrently across the cluster, while target tables retain strict single-writer isolation and centralized checkpointing.\u003C\u002Fp>\n\u003Ch3>Real Key Brokering &amp; Type 7 Semantics\u003C\u002Fh3>\n\u003Cp>Basic SCD2 tracks historical row changes, but analytical users frequently need both point-in-time (“as-was”) reporting and current-state (“as-is”) aggregation. That requires \u003Cstrong>Kimball Type 7 dual-key modeling\u003C\u002Fstrong>:\u003C\u002Fp>\n\u003Col>\n\u003Cli>A \u003Cstrong>Durable Key (\u003Ccode>_dk\u003C\u002Fcode>)\u003C\u002Fstrong>: A deterministic hash identifying the entity across its entire lifetime.\u003C\u002Fli>\n\u003Cli>A \u003Cstrong>Version Surrogate Key (\u003Ccode>_sk\u003C\u002Fcode>)\u003C\u002Fstrong>: A hash derived from the natural key plus the business-effective timestamp, identifying that specific revision.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>The framework manages this key brokering automatically. It seeds standardized non-null reserved members (\u003Ccode>-1\u003C\u002Fcode> for Unknown, \u003Ccode>-2\u003C\u002Fcode> for Not Applicable, \u003Ccode>-3\u003C\u002Fcode> for Invalid, and \u003Ccode>-4\u003C\u002Fcode> for Pending\u002FLate-Arriving). Furthermore, it enforces strict half-open validity intervals (\u003Ccode>valid_from\u003C\u002Fcode> to \u003Ccode>valid_to\u003C\u002Fcode> with explicit high-dates like \u003Ccode>9999-12-31\u003C\u002Fcode>) instead of NULL endpoints, eliminating tricky join bugs in downstream BI tools.\u003C\u002Fp>\n\u003Ch3>Operational State &amp; Unresolved Keys\u003C\u002Fh3>\n\u003Cp>When a fact table arrives with a foreign key referencing a customer that hasn’t landed in the dimension table yet, most naive pipelines either fail or drop the row.\u003C\u002Fp>\n\u003Cp>The framework writes operational state into two metadata tables:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>\u003Ccode>etl_control\u003C\u002Fcode>\u003C\u002Fstrong>: Records source watermarks, batch IDs, commit timestamps, and row counts for every run.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>\u003Ccode>etl_unresolved_dimension_keys\u003C\u002Fcode>\u003C\u002Fstrong>: Captures orphaned fact relationships. When a fact references a missing dimension key, the framework inserts an inferred placeholder member into the dimension with a \u003Ccode>-4\u003C\u002Fcode> status, writes the relationship to the unresolved keys table, and automatically hydrates the full row once the dimension data arrives.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Chr>\n\u003Ch2>2. The dbt Experiment: What Happened When I Built the Sibling\u003C\u002Fh2>\n\u003Cp>After getting the PySpark engine stable, I wanted to test my own assumptions. Could this entire architecture be expressed as pure dbt macros?\u003C\u002Fp>\n\u003Cp>In July, I set up a private sibling experiment—\u003Cstrong>\u003Ccode>databricks_kimball_framework_dbt\u003C\u002Fcode>\u003C\u002Fstrong>—to see if I could implement the exact same observable behaviors using dbt-core and Jinja macros on Databricks SQL and Delta Lake.\u003C\u002Fp>\n\u003Cp>The experience revealed two big takeaways:\u003C\u002Fp>\n\u003Ch3>Where dbt Shines\u003C\u002Fh3>\n\u003Cp>dbt’s ecosystem handles the scaffolding you would otherwise have to build yourself:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>DAG Orchestration\u003C\u002Fstrong>: Model dependencies, \u003Ccode>ref()\u003C\u002Fcode> resolution, and lineage visualization work effortlessly.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Community Tooling\u003C\u002Fstrong>: Documentation generation, package management, and generic schema testing (\u003Ccode>unique\u003C\u002Fcode>, \u003Ccode>not_null\u003C\u002Fcode>, \u003Ccode>accepted_values\u003C\u002Fcode>) are battle-tested and widely understood.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Standardization\u003C\u002Fstrong>: Onboarding a typical analytics engineer to a dbt project is vastly easier than teaching them an internal Python DSL.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch3>Where dbt Hits Friction\u003C\u002Fh3>\n\u003Cp>The catch is that \u003Cstrong>dbt does not eliminate lakehouse state mechanics—it relocates them into Jinja\u003C\u002Fstrong>.\u003C\u002Fp>\n\u003Cp>Implementing complex dimensional logic in dbt means writing hundreds of lines of Jinja SQL macros:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Managing multi-pass \u003Ccode>MERGE\u003C\u002Fcode> statements with out-of-order Change Data Feed (CDF) events.\u003C\u002Fli>\n\u003Cli>Handling dual-key generation, deterministic surrogate hashing, and inferred member hydration.\u003C\u002Fli>\n\u003Cli>Replaying late-arriving dimension changes across historical fact tables.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>When that logic lives inside Jinja, debugging becomes significantly harder. Compiled SQL files stretch into thousands of lines of dense MERGE commands. What is naturally expressed as procedural Python logic (reading Delta commit history, orchestrating multi-step reconciliation, handling rollback upon batch failure) feels stretched when forced into a template language designed primarily for SQL SELECT transformations.\u003C\u002Fp>\n\u003Cp>In short: dbt is fantastic for transformation logic, but using it as a low-level state-and-storage engine on Delta Lake requires substantial macro scaffolding.\u003C\u002Fp>\n\u003Chr>\n\u003Ch2>3. The Real Problem: The Authoring and Modeling Gap\u003C\u002Fh2>\n\u003Cp>Working between PySpark and dbt brought a deeper realization.\u003C\u002Fp>\n\u003Cp>As data engineers, we spend an enormous amount of time debating tools: PySpark vs. dbt, SQL vs. Python, Airflow vs. Dagster. But when analytics projects fail or dashboards display conflicting numbers, \u003Cstrong>it is almost never because someone chose the wrong orchestration framework\u003C\u002Fstrong>.\u003C\u002Fp>\n\u003Cp>The failures come from the modeling layer:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Ambiguous Grain\u003C\u002Fstrong>: A fact table mixes transaction-level rows with daily aggregated rows.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Key Mismatches\u003C\u002Fstrong>: Fact tables joining on mutable natural keys instead of durable keys.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Dimension Leakage\u003C\u002Fstrong>: Slowly changing attributes duplicated across multiple tables without a clear source of truth.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Unreachable Dimensions\u003C\u002Fstrong>: Foreign keys that can never resolve because of misaligned time granularities.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Whether you write your pipeline in YAML or dbt SQL, if the dimensional design is flawed, all the tooling does is automate bad data faster.\u003C\u002Fp>\n\u003Cp>That realization led to two local tools I’ve been prototyping on my machine:\u003C\u002Fp>\n\u003Ch3>\u003Ccode>kimball_lint\u003C\u002Fcode> — Catching Modeling Traps in CI\u003C\u002Fh3>\n\u003Cp>Inspired by compiler checks, \u003Cstrong>\u003Ccode>kimball_lint\u003C\u002Fcode>\u003C\u002Fstrong> is a local, evidence-aware static linter for dimensional models.\u003C\u002Fp>\n\u003Cp>Instead of waiting for data discrepancies to surface in production dashboards or review cycles, it inspects manifests and model metadata to verify warehouse design rules before merging code:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Verifies that every fact table explicitly declares its measurement grain.\u003C\u002Fli>\n\u003Cli>Flags dimension attributes denormalized directly into fact tables without a surrogate relationship.\u003C\u002Fli>\n\u003Cli>Detects mutable keys used where durable entity keys are required.\u003C\u002Fli>\n\u003Cli>Alerts on missing time-dimension associations for temporal facts.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch3>SDL (\u003Ccode>semantic_data_loader\u003C\u002Fcode>) — Visual Authoring with Validated Outputs\u003C\u002Fh3>\n\u003Cp>Most data practitioners know what business process they are trying to model, but translating that mental model into correct surrogate keys, bridge tables, and staging views can be daunting.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>SDL\u003C\u002Fstrong> is an experimental visual workbench I’ve been prototyping locally. The idea is to let engineers model facts, dimensions, and relationships visually, validate the graph against dimensional modeling rules, and compile the result into clean, standard, human-readable dbt projects. dbt remains the execution boundary, but the authoring happens in an environment that prevents structural mistakes before code is committed.\u003C\u002Fp>\n\u003Chr>\n\u003Ch2>Summary: What I Learned\u003C\u002Fh2>\n\u003Cp>Finishing the PySpark framework was never about convincing data teams to abandon dbt. Rather, it was about mastering the mechanics of the lakehouse by building them from the ground up.\u003C\u002Fp>\n\u003Cp>A few core takeaways:\u003C\u002Fp>\n\u003Col>\n\u003Cli>\u003Cstrong>Procedural Python and declarative SQL serve different needs\u003C\u002Fstrong>: PySpark excels at low-level Delta stream handling, CDF state reconciliation, and granular transaction control. dbt excels at DAG dependencies, modular SQL transforms, and team collaboration. The two complement each other when applied at the right boundaries.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Key contracts matter more than merge syntax\u003C\u002Fstrong>: Getting \u003Ccode>WHEN MATCHED THEN UPDATE\u003C\u002Fcode> right takes an afternoon. Getting Type 7 durable vs. version keys, null semantics, and inferred late-arriving members right takes weeks of edge-case testing.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>The star schema remains as relevant as ever\u003C\u002Fstrong>: Modern query engines have gotten faster, but business logic hasn’t gotten simpler. Clean, conformed dimensions and explicit fact grains remain the most reliable way to deliver self-serve analytics that business users can actually trust.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Chr>\n\u003Cp>\u003Cem>The open-source framework is available on GitHub:\u003C\u002Fem>\n👉 \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Faccordionjourneyman\u002Fdatabricks_kimball_framework\">\u003Cstrong>github.com\u002Faccordionjourneyman\u002Fdatabricks_kimball_framework\u003C\u002Fstrong>\u003C\u002Fa>\u003C\u002Fp>\n\u003Cp>\u003Cem>(Note: The dbt parity implementation, kimball_lint, and SDL mentioned above are currently local research prototypes and experiments).\u003C\u002Fem>\u003C\u002Fp>\n","Why I kept building a declarative PySpark dimensional modeling framework for Databricks when everyone uses dbt, and what both stacks taught me.","tech",null,"en",{"id":24,"slug":25,"title":26,"date":27,"tags":28,"image":34,"content":35,"excerpt":36,"section":20,"series":21,"series_part":21,"lang":22},1824,"hermes-agent-memory-crystallization","Hermes Agent + Obsidian: Crystallizing a Persistent Thinking Partner","2026-06-29T00:00:00.000Z",[29,30,31,32,33],"AI","Productivity","Note-Taking","Engineering Leadership","Hermes Agent","\u002Fimg\u002Fblog-hermes-memory.png","\u003Cp>A few months ago, I started using an AI agent called \u003Ca href=\"https:\u002F\u002Fhermes-agent.nousresearch.com\">Hermes Agent\u003C\u002Fa> in a way I had not tried before: not as a chatbot for one-off questions, but as a \u003Cstrong>persistent thinking partner\u003C\u002Fstrong> that remembers what we talk about across sessions.\u003C\u002Fp>\n\u003Cp>The shift in the interface is tiny; the shift in the outcome is massive. A chatbot answers your question and forgets. An agent with memory builds shared understanding over time — and that changes what you can do with it.\u003C\u002Fp>\n\u003Cp>Hermes is built by \u003Ca href=\"https:\u002F\u002Fnousresearch.com\">Nous Research\u003C\u002Fa>, the lab behind the Hermes model series. The pitch is that it is the only agent with a native learning loop: it creates skills from experience, improves them during use, and persists knowledge across sessions. In practice, what that means is the agent builds a working model of who you are, not by reading a profile you filled out, but by remembering what you talk about and how you react.\u003C\u002Fp>\n\u003Ch2>The Problem: Context That Evaporates\u003C\u002Fh2>\n\u003Cp>Most personal growth insights are fleeting. You notice something about yourself: a pattern in how you respond to stress, a career decision that felt right at the time but looks different in hindsight, a creative block that keeps showing up in the same shape. And you tell yourself you’ll remember it. Then you don’t.\u003C\u002Fp>\n\u003Cp>I have tried every system for capturing this: Obsidian vaults, journaling, periodic reviews. They all share the same failure mode: \u003Cstrong>the capture is manual, and getting it back out is a chore\u003C\u002Fstrong>. You have to remember to write it down, and then you have to remember where you wrote it.\u003C\u002Fp>\n\u003Cp>The vault I settled on uses the \u003Cstrong>PARA method\u003C\u002Fstrong> (Projects, Areas, Resources, Archives): a structure designed for actionability, not just storage. \u003Ccode>10 Projects\u002F\u003C\u002Fcode> holds active work with deadlines (a festival, a band album, a career goal). \u003Ccode>20 Areas\u002F\u003C\u002Fcode> holds ongoing responsibilities without a finish line (music practice, professional development, personal reflections). \u003Ccode>30 Resources\u002F\u003C\u002Fcode> holds reference material (repertoire, lesson notes, tech plans). \u003Ccode>90 Archive\u002F\u003C\u002Fcode> holds completed or stalled work. This structure is the backbone of everything that follows.\u003C\u002Fp>\n\u003Ch2>The Shift: From Capture to Crystallization\u003C\u002Fh2>\n\u003Cp>To pair Hermes with Obsidian, I mapped the structure of my notes for Hermes, defined a formal protocol for how it should navigate my PARA folders, and encapsulated that logic into a persistent “skill.” Now, every time we start a session, Hermes doesn’t just respond—it updates its understanding of my evolving context.\u003C\u002Fp>\n\u003Cp>I realized I no longer needed to carry the entire cognitive load of documenting and synthesizing my own growth. I could offload the crystallization to the agent itself.\u003C\u002Fp>\n\u003Cp>Here is how it works in practice:\u003C\u002Fp>\n\u003Cp>\u003Cstrong>0. Initialize the protocol.\u003C\u002Fstrong> Before beginning, I provided Hermes with a map of my vault and dictated the rules for the interaction into a custom skill. This is a one-time architectural investment: it creates the handshake between the agent and my files, allowing Hermes to autonomously read, index, and update my notes without further manual intervention.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>1. Persistent memory across sessions.\u003C\u002Fstrong> Hermes saves durable facts: preferences, environment details, project context, lessons learned. These get injected into every future conversation, saving me from having to re-establish context every time I open a chat.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>2. Session search for past decisions.\u003C\u002Fstrong> Every conversation is indexed and searchable. When I need to recall what we discussed about a specific topic or a past analysis, I can search across sessions. The agent returns not just the relevant messages, but the surrounding context: the goal of the session, the resolution, the decisions made.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>3. Skills as reusable procedures.\u003C\u002Fstrong> When I discover a workflow that works, I save it as a skill. A debugging procedure. A framework for analyzing team dynamics. A template for a recurring task. The next time something similar comes up, the agent loads the skill and follows the established approach.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>4. Cron jobs for recurring analysis.\u003C\u002Fstrong> I have set up scheduled tasks that run autonomously: checking on specific metrics, monitoring trends, running analyses at regular intervals. The agent executes them and delivers the results back to me.\u003C\u002Fp>\n\u003Ch2>The Next Layer: Vault-Native Context Recall\u003C\u002Fh2>\n\u003Cp>The built-in memory system works well for compact facts: preferences, stable truths, environment details. But it has a hard limit (~2,200 characters). For a growing Obsidian vault with hundreds of notes spanning months of project logs, career reflections, and technical analysis, that limit is reached quickly.\u003C\u002Fp>\n\u003Cp>To actually make this work without blowing up a context window requires getting slightly deep into the plumbing. The solution was to add a \u003Cstrong>retrieval layer\u003C\u002Fstrong> between the agent and the vault.\u003C\u002Fp>\n\u003Cdiv class=\"mermaid\">flowchart LR\n    A[Obsidian Vault] --&gt; B[Talon]\n    B --&gt; C[Hybrid Search]\n    C --&gt; D[MCP Server]\n    D --&gt; E[Hermes Agent]\n\n    subgraph C[Hybrid Search]\n        direction TB\n        C1[BM25 Keywords]\n        C2[Dense Vectors]\n        C3[Graph Signals]\n    end\n\u003C\u002Fdiv>\u003Cp>I installed \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fseanmozeik\u002Ftalon\">Talon\u003C\u002Fa>, an open-source Rust binary that indexes an Obsidian vault and serves it back as hybrid search with graph-aware ranking via a stateless MCP server. On every agent turn, Talon distills the prompt into search phrases, runs a hybrid search combining BM25 keyword matching with dense vector embeddings and graph signals, fuses the results, and injects a compact recall block into the agent’s context. The agent sees relevant vault content without reading files. The vault stays the source of truth; the retrieval layer handles the navigation.\u003C\u002Fp>\n\u003Ch2>The Career Strategy Lens\u003C\u002Fh2>\n\u003Cp>One skill worth singling out: I gave Hermes a dedicated lens for \u003Cstrong>career strategy\u003C\u002Fstrong>: a set of frameworks drawn from systems thinking, incentive theory, and organizational behaviour. The idea is simple: given a career situation, the agent can rotate through different perspectives on the same facts.\u003C\u002Fp>\n\u003Cdiv class=\"mermaid\">flowchart TD\n    S[Career Situation] --&gt; I[Incentives Lens]\n    S --&gt; P[Power Lens]\n    S --&gt; Sy[Systems Lens]\n    I --&gt; O1[What does this environment reward?]\n    P --&gt; O2[Where does influence actually sit?]\n    Sy --&gt; O3[What is this organization optimizing for?]\n\u003C\u002Fdiv>\u003Cp>One pass reads it through incentives: what does this environment reward, and does that align with where I want to go? Another reads it through power: where does influence actually sit, and what does that mean for what I can accomplish here? Another reads it through systems: what is this organization optimizing for, whether it knows it or not, and how does that shape my options? The output is structured analysis: the same situation, seen through different lenses, with the contradictions between them often being the most useful part.\u003C\u002Fp>\n\u003Cp>This lens applies to more than the current role. It works for evaluating a career move: what does the new path reward, and does that match what you want to build? It works for long-term goals: if you want to be in a certain position in five years, what does the path actually reward, and does that match what you are willing to do? The real value is not the generated output. It is the forced interaction with your own data.\u003C\u002Fp>\n\u003Cp>When you run these lenses over a long enough timeline, they start revealing things you miss in the day-to-day: which problems are yours to solve and which are not, how to make a case land when being right isn’t enough, why a role that looked perfect on paper feels wrong six months in, how to evaluate a decision when the short-term cost and the long-term gain point in opposite directions. The value is having it brought up consistently across sessions until it sticks.\u003C\u002Fp>\n\u003Cp>The lenses are especially sharp during a job hunt. Interviews are dense with signals: the questions they ask, the ones they dodge, the way the panel interacts with each other, the public reviews that all mention the same thing in different words. Individually, each data point is easy to dismiss. Fed into Hermes across multiple conversations, they start to form a picture. The agent does not tell you whether to take the offer. But it does tell you what you are looking at — which is more than most people get before signing.\u003C\u002Fp>\n\u003Ch2>The Strategy Layer: Turning Reflection into Action\u003C\u002Fh2>\n\u003Cp>To turn reflection into action without handing your internal monologue over to a product manager in Mountain View, you have to talk about architecture. My memory store and retrieval layer run entirely on my personal disk, and I keep my sensitive reflection notes strictly local. For heavy inference tasks, I offload to Ollama Cloud, which operates on a transient, no-retention basis. But the context itself never lives anywhere but my own hardware.\u003C\u002Fp>\n\u003Cp>ChatGPT and Gemini are both adding memory now. I learned what that means the hard way. I asked Gemini for a translation for \u003Ca href=\"https:\u002F\u002Fwww.almamenor.com\">my band’s website\u003C\u002Fa>, and a few conversations later it casually mentioned my upcoming trip to Scotland — something I had never told it directly, pieced together from context scattered across unrelated chats. The convenience is real, and so is the surveillance.\u003C\u002Fp>\n\u003Cp>There is a gray area here that does not get talked about enough. When you reflect on a situation that involves other people, the data is personal but it is not only about you. Put that into a cloud chatbot and you are handing over information that traces back to other actors, often without their knowledge. The workplace is the most common version of this, and in many organizations it already violates AI usage policies. Even where it does not, the sane advice is the same: \u003Cstrong>don’t do it\u003C\u002Fstrong>. Most reflections only belong in your head. Some reflections belong on your own hardware. With Hermes, the platform sees inference requests, not a person.\u003C\u002Fp>\n\u003Cp>I told Hermes early on that I wanted neutral, detached responses. No cheerleading, no validation. Over time, it stopped needing the reminder. It learned the tone from how I reacted, not from anything I put in a config file.\u003C\u002Fp>\n\u003Cp>Here is what that looks like in practice. Once a month, I do a personal retrospective: a wheel of life across career, health, relationships, creative work. I write it in Obsidian. Hermes reads it through the career strategy lens. Over time, it has started connecting dots across months: a theme in how I describe my growth, a goal I keep mentioning but never acting on, a trajectory that only becomes visible when you lay six months side by side.\u003C\u002Fp>\n\u003Cdiv class=\"mermaid\">flowchart LR\n    R[Reflect] --&gt; E[Extract] --&gt; A[Apply] --&gt; R\n\u003C\u002Fdiv>\u003Cp>\u003Cstrong>Reflect.\u003C\u002Fstrong> I write down where I am: what is working, what is not, what I am avoiding. The agent asks questions, offers frames, connects it to past retrospectives. The goal is clarity, not answers.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Extract.\u003C\u002Fstrong> At the end of a session, I ask: what is the one thing worth carrying forward? Sometimes it is a principle, or just a question I need to sit with. Other times it is a raw next step. The agent helps me name it and save it.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Apply.\u003C\u002Fstrong> The next time a similar situation arises, I have the crystallized insight ready. Not as a rule to follow blindly, but as a lens to see through.\u003C\u002Fp>\n\u003Ch2>What This Changes\u003C\u002Fh2>\n\u003Cp>The shifts above did not come from a single conversation. They accumulated over weeks, across sessions, with the agent returning to earlier points each time a new piece of the picture emerged. A chatbot would have given the same advice once and moved on. Hermes held the thread — it turns context into stable knowledge just by talking to it.\u003C\u002Fp>\n\u003Cp>There is a deeper point here about what persistent memory actually does. When you document a reflection, you are creating a reference point that future conversations can connect to. Reading one monthly retrospective tells you how you felt on a Tuesday. Having an agent read six of them side by side, and the thing you have been actively avoiding for half a year gets harder to ignore. The memory turns sporadic observations into something you can act on.\u003C\u002Fp>\n\u003Ch2>The Risk I Am Watching For\u003C\u002Fh2>\n\u003Cp>I wrote earlier about \u003Ca href=\"\u002Fblog\u002Fai-rumination\u002F\">AI Rumination\u003C\u002Fa>, the dynamic where AI expands uncertainty instead of resolving it. Persistent memory amplifies both the benefits and the risks of this.\u003C\u002Fp>\n\u003Cp>The upside is obvious: it connects dots across sessions you would otherwise miss. The trap is subtler. Because the agent trains on your own chat logs, it can quietly institutionalize your blind spots, feeding your own biases back to you disguised as objective analysis.\u003C\u002Fp>\n\u003Cp>The mitigation is the same one I described in that post: \u003Cstrong>anchor the conversation to decisions\u003C\u002Fstrong>. Every session should produce a next step. Reflection is good, but rumination is a loop that will keep you stuck. The difference is whether you are moving toward closure.\u003C\u002Fp>\n\u003Chr>\n\u003Cp>\u003Cem>This post was written in conversation with Hermes Agent, which also holds the persistent memory it describes.\u003C\u002Fem>\u003C\u002Fp>\n","How an AI agent with persistent memory transforms from a chatbot into a long-term thinking partner for engineering leadership and decision making.",{"id":38,"slug":39,"title":40,"date":41,"tags":42,"image":47,"content":48,"excerpt":49,"section":20,"series":21,"series_part":21,"lang":22},1835,"the-thinking-trap","The Thinking Trap","2026-05-26T00:00:00.000Z",[43,44,45,32,46],"Career Growth","Architecture","Senior IC","Organizational Design","\u002Fimg\u002Fblog-thinking-trap.png","\u003Cp>Recently, I wrote a cost analysis for a data platform migration. The original proposal moved compute from one cloud paradigm to another. My counter-proposal scaled the existing solution vertically instead. The original had a 33-year payback period. Mine would have cut costs dramatically. Nobody asked me to do the analysis. I did it to reason through the problem and prove a point to myself. The math seemed correct. The architectural conclusion was wrong.\u003C\u002Fp>\n\u003Cp>I couldn’t see the event-driven microservices context that made the proposed architecture sound. My reasoning was sound. The information I had was incomplete.\u003C\u002Fp>\n\u003Cp>I should have talked to the architect before doing any analysis. That’s on me. Sometimes you just want to challenge yourself and see how far you can get. But the deeper questions stuck with me: at what point do you start thinking at the wrong level, expanding into other people’s territory? When are you stepping outside your lane? Will it bite you later?\u003C\u002Fp>\n\u003Cp>I like to think at the “why” level. Architecture, strategy, trade-offs, system design. But currently, I’m positioned at the “how” level. Implementation, tickets, sprints, pull requests. This isn’t just a personal failing. It’s a structural misalignment that affects every senior individual contributor (IC) whose natural mode is architectural thinking.\u003C\u002Fp>\n\u003Cp>I call it the Thinking Trap.\u003C\u002Fp>\n\u003Ch2>The Four Misalignments\u003C\u002Fh2>\n\u003Cp>I’ve identified four misalignments that create structural frustration.\u003C\u002Fp>\n\u003Ch3>1. Thinking vs. Information Access\u003C\u002Fh3>\n\u003Cp>You’re expected to reason about systems you don’t have full visibility into, or haven’t come across yet.\u003C\u002Fp>\n\u003Cp>You see patterns. You ask questions. You draw conclusions. But the conclusions are only as good as the data you have access to. A senior engineer isn’t always invited to the rooms where long-term strategy is mapped out. When you opine without that context, you’re reasoning from partial data. Sometimes the data you need lies in plain sight. Tribal knowledge that gives you important cues. Blind spots exist everywhere, at every level.\u003C\u002Fp>\n\u003Cp>Learning to navigate an organization for context is itself a skill, one I’m still developing. It’s not just about hunting down documents. It’s about building relationships so that people proactively share context with you. Treating that friction as an excuse for producing a flawed analysis is a trap of its own. When you’re wrong because of missing data, it looks like overstepping. When you’re right, it looks like you got lucky.\u003C\u002Fp>\n\u003Cp>If you’re asked to think critically about a system, produce a decision record, and take ownership of the outcome, you need this skill sharpened. The stakeholders who read your document won’t care about your skill issues.\u003C\u002Fp>\n\u003Cp>If you’re doing it for your own learning, asking “what if I were the one designing this system?”, take the opportunity to sharpen your ability to find context, observe how the process flows in real life, and do a postmortem after the fact. Compare the solutions. Learn from the outcomes.\u003C\u002Fp>\n\u003Ch3>2. Thinking vs. Output Capacity\u003C\u002Fh3>\n\u003Cp>Architectural thinking produces insights, decision records, and strategic recommendations. But your role is primarily measured in PRs, story points, and delivered features.\u003C\u002Fp>\n\u003Cp>The thing you’re trying to get better at isn’t what you’re explicitly evaluated on. Your performance review rarely has a line item for “produced an Architecture Decision Record (ADR) that prevented a six-figure mistake.” It has a line item for “delivered X features on time.”\u003C\u002Fp>\n\u003Cp>If I can’t explain why a decision record mattered in terms my manager or the business cares about, that’s my failure, not the review system’s. The tension is real. Architectural work doesn’t fit neatly into sprint metrics. But the responsibility to translate its value into business outcomes is just as real. Otherwise, you feel like you’re doing two jobs: the “how” job you’re paid for, and the invisible “why” job you can’t stop doing. You may also start feeling impostor syndrome.\u003C\u002Fp>\n\u003Cp>Doing invisible work carries a heavy context-switching cost. It takes a toll on your mental energy and decision quality. The secret is calibration. Don’t underdeliver, don’t overpromise. Keep commitments. Learn and experiment only when you have enough time.\u003C\u002Fp>\n\u003Cp>Sometimes “stay in your lane” is code for “you haven’t delivered enough trust yet to earn a voice in this conversation.” Other times it’s genuine structural rigidity. Learning to tell the difference is part of the growth. And the only reliable way to earn that trust is to execute flawlessly at your current scope first.\u003C\u002Fp>\n\u003Ch3>3. The Idea Graveyard\u003C\u002Fh3>\n\u003Cp>I’ve contributed to multiple “idea graveyards.” Those backlog buckets where proposals go to wait for later refinement and discussion. Confluence pages, JIRA tickets, MkDocs repositories. I’ve done them all. I generated prototypes, worked on them in great detail, only to watch them be completely ignored. I thought the content would speak for itself. I thought people would recognize the necessity of what I’d written with such care. And because most people in teams don’t step out of their comfort zone, I’d gladly share my thoughts just to avoid keeping the room silent.\u003C\u002Fp>\n\u003Cp>In reality, I probably think I’m smarter than I really am. Coupled with occasional team dynamics, this led to a distorted view of myself as the only person capable of solving all the problems.\u003C\u002Fp>\n\u003Cp>My biggest recurring mistake was letting ideas sit and rot instead of actively discussing them. One at a time, aligned with what the team, the manager, or the stakeholders already consider the next priority. Reading the room is a skill. Humans are political. Pre-wire the key people before the meeting. Frame the proposal in terms the decision-maker cares about — cost for the CFO, risk for the CTO, velocity for the VP of Engineering. Producing too much, too fast is just noise. It wastes time and energy.\u003C\u002Fp>\n\u003Ch3>4. Thinking vs. Career Trajectory\u003C\u002Fh3>\n\u003Cp>The skills that make you valuable at the “how” level are not the exact skills that get you promoted to the “why” level.\u003C\u002Fp>\n\u003Cp>Speed, volume, reliability. Those get you to senior. Judgment, systemic influence, and architectural vision. Those get you beyond it. You have to demonstrate competence at a level you’re not yet positioned to operate at, while still delivering at your current level. It’s the career equivalent of needing experience to get the job, but needing the job to get experience.\u003C\u002Fp>\n\u003Cp>Doing this from the “how” position means doing it without structural authority. It’s the hardest possible way to build the evidence you need for promotion, which is why thinking strategically about your career path matters early, not after two years of friction.\u003C\u002Fp>\n\u003Cp>I’ve produced Architecture Decision Records on FinOps, compliance, and ingestion architecture. All Staff-level artifacts, all from a Senior role. None of them moved the needle on my title. They built the case, but the case lives in documents, not in how the organization sees me. That’s the trap: the evidence is real, but it’s invisible unless you actively surface it. And surfacing it takes time; titles move on years, not documents.\u003C\u002Fp>\n\u003Ch2>Recognizing Which One Has You\u003C\u002Fh2>\n\u003Cp>Not all four hit at once. At any given moment, one is the primary source of frustration. The first step is diagnosing which one.\u003C\u002Fp>\n\u003Cp>If you’re drawing conclusions but feel unsure because you know you’re missing context, it’s \u003Cstrong>Information Access\u003C\u002Fstrong>. Ask for architectural briefings. Request access to upstream design docs. Frame it directly: “I want to make better implementation decisions today so we don’t rewrite this tomorrow.”\u003C\u002Fp>\n\u003Cp>If you’re producing analyses but feeling invisible, it’s \u003Cstrong>Output Capacity\u003C\u002Fstrong>. Don’t let your architectural work stay in a private scratchpad. Share design docs with your tech lead and engineering manager. Quantify the risk mitigated. Build the paper trail.\u003C\u002Fp>\n\u003Cp>If you’re generating proposals that vanish into a backlog never to be seen again, it’s the \u003Cstrong>Idea Graveyard\u003C\u002Fstrong>. Stop producing in bulk. Pick one idea, discuss it actively, and align it with what the team or stakeholders already care about. Reading the room matters more than writing the perfect document.\u003C\u002Fp>\n\u003Cp>If you’re doing the work but not getting the title, it’s \u003Cstrong>Career Trajectory\u003C\u002Fstrong>. Build an internal evidence portfolio. Document the decisions you influenced, the mentorship you provided, and the architectural landmines you helped the team avoid.\u003C\u002Fp>\n\u003Ch2>What to Do About It\u003C\u002Fh2>\n\u003Cp>If you recognize yourself in this pattern, these six principles are crucial:\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Master your current scope first.\u003C\u002Fstrong> None of this matters if you’re dropping tickets. An architect who can’t ship is just a commentator. The fastest way to earn a seat at the architectural table is to be so ruthlessly reliable at the “how” level that people actively want your brain on harder problems.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Make the invisible visible.\u003C\u002Fstrong> Every analysis, every design doc, every governance assessment. Document it, share it, and include it in your evidence file. The “why” work you are doing out of necessity \u003Cem>is\u003C\u002Fem> your promotion case. Tie it directly to team velocity or cost savings.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Talk to your manager. Frequently.\u003C\u002Fstrong> Don’t assume they see the invisible work. Don’t assume they know you want to grow toward architecture. Have the explicit conversation. Ask what the bar looks like for the next level and where you stand against it. And when you’re unsure whether your “why” work is valued or seen as a distraction, ask directly: \u003Cem>do you want me to keep doing this, or is it against what you want from my role and expertise?\u003C\u002Fem> Clarity is cheaper than resentment.\u003C\u002Fp>\n\u003Cp>If your manager can’t or won’t have this conversation, find someone who will. A skip-level leader, a mentor in another team, a senior architect who’s walked the path. Career growth shouldn’t depend on a single relationship. Build a small circle of people who see your work and can advocate for it.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Build external clarity in parallel.\u003C\u002Fstrong> Write articles and give talks, but do it to clarify your own thinking. If that builds a reputation outside your company, it’s a valuable side effect, not the primary goal. Your promotion case must live on internal impact; external writing simply helps you articulate that impact more clearly.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Champion one idea at a time.\u003C\u002Fstrong> The Idea Graveyard fills up when you produce faster than you advocate. Pick one proposal. Discuss it actively. Align it with what the team or stakeholders already care about. A single idea that ships is worth more than ten that sit in a backlog forever.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Pay attention as the years go by.\u003C\u002Fstrong> I’ve watched this pattern play out across multiple organizations. Early on, the friction is motivating — you’re learning, producing, proving yourself. As time passes, the same friction can become draining. The analyses get harder to produce without the right context. The stakeholder tension compounds. The career trajectory stalls. There’s no fixed deadline, but there is a point where the cost of staying starts to outweigh the cost of leaving. Don’t set it in stone. Just check in with yourself honestly, year by year. Is the gap closing, or is it widening?\u003C\u002Fp>\n\u003Ch2>For Managers Reading This\u003C\u002Fh2>\n\u003Cp>If you manage a senior IC who thinks like this, the cheapest retention move you can make is simple: \u003Cstrong>give them access to the architectural picture.\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>Not a promotion. Not an immediate title change. Just the information.\u003C\u002Fp>\n\u003Cp>Invite them to the design reviews. Share the long-term roadmap. Let them see the full context. The cost to you is zero. The alternative is losing a high-performer who is already doing Staff-level thinking on a Senior-level salary. They will eventually notice the disparity.\u003C\u002Fp>\n\u003Ch2>What This Costs the Organization\u003C\u002Fh2>\n\u003Cp>The Thinking Trap isn’t just a career problem for individuals. It’s a quiet tax on the business.\u003C\u002Fp>\n\u003Cp>When senior ICs can’t access the architectural picture, decisions get made without their input. The same mistakes repeat. The same shortcuts accumulate. The same technical debt compounds. The people who could have flagged it early weren’t in the room.\u003C\u002Fp>\n\u003Cp>When their “why” work goes unrecognized, they stop producing it. Not because they stop caring, but because the system trained them that it doesn’t count. The ADRs dry up. The FinOps analyses stop. The governance assessments disappear. The organization loses a layer of strategic thinking it never knew it had.\u003C\u002Fp>\n\u003Cp>When they burn out and leave, the cost is concrete: recruitment, onboarding, lost context, delayed projects. But the deeper cost is the signal it sends to everyone who stays. \u003Cem>Strategic thinking isn’t rewarded here. Stay in your lane. Keep your head down.\u003C\u002Fem> That’s how organizations that were once fast become slow.\u003C\u002Fp>\n\u003Cp>The cheapest fix is the one already described: give them access. The second cheapest is recognizing the work they’re already doing. Both cost nothing. Both prevent a loss that’s expensive to replace.\u003C\u002Fp>\n\u003Ch2>Closing the Gap\u003C\u002Fh2>\n\u003Cp>I am highly aware this perspective can risk sounding entitled. Like the classic “the organization doesn’t recognize my genius” complaint.\u003C\u002Fp>\n\u003Cp>Much of this frustration is self-inflicted. It comes from a communication gap we, as engineers, have to learn to close. It requires learning the business metrics that matter and earning the political capital required to influence systems larger than our immediate sprint.\u003C\u002Fp>\n\u003Cp>But some of the friction \u003Cem>is\u003C\u002Fem> structural.\u003C\u002Fp>\n\u003Cp>Companies frequently hire senior engineers for their judgment and experience, then position them in execution frameworks that isolate them from the context needed to apply that judgment effectively. The result is a quiet pipeline of burned-out senior talent. Naming this pattern isn’t about demanding a free pass to the strategy table; it’s about figuring out whether the friction you feel today is a gap in your own execution, or a structural boundary you need to learn to break through.\u003C\u002Fp>\n","When your natural mode is architectural thinking but your role measures you on delivery velocity, you're not failing. You're misaligned.",{"id":51,"slug":52,"title":53,"date":54,"tags":55,"image":58,"content":59,"excerpt":60,"section":20,"series":21,"series_part":21,"lang":22},1820,"ai-rumination","AI Rumination: How Overthinking with LLMs Steals Brain Power and Decision Quality","2026-04-30T00:00:00.000Z",[29,30,56,57],"Mental Health","Impostor Syndrome","\u002Fimg\u002Fblog-ai-rumination.png","\u003Cblockquote>\n\u003Cp>\u003Cstrong>Note:\u003C\u002Fstrong> \u003Cem>“AI rumination”\u003C\u002Fem> is not a formally recognized construct in clinical psychology or psychiatry. Here, the term is used informally to describe a pattern of \u003Cstrong>rumination and reassurance-seeking\u003C\u002Fstrong> that can be amplified by AI-generated output. It is not a diagnostic label.\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Ch2>Why I Call It \u003Cem>AI Rumination\u003C\u002Fem>\u003C\u002Fh2>\n\u003Cp>There’s a familiar mental pattern where a thought doesn’t resolve. You go over it again, slightly differently each time, expecting that the next pass will unlock something new.\u003C\u002Fp>\n\u003Cp>In psychology, this is called \u003Cstrong>rumination\u003C\u002Fstrong>: a repetitive, difficult-to-disengage form of thinking that doesn’t converge on an outcome.\u003C\u002Fp>\n\u003Cp>Large language models interact with this pattern in a particular way. They don’t close the loop—they expand it. Every prompt generates more plausible explanations and angles that feel like progress without actually resolving anything. The result is a feedback loop that increases cognitive load and prolongs indecision.\u003C\u002Fp>\n\u003Cp>I am not naming AI rumination as a clinical category, but rather as a shorthand for a pattern that becomes easier to understand once you’ve experienced it.\u003C\u002Fp>\n\u003Ch2>When the Loop Took Over\u003C\u002Fh2>\n\u003Ch3>The Bandmate\u003C\u002Fh3>\n\u003Cp>A former bandmate stopped appearing at rehearsals for months—no explanation, no messages, just silence.\u003C\u002Fp>\n\u003Cp>I kept replaying past conversations, trying to figure out what had gone wrong. The question became a loop: “What did I do wrong? Nothing, right?” Instead of letting the uncertainty sit, I kept feeding it into an AI chatbot, looking for reassurance that I hadn’t missed something obvious.\u003C\u002Fp>\n\u003Cp>When I finally confronted him, the response was abrupt. I ended the band and cut ties on the spot. The lack of communication had felt like the central issue, and in the moment, I felt justified in reacting strongly.\u003C\u002Fp>\n\u003Cp>With distance, though, I can see how much rumination had amplified the intensity of my response. There was no clear alignment between that reaction and any rational or strategic goal for preserving the three years we had invested in the band.\u003C\u002Fp>\n\u003Cp>Later, I learned he had been planning to leave town and simply hadn’t communicated it. With that context, my reaction feels disproportionate in hindsight. The band could likely have continued with a replacement, and the friendship might have been preserved. People move on from projects all the time.\u003C\u002Fp>\n\u003Ch3>The Architecture Decision\u003C\u002Fh3>\n\u003Cp>In a previous team, I respectfuly disagreed with the engineering manager about a core data modeling approach, that would shape years of data pipelines and reports to come.\u003C\u002Fp>\n\u003Cp>I had (and still have) a strong belief that the Ralph Kimball methodology represents one of the most robust approaches to data warehousing. With that framing, I turned to an LLM and asked it to take on the role of a “grumpy Kimball purist” and argue against the alternative.\u003C\u002Fp>\n\u003Cp>It did exactly that. The argument was clean, confident, and hard to refute. I revisited it multiple times, refining prompts and strengthening the case. I reinforced my prior knowledge of Kimball practices, and my assumptions that it was the absolute best fit.\u003C\u002Fp>\n\u003Cp>What I didn’t account for in AI prompting were the surrounding constraints: the team’s skill set, management direction, and my colleague’s experience managing different systems and tools. The solution that moved forward wasn’t what I would have chosen. It wasn’t theoretically optimal, but it was good enough to meet most of the company’s reporting needs.\u003C\u002Fp>\n\u003Cp>There’s also a structural reality in team environments: not every decision is yours to make. Influence has boundaries. If you’re not part of the final decision-making process, responsibility doesn’t fully sit with you either. That doesn’t mean disengaging or withholding input. It means recognizing where your role ends. If things don’t work out, your understanding doesn’t disappear, but neither does the distribution of accountability.\u003C\u002Fp>\n\u003Cp>Looking back, the issue was that I used AI to ruminate on a position that had already formed, rather than to test its limits or understand the trade-offs more fully, and I ended up wasting time in the process. I lost count of how many times I revisited the approach, repeatedly stress-testing it in my head and with prompts, even guesstimating the amount of money potentially wasted on storage and compute.\u003C\u002Fp>\n\u003Cp>In this particular case, the worst outcome of my rumination was definitely cynicism. I found myself internally expecting the alternative to fail, as if that would validate my original view or prove my worth. In that frame, it becomes easy to mentally split approaches into “how we do things here” versus “the right way”, even when reality is more constrained and less absolute.\u003C\u002Fp>\n\u003Ch2>Seeing the Loop in Action\u003C\u002Fh2>\n\u003Cp>The mechanisms behind it are well understood and some individuals may be more prone than others.\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Intolerance of uncertainty\u003C\u002Fstrong>: ambiguity is uncomfortable, so the mind keeps searching for resolution\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Reassurance-seeking\u003C\u002Fstrong>: asking for confirmation even when no new information is available\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Cognitive biases\u003C\u002Fstrong>: especially confirmation bias and the illusion of explanatory depth\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Metacognitive beliefs\u003C\u002Fstrong>: the assumption that continued analysis will eventually produce certainty\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>With AI, most questions return an answer that is coherent, structured, and convincing. Over time, it becomes harder to recognize when you’re actually learning versus just circling the same idea from different angles.\u003C\u002Fp>\n\u003Cp>You spend more time reviewing generated arguments than moving forward. Decisions stretch longer than they should. Confidence starts to come from how persuasive an answer sounds, rather than how well it reflects reality.\u003C\u002Fp>\n\u003Cp>When nothing feels resolved, the instinct is to try again: rephrase the prompt, explore another angle, generate a stronger argument. The loop repeats.\u003C\u002Fp>\n\u003Cp>That pattern doesn’t always stay contained. It can show up in team discussions as well—bringing increasingly refined questions or critiques about decisions that have already been explored. The intention is usually to improve outcomes, but the effect is often different. When repeated without new information or clear alternatives, constant questioning tends to create friction without changing direction. Conversations slow down, alignment becomes harder, and the signal-to-noise ratio drops.\u003C\u002Fp>\n\u003Cp>Over time, this can start to affect trust. What feels like careful analysis on one side can be experienced as churn on the other. Teams may become less responsive, not necessarily because the questions are invalid, but because they don’t appear to lead anywhere actionable.\u003C\u002Fp>\n\u003Ch2>Reflection vs. Rumination\u003C\u002Fh2>\n\u003Cp>Not all repeated thinking is a problem.\u003C\u002Fp>\n\u003Cp>Reflection is usually anchored to something: evidence, a decision, a next step. It tends to move toward closure.\u003C\u002Fp>\n\u003Cp>Rumination feels different. It repeats without resolving. The question shifts slightly each time, but the underlying uncertainty stays intact.\u003C\u002Fp>\n\u003Cp>The system tends to align with the framing you give it. Not perfectly, but enough that it reinforces the direction you’re already leaning toward.\u003C\u002Fp>\n\u003Cp>This is functionally similar to reassurance loops seen in anxiety-related patterns. The act of asking provides temporary relief, but it doesn’t resolve the underlying uncertainty.\u003C\u002Fp>\n\u003Cp>The important distinction is that this is a behavioral pattern, not a clinical condition. The similarity is structural, not diagnostic.\u003C\u002Fp>\n\u003Ch2>Interrupting the Loop\u003C\u002Fh2>\n\u003Cp>In practice, a few things tend to help:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Deciding in advance how long you’ll explore a question before acting\u003C\u002Fli>\n\u003Cli>Writing down what would actually change your mind before asking the model\u003C\u002Fli>\n\u003Cli>Bringing another person into the conversation, especially when you notice you’re looking for agreement rather than information\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>The shift is subtle but important: from generating more interpretations to testing reality.\u003C\u002Fp>\n\u003Ch2>On Adversarial Prompting\u003C\u002Fh2>\n\u003Cp>Using AI to explore worst-case scenarios can be useful in a controlled context.\u003C\u002Fp>\n\u003Cp>But when used as a personal check, it tends to amplify negativity, giving disproportionate weight to unlikely negative outcomes.\u003C\u002Fp>\n\u003Cp>Over time, that can skew perception in a way that reinforces the loop rather than breaking it.\u003C\u002Fp>\n\u003Ch2>Where AI Actually Helps\u003C\u002Fh2>\n\u003Cp>None of this means AI-driven iteration is inherently problematic.\u003C\u002Fp>\n\u003Cp>Used well, it can:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>expand the solution space\u003C\u002Fli>\n\u003Cli>accelerate learning\u003C\u002Fli>\n\u003Cli>generate hypotheses that can be tested\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2>When It Becomes a Problem\u003C\u002Fh2>\n\u003Cp>There isn’t a clean boundary, but certain signals tend to show up together:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>You keep revisiting the same question without getting closer to a decision\u003C\u002Fli>\n\u003Cli>The process increases tension instead of reducing it\u003C\u002Fli>\n\u003Cli>You delay acting because you feel like you’re “almost there”\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>At that point, the issue is usually difficulty tolerating the uncertainty that remains.\u003C\u002Fp>\n\u003Cp>Like most cognitive patterns, this one is easier to recognize in hindsight than in the moment.\u003C\u002Fp>\n\u003Ch2>Parting Thought\u003C\u002Fh2>\n\u003Cp>AI reflects what you bring into it. When used deliberately, it can sharpen thinking. When used to resolve uncertainty that can’t be resolved through more analysis, it tends to do the opposite.\u003C\u002Fp>\n\u003Cp>Recognizing that difference is less about discipline and more about awareness. Once you see the \u003Cstrong>rumination pattern\u003C\u002Fstrong>, it becomes easier to step out of it.\u003C\u002Fp>\n","When we let AI become a mirror for our insecurities, we end up in a loop of rumination that harms both ourselves and the teams around us.",{"id":62,"slug":63,"title":64,"date":65,"tags":66,"image":69,"content":70,"excerpt":71,"section":20,"series":21,"series_part":21,"lang":22},1826,"overengineering-notes","The Hidden Dangers of Productivity Hype","2026-03-26T00:00:00.000Z",[30,67,31,29,68],"Obsidian","Creativity","\u002Fimg\u002Fblog-overengineering-notes.png","\u003Cp>I recently stumbled on a YouTube channel promoting an elaborate Obsidian workflow—diagrams, taxonomies, nested templates—with a Patreon attached. My gut reaction was: \u003Cstrong>this is overengineering\u003C\u002Fstrong>. Not because the ideas are bad, but because it’s not always clear whether all that structure genuinely supports thinking or just creates the appearance of insight.\u003C\u002Fp>\n\u003Cp>I say this as someone who lived that lifestyle. A decade ago I devoured every article on the Asian Efficiency blog, cycled through countless note-taking apps, installed every Pomodoro timer and distraction-free writing screen I could find. If there was a system that promised to optimize how I worked or thought, I tried it.\u003C\u002Fp>\n\u003Ch2>Structure is a tool, not a cage\u003C\u002Fh2>\n\u003Cp>I used Obsidian for years and built an extensive \u003Ca href=\"https:\u002F\u002Ffortelabs.com\u002Fblog\u002Fpara\u002F\">PARA-style\u003C\u002Fa> system for almost everything: journaling, monthly self-reviews, tracking progress on personal projects, structuring my engineering work, even \u003Ca href=\"\u002Fblog\u002Fpractice-system-obsidian\u002F\">scripting music practice sessions\u003C\u002Fa>. I wouldn’t start any task without a plan or attend a meeting without pre-structuring my thoughts. I was getting things done, but I wasn’t enjoying the process—and I began to question why.\u003C\u002Fp>\n\u003Cp>I even built a personal dashboard that pulled data from my watch, my practice sessions, and my activity logs across work and side projects—a single pane of glass for my own productivity. It felt like the ultimate system. Then I added a metric based on total hours worked per week across everything: my job, side projects, and music development. I was averaging well over 70 hours. The WHO and ILO \u003Ca href=\"https:\u002F\u002Fwww.who.int\u002Fnews\u002Fitem\u002F17-05-2021-long-working-hours-increasing-deaths-from-heart-disease-and-stroke-who-ilo\">consider 55 hours per week a serious health hazard\u003C\u002Fa>, linked to a 35% higher risk of stroke and 17% higher risk of heart disease. That number was the wake-up call I needed to stop living inside the cage I’d built for myself.\u003C\u002Fp>\n\u003Cp>The irony wasn’t lost on me. The system I’d built to support thinking had quietly become a prerequisite for it. If the structure wasn’t there, neither was the thinking. In a way it short-circuited my ability to just sit, reflect, and explore ideas freely.\u003C\u002Fp>\n\u003Ch2>When the system becomes the driver\u003C\u002Fh2>\n\u003Cp>Over time I noticed something deeper: the more I structured my vault, the more I constrained my intuition and intrinsic motivation. What was meant to support reflection turned it into a prerequisite-driven process: a pipeline of templates and reviews that had to be “fed” before any real thought could happen.\u003C\u002Fp>\n\u003Cp>That’s the risk I rarely see acknowledged in the productivity space. People talk about the benefits of structure endlessly, but almost nobody talks about what happens when the structure starts driving you instead of the other way around. I’m fairly certain I’m not the only one who ended up there.\u003C\u002Fp>\n\u003Ch2>The joy of forgetting\u003C\u002Fh2>\n\u003Cp>When I started taking notes more sparingly, something unexpected happened. The anxiety of losing ideas was replaced by the \u003Cstrong>joy of rediscovering them\u003C\u002Fstrong>. Not every thought deserves to be captured immediately. Some need time to mature, to be forgotten and remembered again, reshaped by whatever you’ve lived through since.\u003C\u002Fp>\n\u003Cp>Letting go of compulsive capture helped me relearn how to remember. It made me less dogmatic in how I process information and more inclined to think through problems only when they’re genuinely ready to be worked through, not because a template demanded it.\u003C\u002Fp>\n\u003Cp>There’s a name for this in psychology: \u003Cstrong>incubation\u003C\u002Fstrong>. Letting ideas rest before consciously revisiting them often leads to richer, more original thinking. The note-taking orthodoxy, with its emphasis on capturing everything immediately, can actively work against that process.\u003C\u002Fp>\n\u003Ch2>Scarcity isn’t what it used to be\u003C\u002Fh2>\n\u003Cp>There’s a broader context worth acknowledging. A decade ago, the value proposition of dense personal knowledge bases was obvious: information was scattered, retrieval was hard, and the person who had it written down won. That scarcity justified elaborate systems.\u003C\u002Fp>\n\u003Cp>Today, that equation has shifted. LLMs and modern search have made retrieval near-trivial for most technical knowledge. Dense notes on APIs, frameworks, and configuration details are redundant. What retains value is \u003Cstrong>insight, synthesis, and perspective\u003C\u002Fstrong>: the things that come from thinking, not from filing.\u003C\u002Fp>\n\u003Cp>A rigid vault optimized for information scarcity can actually become a liability in an era of information abundance. The hours spent maintaining it would be better spent developing the kind of judgment and creative thinking that no retrieval system can replicate.\u003C\u002Fp>\n\u003Ch2>Systems should serve thinking, not replace it\u003C\u002Fh2>\n\u003Cp>None of this is an argument against note-taking, it’s a tool I still use, just more selectively. The argument is against letting any system dictate the process of thought itself.\u003C\u002Fp>\n\u003Cp>Use structure for what it genuinely supports: capturing fleeting ideas that would otherwise be lost, planning work that benefits from a checklist, logging progress that future-you will thank you for. But don’t let the system become so elaborate that maintaining it crowds out the unstructured reflection where real insight happens.\u003C\u002Fp>\n\u003Cp>Overengineering productivity can undermine the very goals it’s meant to serve. That’s not a warning I read somewhere. It’s something I lived.\u003C\u002Fp>\n","How elaborate productivity systems can quietly erode motivation, replace genuine thinking with process, and push you past healthy working limits.",{"id":73,"slug":74,"title":75,"date":76,"tags":77,"image":79,"content":80,"excerpt":81,"section":20,"series":21,"series_part":21,"lang":22},1834,"the-ai-code-review-delusion","The AI Code Review Delusion: Why I Replaced My Multi-Agent Swarm with a Folder of Prompts","2026-03-16T00:00:00.000Z",[10,29,44,78],"LLMs","\u002Fimg\u002Fblog-ai-review-castle.png","\u003Ch2>1. Introduction: The Sycophancy Trap\u003C\u002Fh2>\n\u003Cp>It usually starts with an “AI-first Monday.” Your CEO watched a YouTube video over the weekend about how autonomous AI agents are going to replace entire engineering departments. Now, every new feature must be an “agent,” every process must be an “A2A (Agent-to-Agent) workflow,” and deterministic code is suddenly viewed as legacy software.\u003C\u002Fp>\n\u003Cp>I fell for it, too.\u003C\u002Fp>\n\u003Cp>I spent several weeks building a system called “Git Diff RAG” in my spare time: an over-engineered, multi-agent code reviewer designed to automate pull request (PR) reviews end-to-end. It was a massive, sprawling architecture of hallucinating text generators. And I built it directly into the trap of \u003Cstrong>LLM Sycophancy\u003C\u002Fstrong>.\u003C\u002Fp>\n\u003Cp>When you ask an LLM, \u003Cem>“How should I design a multi-agent RAG system for code review?”\u003C\u002Fem> it enthusiastically helps you build it. It suggests architectures, orchestrators, and vector databases. It never stops to ask the most important question: \u003Cem>“Do you actually need this?”\u003C\u002Fem>\u003C\u002Fp>\n\u003Cp>To be clear: having \u003Cem>some\u003C\u002Fem> automated review system: even a flawed one: is better than having none. It is vastly superior to the “LGTM” rubber-stamp culture or the non-existent coding hygiene that plagues so many teams. The goal of this project was learning and sharing those findings, and I certainly learned a lot. But we can do much better than building over-engineered systems out of probabilities.\u003C\u002Fp>\n\u003Cp>One antidote to this sycophancy trap? \u003Cstrong>Adversarial Prompting\u003C\u002Fstrong>. Try to roast your own idea using AI. If I had asked a model to act as a grumpy veteran coder and ruthlessly deconstruct my A2A plan, I probably would have saved myself weeks of over-engineering. Since this was a personal project, I could afford that luxury: though my wife, family, and music teachers might disagree. But what happens when you build these sprawling systems on company time?\u003C\u002Fp>\n\u003Ch2>2. The Over-Engineered System (What I Built)\u003C\u002Fh2>\n\u003Cp>On paper, “Git Diff RAG” was a masterpiece of modern AI hype.\u003C\u002Fp>\n\u003Cp>I ingested Git diffs, ran them through a RAG layer filled with vector search hits, architecture trees, and build tool output. I built a parliament of AI “personas”: an Architect agent, a Reviewer agent, a Security expert, a Frontend expert. The idea was that these agents would debate the diff, reach a consensus, produce revision artifacts, update the specs, and rewrite the documentation.\u003C\u002Fp>\n\u003Cp>When you combine multiple steps of analysis, modification, and documentation without accountability gates, you institutionalize subtle defects. If an AI agent proposes a change, another AI agent approves it, and a third AI agent writes the spec to match the change… who is responsible when production breaks?\u003C\u002Fp>\n\u003Cp>And that assumes the system acts logically. The worst footgun I experienced was due to a lack of guardrails. Early on, I ran the agent against a “vibe-coded” job search assistant I was actively using for real applications. My adversarial prompt had instructed the AI to ensure the code was “GitHub-ready.” The agent reviewed the code, concluded the idea wasn’t valuable enough to exist, and slipped an obscure PowerShell deletion command into its suggestions. I blindly accepted it, instantly wiping out my entire project.\u003C\u002Fp>\n\u003Cp>Interestingly, when I asked the AI how to prevent this, the sycophancy kicked back in. Instead of telling me to pause and reconsider why an autonomous code reviewer had unrestricted write access to a local file system in the first place, it happily suggested continuing with complex CLI SDKs, configuring read-only flags, and mounting virtual file systems. It focused entirely on engineering a \u003Cem>solution\u003C\u002Fem> to the problem instead of asking: \u003Cem>“Wait, what was the original premise here?”\u003C\u002Fem>\u003C\u002Fp>\n\u003Cp>Another weird idea suggested by the LLMs was an “intelligent prompt selection module.” The theory was that the system could dynamically compose the exact right prompt at runtime based on the code context. I ended up generating hundreds of prompt snippets for this system, and at some point, I was completely lost in my own architecture.\u003C\u002Fp>\n\u003Cp>I didn’t build an AI code reviewer. I built a \u003Cstrong>castle of AI prompts\u003C\u002Fstrong>.\u003C\u002Fp>\n\u003Ch2>3. Accountability: The Line Between Recommenders and Cancer Diagnoses\u003C\u002Fh2>\n\u003Cp>AI is inherently probabilistic. That is perfectly fine for tasks with rapid feedback loops and low stakes: like A\u002FB testing ad copy, building recommender systems, or even some high-frequency micro-trading. The cost of being wrong is effectively zero.\u003C\u002Fp>\n\u003Cp>But when an autonomous system blindly merges a data pipeline change that drops a production table, or worse, is diagnosing a patient with cancer, the math completely changes.\u003C\u002Fp>\n\u003Cp>Code review is not just about catching typos; it is an \u003Cstrong>accountability gate\u003C\u002Fstrong>. It is a governance checkpoint where an engineer says, \u003Cem>“I authorize this change.”\u003C\u002Fem> Treating code review like a generative task meant to be fully automated erases the vital boundary between critique, decision, and institutional authorization. High-consequence errors require a deterministic, auditable system.\u003C\u002Fp>\n\u003Ch2>4. Lost in the Middle: The Physics of Context Pollution\u003C\u002Fh2>\n\u003Cp>Why did all those “expert” agents produce such shallow, noisy, and often hallucinated reviews? Because my agents were shallow precisely because I was drowning them in context.\u003C\u002Fp>\n\u003Cp>I threw architecture diagrams, Jira tickets, repository metadata, and full build logs into the prompt alongside the diff. But LLMs do not get smarter with more context; they get smarter with higher \u003Cstrong>signal density\u003C\u002Fstrong>.\u003C\u002Fp>\n\u003Cp>This phenomenon is documented in the Stanford\u002FBerkeley paper \u003Cem>“Lost in the Middle: How Language Models Use Long Contexts”\u003C\u002Fem>. The researchers found that LLMs heavily index on information placed at the very beginning or the very end of a prompt. If you bury the actual important detail (the code diff) in the middle of a massive RAG blizzard of irrelevant vector search hits, the model’s attentional blind spot takes over. My over-engineered architecture was effectively an expensive way to blind the model to the very code it was supposed to review.\u003C\u002Fp>\n\u003Cp>In review systems, \u003Cstrong>precision beats volume\u003C\u002Fstrong>. An LLM that highlights a single critical SQL performance bottleneck with certainty is far more valuable than one that generates a checklist of twenty “best practice” hallucinations because it got distracted by unrelated documentation. Every marginally relevant file you add acts as camouflage for the bug you’re actually trying to catch.\u003C\u002Fp>\n\u003Ch2>5. The Antidote: Deterministic Workflows over Emergent Chaos\u003C\u002Fh2>\n\u003Cp>After two months of drowning in emergent chaos, I reached for the axe. I killed the agents, purged the vector database, and burned the RAG layer to the ground. I deleted the entire repository and started from zero.\u003C\u002Fp>\n\u003Cp>I was about to give up, convinced that my time spent on a worthless idea could not be reclaimed. Then I realized I had learned something valuable: the limitations of LLMs and the importance of deterministic workflows. Determined to do better, I decided to build the simplest thing that could possibly work using a procedural approach. I tested it relentlessly with adversarial prompting until I felt confident I wouldn’t repeat the same mistakes.\u003C\u002Fp>\n\u003Cp>I replaced the whole architecture with a simple \u003Ccode>.review\u003C\u002Fcode> folder full of plain text prompts and a \u003Cstrong>deterministic pipeline\u003C\u002Fstrong>, that you can add on a per-project basis.\u003C\u002Fp>\n\u003Cp>Instead of independent agents talking to each other, the pipeline became a sequence of operations that outputs verifiable, timestamped artifacts. Inside the \u003Ccode>.review\u003C\u002Fcode> directory, we added a \u003Ccode>runs\u002F\u003C\u002Fcode> folder. Every time a review is triggered, it creates a timestamped subfolder (e.g., \u003Ccode>runs\u002F2026-03-16_10-15-30\u002F\u003C\u002Fcode>) that stores the exact intermediate file produced by each step of the LLM timeline.\u003C\u002Fp>\n\u003Cp>This means you can audit exactly what the model saw and what it output at every phase:\u003C\u002Fp>\n\u003Col>\n\u003Cli>\u003Cstrong>Scan:\u003C\u002Fstrong> Identify risk areas based on the diff (e.g., does this touch auth? does this alter a core schema?).\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Review:\u003C\u002Fstrong> Produce claims about the code, but \u003Cem>only\u003C\u002Fem> with solid evidence. If a claim lacked a code snippet or a quoted spec, it was ignored.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Refine:\u003C\u002Fstrong> Remove weak or duplicate findings.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Summarize:\u003C\u002Fstrong> Output actionable conclusions for a human engineer to review.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>The core principle: Keep the workflow deterministic. Use LLMs for semantic extraction and analysis \u003Cem>within\u003C\u002Fem> strictly bounded steps, but require human accountability gates for any state changes. And most importantly, keep the receipts.\u003C\u002Fp>\n\u003Ch2>6. Practical Application: A \u003Ccode>.review\u003C\u002Fcode> Folder for Data Engineering\u003C\u002Fh2>\n\u003Cp>How does this actually look in practice? It looks like a folder of highly specific, role-based prompts that are invoked by a simple wrapper script (like a GitHub Action or a local Python script) based on the files touched in the PR.\u003C\u002Fp>\n\u003Cp>Here are simplistic examples of how you segment the reasoning instead of relying on one massive, confused agent. It starts with a \u003Cstrong>Meta Prompt\u003C\u002Fstrong>: essentially the system instructions for your orchestrator. That defines the workflow and enforces artifact persistence.\u003C\u002Fp>\n\u003Ch3>\u003Ccode>meta.prompt\u003C\u002Fcode>\u003C\u002Fh3>\n\u003Cpre>\u003Ccode class=\"language-text\">You are executing a deterministic, multi-stage pull request review. You will adopt three subsequent personas to analyze the provided diff.\n\nIMPORTANT SAFETY RULE:\n- You have read-only access to the repository.\n- You may NOT modify any files outside the `.review` folder.\n- All outputs, intermediate files, and artifacts must be written only to `.review\u002Fruns\u002F{{timestamp}}\u002F`.\n\nWorkflow:\n1. Load `architect.prompt`. Perform structural analysis. Save output to `architect_findings.md`.\n2. Load `reviewer.prompt`. Perform code-level review on the output of step 1. Save output to `reviewer_findings.md`.\n3. Load `verifier.prompt`. Audit all previous findings against the diff evidence. Save output to `verified_findings.md`.\n\nOnce all three files are written to the runs folder, output a final `summary.md` containing only the surviving, verified findings for human review. Do NOT approve or reject the code yourself.\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch3>\u003Ccode>architect.prompt\u003C\u002Fcode>\u003C\u002Fh3>\n\u003Cpre>\u003Ccode class=\"language-text\">You are analyzing a data engineering pull request. Your goal is structural analysis.\n- Identify if this change affects DAG dependency trees.\n- State whether the new data transformations maintain table idempotency.\n- Highlight any breaking schema changes (dropped columns, altered types).\n- Provide evidence for your claims from the provided diff. \nDo not approve or reject the code. Only state your findings.\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch3>\u003Ccode>reviewer.prompt\u003C\u002Fcode>\u003C\u002Fh3>\n\u003Cpre>\u003Ccode class=\"language-text\">You are conducting a code-level review on a SQL\u002Fdbt pull request.\n- Focus on SQL performance: Look for missing partition filters, expensive JOINs, or window function abuse.\n- Identify potential data quality issues: Are there protections against NULL values or fan-outs?\n- Check test coverage: Are there `not_null` and `unique` tests applied to newly created fundamental columns?\nQuote the exact lines of code from the diff that support your findings. If you have no findings, output &quot;NO_FINDINGS&quot;.\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch3>\u003Ccode>verifier.prompt\u003C\u002Fcode>\u003C\u002Fh3>\n\u003Cpre>\u003Ccode class=\"language-text\">You are an adversarial verifier. \nReview the output of the previous analysis alongside the original diff. \nFor every finding reported:\n1. Does the evidence in the diff match the claim?\n2. If the answer is no, mark the finding as &quot;HALLUCINATION&quot; and remove it.\nYou are the final filter before a human sees this review.\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch2>7. Conclusion: The Real Value of AI\u003C\u002Fh2>\n\u003Cp>After building a complex AI review system that completely spiraled out of control, the best solution was embarrassingly simple: deterministic workflows, structured prompts, and curated evidence—closer to the scientific method than the hype.\u003C\u002Fp>\n\u003Cp>One simple question could have saved me: before greenlighting a multi-agent RAG swarm, I should have asked, “What is the simplest deterministic workflow that solves this?” If you can’t answer that first, adding AI agents will only get you to the wrong destination faster.\u003C\u002Fp>\n\u003Cp>I had chased the hype, eager to learn multi-agent systems as a supposed silver bullet. In the end, it’s often wiser to let the dust settle and see which design patterns actually stand the test of time. Remember: you won’t be fired for skipping the latest AI trends.\u003C\u002Fp>\n","Why replacing an autonomous multi-agent swarm with deterministic prompts proved that predictable workflows beat agentic complexity in code reviews.",{"id":83,"slug":84,"title":85,"date":86,"tags":87,"image":90,"content":91,"excerpt":92,"section":20,"series":93,"series_part":94,"lang":22},1832,"spec-driven-bi-part2-semantic-gap","The Semantic Gap: Specifying Business Reality","2026-03-12T00:00:00.000Z",[10,88,16,89],"Business Intelligence","Semantic Layer","\u002Fimg\u002Fblog-semantic-gap-header.png","\u003Ch2>The Goal of This Post\u003C\u002Fh2>\n\u003Cp>In \u003Ca href=\"\u002Fblog\u002Fspec-driven-bi-part1-when-it-works\u002F\">Part 1\u003C\u002Fa>, we established that specification-driven development excels at generating physical infrastructure. You can write a configuration file, and a deterministic parser can reliably generate staging models, apply basic transformations, and deploy tests.\u003C\u002Fp>\n\u003Cp>But extending this same basic configuration approach to core business logic reveals its limitations.\u003C\u002Fp>\n\u003Cp>The “Semantic Gap” is the fundamental difference between specifying a \u003Cem>physical constraint\u003C\u002Fem> and defining \u003Cem>business reality\u003C\u002Fem>. Flat configuration files cannot capture the interwoven complexity of dimensional modeling—a challenge mature data teams address using dedicated Semantic Layers.\u003C\u002Fp>\n\u003Ch2>Physical Contracts vs. Semantic Reality\u003C\u002Fh2>\n\u003Cp>To understand the semantic gap, we must first look at what deterministic tooling already does well. Modern data teams use \u003Cstrong>strict data contracts\u003C\u002Fstrong> (e.g., \u003Ccode>schema.yml\u003C\u002Fcode> in dbt or native data platform contracts) to enforce the physical structure of a table:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-yaml\"># A physical data contract\nmodels:\n  - name: fct_transactions\n    columns:\n      - name: transaction_id\n        data_type: integer\n        constraints:\n          - type: not_null\n          - type: unique\n      - name: customer_id\n        data_type: integer\n        tests:\n          - relationships:\n              to: ref('dim_customers')\n              field: customer_id\n      - name: amount_usd\n        data_type: numeric(18,2)\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>This ensures column types are correct, nulls are prevented, and referential integrity holds. If the contract is violated, the pipeline fails. This is deterministic, automatable governance at the infrastructure layer.\u003C\u002Fp>\n\u003Ch3>The Metadata Layer: Catalogs and Information Schemas\u003C\u002Fh3>\n\u003Cp>There is also an incredibly valuable intermediate layer between raw physical tables and complex semantic query engines: \u003Cstrong>the Data Catalog\u003C\u002Fstrong>.\u003C\u002Fp>\n\u003Cp>By querying a modern data warehouse’s \u003Ccode>information_schema\u003C\u002Fcode> (or using tools like dbt docs, DataHub, and Alation), engineering teams can automatically extract referential relationships, primary keys, and constraints. Adding table- and column-level comments directly to the underlying SQL views or Python scripts means the Catalog can expose an always-up-to-date, highly visible version of the data model to the whole company.\u003C\u002Fp>\n\u003Cp>But there is a critical limit here: a data catalog provides discoverability, but it doesn’t quantify logic.\u003C\u002Fp>\n\u003Cp>It informs a human analyst that \u003Ccode>amount_usd\u003C\u002Fcode> is a number, but it doesn’t define the mathematical rules that prevent a BI query from returning incorrect results.\u003C\u002Fp>\n\u003Chr>\n\u003Ch2>The Nuances of Semantic Modeling\u003C\u002Fh2>\n\u003Cp>A true dimensional model—the kind that analysts can explore safely without generating silent errors—must encapsulate complex business logic. Here is what a simple physical spec fails to articulate:\u003C\u002Fp>\n\u003Ch3>1. What Is the Grain of This Fact Table?\u003C\u002Fh3>\n\u003Cp>Grain is the \u003Cstrong>atomic level of your fact table\u003C\u002Fstrong>. It answers the question: \u003Cem>What does one row represent?\u003C\u002Fem>\u003C\u002Fp>\n\u003Cp>Consider this definition:\u003C\u002Fp>\n\u003Cblockquote>\n\u003Cp>\u003Cem>“One row per transaction line item, per product, per day. Natural key: (product_id, transaction_id_line). Surrogate key: revenue_line_sk.”\u003C\u002Fem>\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Cp>Aggregating this table by \u003Ccode>product_category\u003C\u002Fcode> without respecting its grain can produce subtle, catastrophic errors. Analysts must understand the correct joins and cardinalities to avoid fan-outs or double-counting, which flat specifications cannot enforce.\u003C\u002Fp>\n\u003Chr>\n\u003Ch3>2. How Do You Handle Slowly Changing Dimensions (SCDs)?\u003C\u002Fh3>\n\u003Cp>Imagine a product gets recategorized from “Electronics” to “Accessories” on March 15th. Does historical revenue from February shift to Accessories, or does it stay in Electronics?\u003C\u002Fp>\n\u003Cblockquote>\n\u003Cp>\u003Cem>“Historical transactions use the category as it existed at the time of purchase (Type 2 SCD). Only transactions after March 15th use the new category.”\u003C\u002Fem>\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Cp>SCDs dictate how history is written. A physical spec knows that a \u003Ccode>category_id\u003C\u002Fcode> exists. But a semantic model must understand the \u003Cem>temporal relationship\u003C\u002Fem> between the fact event and the dimension state. If a system doesn’t understand that a join must include an \u003Ccode>active_daterange\u003C\u002Fcode> qualifier, the history of the data warehouse rewrites itself every time a source system updates a record.\u003C\u002Fp>\n\u003Chr>\n\u003Ch3>3. Is ‘amount_usd’ Fully Additivity?\u003C\u002Fh3>\n\u003Cp>Additivity determines what aggregations are mathematically valid:\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Measure\u003C\u002Fth>\n\u003Cth>Additivity\u003C\u002Fth>\n\u003Cth>Valid Operations\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>\u003Ccode>amount_usd\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>✅ Fully Additive\u003C\u002Ftd>\n\u003Ctd>\u003Ccode>SUM()\u003C\u002Fcode> across all dimensions\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>inventory_qty\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>⚠️ Semi-Additive\u003C\u002Ftd>\n\u003Ctd>Can \u003Ccode>SUM()\u003C\u002Fcode> within a period, but not across periods.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>unit_price\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>❌ Non-Additive\u003C\u002Ftd>\n\u003Ctd>Must \u003Ccode>AVG()\u003C\u002Fcode>, never \u003Ccode>SUM()\u003C\u002Fcode>.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003Cp>If an analyst writes \u003Ccode>SELECT SUM(unit_price)\u003C\u002Fcode> because a basic spec didn’t prevent it, the database will return a meaningless result. Defining additivity upfront prevents invalid aggregations and protects analysts from silent errors.\u003C\u002Fp>\n\u003Chr>\n\u003Ch3>4. What Are the Conformed Dimensions?\u003C\u002Fh3>\n\u003Cblockquote>\n\u003Cp>\u003Cem>“This fact table conforms to \u003Ccode>dim_product\u003C\u002Fcode> (Enterprise v2.0) and \u003Ccode>dim_date\u003C\u002Fcode>. All domain-specific facts must join through these conformed dimensions to ensure cross-departmental reporting matches.”\u003C\u002Fem>\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Cp>If the marketing team builds a \u003Ccode>fct_campaign_ROI\u003C\u002Fcode> table and creates their own local \u003Ccode>dim_product\u003C\u002Fcode>, their reports will inevitably disagree with the finance team’s reports. Specifying the physical columns doesn’t solve this; the organization needs an architectural constraint—a \u003Cstrong>conformed dimension strategy\u003C\u002Fstrong>—to ensure everyone is measuring the same entities.\u003C\u002Fp>\n\u003Chr>\n\u003Ch3>5. What Is the Business Definition of “Revenue”?\u003C\u002Fh3>\n\u003Cblockquote>\n\u003Cp>\u003Cem>“Revenue = settled_amount + pending_adjustments, excluding refunds issued &gt; 30 days post-purchase. Do NOT include internal transfers.”\u003C\u002Fem>\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Cp>A column named \u003Ccode>amount_usd\u003C\u002Fcode> hides edge cases that matter to accounting and compliance. Generic documentation often fails to capture the conditional logic required for business accuracy.\u003C\u002Fp>\n\u003Chr>\n\u003Cp>Governance requires a translation layer between the raw database and the business user. Enterprise BI tools like Cognos or BusinessObjects provided this, but their rigidity eventually drove teams toward the Modern Data Stack. We replaced heavy, GUI-driven tools with version-controlled SQL, but in the process, we lost the semantic abstraction itself.\u003C\u002Fp>\n\u003Cp>Data teams are now re-establishing these guardrails using code-first \u003Cstrong>Semantic Layers\u003C\u002Fstrong> like \u003Ca href=\"https:\u002F\u002Fcube.dev\u002F\">Cube\u003C\u002Fa> (a headless BI platform) or \u003Ca href=\"https:\u002F\u002Fwww.getdbt.com\u002Fproduct\u002Fsemantic-layer\">dbt Semantic Layer\u003C\u002Fa>.\u003C\u002Fp>\n\u003Ch3>The Reporting Tool “Solution”\u003C\u002Fh3>\n\u003Cp>Modern BI tools like Power BI or Tableau offer sophisticated semantic layers, but embedding logic directly into visualization tools creates vendor lock-in and decentralized silos. Business rules become inaccessible to downstream consumers like data scientists using Python or analysts querying via other platforms.\u003C\u002Fp>\n\u003Cp>A centralized, code-first Semantic Layer addresses this by establishing the abstraction below the BI tool and above the database, serving as a governed source of truth for all consumers.\u003C\u002Fp>\n\u003Cp>These tools succeed because they don’t just specify physical columns—they introduce \u003Cstrong>domain-specific abstractions\u003C\u002Fstrong> designed expressly for semantic concepts. Let’s look at how a mature semantic layer handles additivity and join paths:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-javascript\">\u002F\u002F A Semantic Layer configuration (e.g., Cube)\ncube(`transactions`, {\n  sql_table: `public.fct_transactions`,\n\n  joins: {\n    products: {\n      relationship: `many_to_one`, \u002F\u002F Prevents fan-out errors\n      sql: `${CUBE}.product_id = ${products.product_id}`,\n    },\n  },\n\n  measures: {\n    revenue: {\n      sql: `amount_usd`,\n      type: `sum`,           \u002F\u002F Explicitly fully additive\n    },\n    avg_price: {\n      sql: `price`,\n      type: `avg`,           \u002F\u002F System prevents invalid SUMs\n    },\n  },\n});\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>Here, the abstraction \u003Cem>understands\u003C\u002Fem> the business rules:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Aggregation type is codified:\u003C\u002Fstrong> The system structurally prevents \u003Ccode>SUM(price)\u003C\u002Fcode>.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Joins declare cardinality:\u003C\u002Fstrong> \u003Ccode>many_to_one\u003C\u002Fcode> tells the generation engine how to handle the grain correctly during a join.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2>Mapping the Control Boundaries: Spec vs. Code\u003C\u002Fh2>\n\u003Cp>To return to the original promise of spec-driven development—where does it actually fit? What does the spec control, and what does the code control?\u003C\u002Fp>\n\u003Cp>In a mature, correctly layered data architecture, you use different tools to govern different phases of execution:\u003C\u002Fp>\n\u003Col>\n\u003Cli>\u003Cstrong>The Physical Spec (Data Contracts \u002F \u003Ccode>schema.yml\u003C\u002Fcode>)\u003C\u002Fstrong>\n\u003Cul>\n\u003Cli>\u003Cstrong>Controls:\u003C\u002Fstrong> Database DDL. It enforces \u003Cem>what shapes\u003C\u002Fem> the data takes (types, nullability, unique keys, hard referential integrity).\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Goal:\u003C\u002Fstrong> Fails the pipeline if bad data arrives.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003C\u002Fli>\n\u003Cli>\u003Cstrong>The Execution Code (SQL \u002F Python)\u003C\u002Fstrong>\n\u003Cul>\n\u003Cli>\u003Cstrong>Controls:\u003C\u002Fstrong> The actual data transformations. It takes raw data and shapes it into dimensional models, handling edge cases and data cleansing.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>The Micro-Spec (Inline Comments):\u003C\u002Fstrong> Code isn’t just execution; it’s also the most accurate source of business logic. A one-line comment (\u003Ccode>-- Exclude test accounts prefix 'QA_'\u003C\u002Fcode>) explains \u003Cem>why\u003C\u002Fem> a filter exists. If written using a strict convention (e.g., Javadoc style, dbt doc blocks, or Sphinx-SQL), these comments can be programmatically extracted by a CI\u002FCD process to automatically populate your data catalog. This ensures the granular “why” stays perfectly synced with the execution code.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Goal:\u003C\u002Fstrong> Populates the physical tables with correct historical state and embeds the micro-reasoning alongside the logic.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003C\u002Fli>\n\u003Cli>\u003Cstrong>The Metadata Spec (Data Catalogs \u002F \u003Ccode>information_schema\u003C\u002Fcode>)\u003C\u002Fstrong>\n\u003Cul>\n\u003Cli>\u003Cstrong>Controls:\u003C\u002Fstrong> Discoverability and human understanding. It surfaces descriptions, owner assignments, and extracted relationships.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Goal:\u003C\u002Fstrong> Tells a human analyst what the table means.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003C\u002Fli>\n\u003Cli>\u003Cstrong>The Semantic Spec (Cube \u002F dbt Semantic Layer)\u003C\u002Fstrong>\n\u003Cul>\n\u003Cli>\u003Cstrong>Controls:\u003C\u002Fstrong> Runtime BI query generation. It maps exactly \u003Cem>how\u003C\u002Fem> a BI tool is allowed to aggregate or join facts to dimensions dynamically.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Goal:\u003C\u002Fstrong> Prevents analysts from generating technically correct SQL that returns mathematically wrong numbers.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>When a team tries to use a flat “Physical Spec” to govern “Semantic Rules” (e.g., feeding a \u003Ccode>schema.yml\u003C\u002Fcode> to an LLM and hoping it generates valid BI logic), they are using the wrong abstraction for the layer.\u003C\u002Fp>\n\u003Ch3>The Semantic Layer as a Service\u003C\u002Fh3>\n\u003Cp>Rather than forcing AI to rediscover business logic from raw tables, we can expose the semantic model as a structured service. Agents gain access to human-authored definitions—metrics, grain specifications, and join paths—that are unavailable in standard database catalogs.\u003C\u002Fp>\n\u003Cp>Tools like \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fdbt-labs\u002Fdbt-mcp\">dbt-mcp\u003C\u002Fa> or Cube’s REST\u002FGraphQL APIs enable this by providing structured endpoints for semantic metadata. This context allows an agent to understand that \u003Ccode>revenue\u003C\u002Fcode> requires specific aggregation rules, preventing common errors like double-counting during joins.\u003C\u002Fp>\n\u003Cp>The architecture looks like this:\u003C\u002Fp>\n\u003Cdiv class=\"mermaid\">graph TD\n    Consumer[\"AI Agent \u002F Consumer\"]\n    \n    InfraGov[\"Infrastructure Governance\"]\n    LogicGov[\"Business Logic Governance\"]\n    \n    Warehouse[\"Data Warehouse&lt;br\u002F&gt;(Physical tables\u002Fcontracts)\"]\n    \n    Consumer -- \"&lt;b&gt;Database Metadata&lt;\u002Fb&gt;&lt;br\u002F&gt;(Tables, Columns, Lineage, Tests)\" --&gt; InfraGov\n    Consumer -- \"&lt;b&gt;Semantic Model Access&lt;\u002Fb&gt;&lt;br\u002F&gt;(Metrics, Grain, Joins, Aggregation Rules)\" --&gt; LogicGov\n    \n    InfraGov --&gt; Warehouse\n    LogicGov --&gt; Warehouse\n\n    style Consumer fill:#f9f,stroke:#333,stroke-width:2px\n    style Warehouse fill:#bbf,stroke:#333,stroke-width:2px\n    style InfraGov fill:#fff,stroke:#333,stroke-dasharray: 5 5\n    style LogicGov fill:#fff,stroke:#333,stroke-width:2px\n\u003C\u002Fdiv>\u003Cp>By interacting with the semantic model, consumers leverage pre-defined logic rather than attempting to derive it from first principles. The choice of protocol (MCP, REST, GraphQL) is incidental; the value lies in the human-curated semantic asset.\u003C\u002Fp>\n\u003Ch3>Federated Semantic Contracts\u003C\u002Fh3>\n\u003Cp>While this pattern is distinct from \u003Ca href=\"https:\u002F\u002Fwww.datamesh-architecture.com\u002F\">Data Mesh\u003C\u002Fa>, it shares the core principle of \u003Cstrong>self-serve consumption\u003C\u002Fstrong>. Downstream consumers—whether AI agents, BI tools, or analysts—interact with a governed interface rather than writing arbitrary SQL against raw tables. The interface enforces domain rules without requiring consumers to manage the physical implementation.\u003C\u002Fp>\n\u003Cp>This \u003Cstrong>federated semantic contract\u003C\u002Fstrong> pattern works whether the semantic layer is centralized or domain-owned. In both cases, the contract surface is the semantic model, and the interaction happens through a structured interface.\u003C\u002Fp>\n\u003Ch2>The Verdict\u003C\u002Fh2>\n\u003Cp>Use deterministic specifications like Data Contracts to govern infrastructure and physical pipelines. Use a semantic layer to govern business logic and provide a consistent API for both humans and machines.\u003C\u002Fp>\n\u003Cp>By separating the governance of the shape of data from the governance of its meaning, you build a system that is both technically reliable and business-accurate.\u003C\u002Fp>\n\u003Cp>To see how to capture the “why” behind these human decisions and automate the review of your documentation, check out \u003Ca href=\"\u002Fblog\u002Fbi-docs\u002F\">BI Docs: Automated Documentation for Data Platforms\u003C\u002Fa>.\u003C\u002Fp>\n\u003Chr>\n\u003Cp>\u003Cem>This post is part of the series \u003Ca href=\"\u002Fblog\u002Fspec-driven-development-bi-series\u002F\">Spec-Driven Development in BI: A Pragmatic Series\u003C\u002Fa>.\u003C\u002Fem>\u003C\u002Fp>\n","Why simple specs struggle with dimensional modeling nuances like grain, SCDs, and additivity, and how to bridge the semantic gap in BI architectures.","Spec-Driven Development in BI: A Pragmatic Series",2,{"id":96,"slug":97,"title":98,"date":99,"tags":100,"image":102,"content":103,"excerpt":104,"section":20,"series":93,"series_part":105,"lang":22},1831,"spec-driven-bi-part1-when-it-works","Spec-Driven BI: When It Works (And When It Doesn't)","2026-03-11T00:00:00.000Z",[10,88,101,13],"Spec-Driven Development","\u002Fimg\u002Fblog-spec-driven-bi-part1.png","\u003Ch2>Introduction\u003C\u002Fh2>\n\u003Cp>At its core, \u003Cstrong>spec-driven development\u003C\u002Fstrong> in Business Intelligence is a simple idea: instead of writing hundreds of SQL files and test scripts by hand, you write a configuration file (a specification) and let automation generate the code for you. Write the spec once, generate the pipelines everywhere.\u003C\u002Fp>\n\u003Cp>When executed well, it makes data pipelines faster to build, more maintainable, and remarkably consistent.\u003C\u002Fp>\n\u003Cp>But here is the reality: \u003Cstrong>specs work brilliantly for some parts of your analytics stack and create massive friction for others.\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>The key insight—and the one that dictates whether automation helps or hurts—is understanding the difference between \u003Cstrong>infrastructure\u003C\u002Fstrong> and \u003Cstrong>exploration\u003C\u002Fstrong>.\u003C\u002Fp>\n\u003Chr>\n\u003Ch2>The Essential Distinction: Infrastructure vs. Exploratory\u003C\u002Fh2>\n\u003Cp>Before anything else, you need to answer this question about any given piece of analytics work:\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Is this infrastructure (stable, widely-reused, high-impact) or exploratory (fast iteration, low risk, transient)?\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cp>Your answer determines everything.\u003C\u002Fp>\n\u003Ch3>Infrastructure Characteristics\u003C\u002Fh3>\n\u003Cul>\n\u003Cli>Changes \u003Cstrong>quarterly or less\u003C\u002Fstrong>\u003C\u002Fli>\n\u003Cli>Used by \u003Cstrong>many downstream dashboards or teams\u003C\u002Fstrong>\u003C\u002Fli>\n\u003Cli>\u003Cstrong>High impact\u003C\u002Fstrong> if broken (CFO’s scorecard goes dark, 50 people can’t do their jobs)\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Small group\u003C\u002Fstrong> maintains it (data engineers, not analysts)\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Risk of failure is critical\u003C\u002Fstrong> (breaking changes cascade across reporting)\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>\u003Cstrong>Examples:\u003C\u002Fstrong> conformed dimensions, core fact tables, shared KPI definitions.\u003C\u002Fp>\n\u003Ch3>Exploratory Characteristics\u003C\u002Fh3>\n\u003Cul>\n\u003Cli>Changes \u003Cstrong>weekly or daily\u003C\u002Fstrong>\u003C\u002Fli>\n\u003Cli>Used by \u003Cstrong>one person or team\u003C\u002Fstrong>\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Lower blast radius\u003C\u002Fstrong> if wrong (one dashboard needs a fix, easier to revert)\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Many people\u003C\u002Fstrong> touch it (analysts, business users)\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Risk of failure is deceptive\u003C\u002Fstrong> (wrong metric → wrong decision, often compounding silently for weeks before anyone notices)\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>\u003Cstrong>Examples:\u003C\u002Fstrong> one-off dashboards, quick analyses, metrics under active discussion.\u003C\u002Fp>\n\u003Ch3>The Honest Big-Tech Lesson\u003C\u002Fh3>\n\u003Cp>Large-scale data organizations \u003Cem>do\u003C\u002Fem> use spec-driven approaches. Airbnb’s \u003Ca href=\"https:\u002F\u002Fmedium.com\u002Fairbnb-engineering\u002Fhow-airbnb-achieved-metric-consistency-at-scale-f23cc53dea70\">Minerva\u003C\u002Fa> centralizes metric definitions into a single semantic layer that all dashboards consume. Uber’s \u003Ca href=\"https:\u002F\u002Fwww.uber.com\u002Fblog\u002Fumetric\u002F\">uMetric\u003C\u002Fa> enforces metric governance so teams can’t redefine “trip” differently. LinkedIn’s \u003Ca href=\"https:\u002F\u002Fengineering.linkedin.com\u002Fblog\u002F2019\u002F01\u002Fbuilding-a-large-scale-metrics-platform\">Unified Metrics Platform\u003C\u002Fa> standardizes how metrics are computed, stored, and served across products.\u003C\u002Fp>\n\u003Cp>The common pattern across all of them is \u003Cstrong>separating tiers while enforcing governed exploration\u003C\u002Fstrong>:\u003C\u002Fp>\n\u003Col>\n\u003Cli>\u003Cstrong>Infrastructure layer (specs, strict governance)\u003C\u002Fstrong> – Data engineers maintain pipelines, schemas, and conformed dimensions using deterministic specs (e.g., \u003Ccode>schema.yml\u003C\u002Fcode>, data contracts).\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Semantic layer (governed metrics)\u003C\u002Fstrong> – A unified, tool-agnostic layer (like Cube or dbt Semantic Layer) where metrics are defined explicitly.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Exploratory layer (flexible, but within guardrails)\u003C\u002Fstrong> – Analysts iterate and build one-off dashboards, but they do so \u003Cem>by querying the semantic layer\u003C\u002Fem>, not by writing rogue, ad-hoc SQL that re-invents metric definition logic.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>The spec discipline lives in the infrastructure and semantic layers. The exploratory layer stays flexible, but never bypasses the governed metrics.\u003C\u002Fp>\n\u003Ch3>Promotion Pathways: When Exploration Becomes Infrastructure\u003C\u002Fh3>\n\u003Cp>The infrastructure\u002Fexploratory split isn’t static. Today’s one-off dashboard becomes tomorrow’s CFO scorecard. You need a deliberate \u003Cstrong>promotion pathway\u003C\u002Fstrong>—a process for hardening an exploratory artifact into governed infrastructure:\u003C\u002Fp>\n\u003Col>\n\u003Cli>\u003Cstrong>Trigger:\u003C\u002Fstrong> The dashboard is referenced by more than one team, is cited in a recurring meeting, or has survived two quarterly planning cycles without being retired.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Promotion checklist:\u003C\u002Fstrong> The analyst (or sponsoring team) opens a PR that moves the logic into the governed layer: defines the metrics in the semantic layer, adds data quality tests, writes a DDR documenting the grain and business rules, and links the dashboard to the conformed dimensions.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Governance change:\u003C\u002Fstrong> Once promoted, changes to the underlying model require code review and CI validation—the same bar as any other infrastructure artifact.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>Without an explicit promotion process, you end up with shadow infrastructure: exploratory work that \u003Cem>is\u003C\u002Fem> infrastructure in practice but has none of the governance. This is how the “two dashboards, two answers” problem starts.\u003C\u002Fp>\n\u003Chr>\n\u003Ch2>Where Specs Actually Shine\u003C\u002Fh2>\n\u003Ch3>✅ Boilerplate Generation (The Staging Layer)\u003C\u002Fh3>\n\u003Cp>\u003Cstrong>The staging layer (stg_*)\u003C\u002Fstrong> is where you pull raw data from source databases and apply only mechanical transformations:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Rename columns: \u003Ccode>CUST_ID\u003C\u002Fcode> → \u003Ccode>customer_id\u003C\u002Fcode>\u003C\u002Fli>\n\u003Cli>Cast types: \u003Ccode>VARCHAR\u003C\u002Fcode> → \u003Ccode>STRING\u003C\u002Fcode>\u003C\u002Fli>\n\u003Cli>Add UTC timestamps\u003C\u002Fli>\n\u003Cli>Apply basic null handling\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>A tool or LLM can generate 50 perfect staging models in seconds by reading the source schema. No complex business logic. No grain mismatches.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Why it works:\u003C\u002Fstrong> This is syntax, not semantics.\u003C\u002Fp>\n\u003Ch3>✅ Metric Definition (The Semantic Layer)\u003C\u002Fh3>\n\u003Cp>You cannot write a universal spec that generates Snowflake SQL, Power BI DAX, and LookML simultaneously—those paradigms are incompatible. But you \u003Cem>can\u003C\u002Fem> create a single, version-controlled metric definition that multiple tools consume.\u003C\u002Fp>\n\u003Cp>Here is what this looks like using \u003Cstrong>dbt Semantic Layer \u002F MetricFlow\u003C\u002Fstrong>—not a homegrown schema, but the actual syntax a mature tool uses:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-yaml\"># dbt Semantic Layer — MetricFlow metric definition\nsemantic_models:\n  - name: transactions\n    defaults:\n      agg_time_dimension: transaction_date\n    entities:\n      - name: transaction\n        type: primary\n        expr: transaction_id\n      - name: customer\n        type: foreign\n        expr: customer_id\n    dimensions:\n      - name: transaction_date\n        type: time\n        type_params:\n          time_granularity: day\n    measures:\n      - name: transaction_amount\n        agg: sum\n        expr: amount_usd\n\nmetrics:\n  - name: daily_revenue\n    description: Sum of settled transactions, per day, excluding internal accounts\n    type: simple\n    type_params:\n      measure: transaction_amount\n    filter: |\n      {{ Dimension('transaction__account_type') }} != 'internal'\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>MetricFlow renders this into the correct SQL dialect for your warehouse. Different BI tools consume it. The key difference from a generic physical spec: MetricFlow’s schema enforces entities, grain, and aggregation type—it won’t let you build a metric that silently double-counts.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Why it works:\u003C\u002Fstrong> You prevent silent metric drift. You don’t necessarily eliminate the “Finance says $10k, Sales says $12k” discrepancy—often, those are legitimately different metrics from different bounded contexts (e.g., booked vs. recognized revenue). But a semantic spec forces them to have distinct, governed definitions (\u003Ccode>sales_bookings\u003C\u002Fcode> vs \u003Ccode>finance_recognized_revenue\u003C\u002Fcode>) rather than colliding as ambiguous \u003Ccode>revenue\u003C\u002Fcode> columns.\u003C\u002Fp>\n\u003Ch3>✅ Test Automation (Data Contracts)\u003C\u002Fh3>\n\u003Cp>If a stakeholder says “Transaction ID must be unique,” a script can automatically generate the data quality test:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-yaml\">constraints:\n  - column: transaction_id\n    test: unique\n    critical: true\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>You’re not writing complex logic—you’re translating explicit data-quality rules into deterministic test code.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Why it works:\u003C\u002Fstrong> Automation for explicit contracts, not implicit semantics.\u003C\u002Fp>\n\u003Ch3>✅ Preventing Scope Creep: Infrastructure Specs\u003C\u002Fh3>\n\u003Cp>Apply specs ruthlessly to:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Conformed dimensions\u003C\u002Fstrong> (dim_product, dim_date, dim_customer) – change quarterly\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Core fact tables\u003C\u002Fstrong> (fct_transactions, fct_events) – spine of the warehouse\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Shared KPI definitions\u003C\u002Fstrong> (revenue, churn, LTV) – 50+ downstream dependencies\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>\u003Cstrong>Don’t\u003C\u002Fstrong> use rigid infrastructure specs for:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>One-off dashboards\u003C\u002Fli>\n\u003Cli>Exploratory analysis (as long as it queries the semantic layer)\u003C\u002Fli>\n\u003Cli>Quick iterations while stakeholders pivot\u003C\u002Fli>\n\u003Cli>Local analyst work\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Chr>\n\u003Ch2>The Golden Rule\u003C\u002Fh2>\n\u003Cp>\u003Cstrong>Use automation for syntax, not for semantics.\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Ch3>Automate:\u003C\u002Fh3>\n\u003Cul>\n\u003Cli>Boilerplate (stg_ models, naming conventions, casts)\u003C\u002Fli>\n\u003Cli>Test generation (from explicit constraints)\u003C\u002Fli>\n\u003Cli>Metric definitions (version-controlled, tool-agnostic)\u003C\u002Fli>\n\u003Cli>Staging scaffolding (from source schemas)\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch3>Do NOT automate:\u003C\u002Fh3>\n\u003Cul>\n\u003Cli>Grain decisions (what’s a “daily revenue” row?)\u003C\u002Fli>\n\u003Cli>SCD logic (product category changes)\u003C\u002Fli>\n\u003Cli>Dimensional modeling (conformed dimensions)\u003C\u002Fli>\n\u003Cli>Complex business rules (how to calculate adjusted revenue)\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>Those require conversation. Code generation can draft them; \u003Cstrong>human review is mandatory\u003C\u002Fstrong>.\u003C\u002Fp>\n\u003Chr>\n\u003Ch2>Where This Sits in the Ecosystem\u003C\u002Fh2>\n\u003Cp>If you’re reading this in 2026, you should know: \u003Cstrong>the data contracts movement already solves many of these problems with deterministic tooling.\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Tool \u002F Pattern\u003C\u002Fth>\n\u003Cth>What It Does\u003C\u002Fth>\n\u003Cth>How It Compares\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>\u003Cstrong>Native Data Contracts\u003C\u002Fstrong> (e.g., \u003Ccode>schema.yml\u003C\u002Fcode> constraints)\u003C\u002Ftd>\n\u003Ctd>Enforce column types, not-null, uniqueness at build time\u003C\u002Ftd>\n\u003Ctd>Deterministic enforcement — no LLM needed. If you already use contracts natively, you may not need YAML specs for schema governance at all.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Cstrong>Great Expectations \u002F Soda\u003C\u002Fstrong>\u003C\u002Ftd>\n\u003Ctd>Data quality assertions on pipeline output\u003C\u002Ftd>\n\u003Ctd>Catches semantic issues (row counts, value ranges, distributions) that specs can’t express.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Cstrong>dbt Semantic Layer \u002F MetricFlow\u003C\u002Fstrong>\u003C\u002Ftd>\n\u003Ctd>Centralized metric definitions consumed by multiple BI tools\u003C\u002Ftd>\n\u003Ctd>The mature version of the “metric definition” pattern described above. If you’re on dbt Cloud, evaluate this before building your own.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Cstrong>Monte Carlo \u002F data observability\u003C\u002Fstrong>\u003C\u002Ftd>\n\u003Ctd>Anomaly detection, freshness SLAs, lineage tracking\u003C\u002Ftd>\n\u003Ctd>Solves drift detection without AI review agents.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003Ch3>Where spec-driven development fits\u003C\u002Fh3>\n\u003Cp>Spec-driven dev is a \u003Cstrong>lightweight bridge for teams that aren’t ready for full data contract infrastructure.\u003C\u002Fstrong> If your org:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Doesn’t use an orchestration and testing framework natively yet\u003C\u002Fli>\n\u003Cli>Has no schema registry or contract framework\u003C\u002Fli>\n\u003Cli>Needs a low-friction starting point\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>If you are just modernizing a legacy platform, then writing infrastructure-level specs and generating staging models from them is a pragmatic first step. But understand the ceiling: \u003Cstrong>as your maturity grows, you’ll likely replace custom specs with native data contracts and a semantic layer tool.\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Ch3>Scope honesty\u003C\u002Fh3>\n\u003Cp>This approach is \u003Cstrong>SQL-ecosystem specific.\u003C\u002Fstrong> The physical specifications described here generate SQL. They don’t generate DAX, LookML, or anything outside the SQL paradigm. If your BI stack spans multiple paradigms, specs help within each one but don’t bridge them.\u003C\u002Fp>\n\u003Chr>\n\u003Ch2>Orchestrating Spec-Driven Code Generation: Determinism is Mandatory\u003C\u002Fh2>\n\u003Cp>If you want to automate the workflow (spec → code → orchestration pipeline), you must rely on \u003Cstrong>deterministic AST parsers and templating engines\u003C\u002Fstrong>, not probabilistic LLMs.\u003C\u002Fp>\n\u003Cp>Using an LLM to hallucinate SQL for core infrastructure based on a physical specification is an architectural anti-pattern. It injects non-determinism, fragility, and silent semantic failures into a tier that requires absolute predictability.\u003C\u002Fp>\n\u003Cp>Instead, use tools designed for safe, programmatic scaffolding:\u003C\u002Fp>\n\u003Col>\n\u003Cli>\u003Cstrong>Jinja Macros:\u003C\u002Fstrong> Many tools allow you to write macros that loop through a graph object and dynamically generate SQL for staging layers.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Code generators:\u003C\u002Fstrong> Use established packages or scripts to generate baseline models deterministically from source schemas.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Alternative dialects (e.g., PRQL):\u003C\u002Fstrong> If SQL is too verbose for metric definitions, compile simpler declarative languages into SQL deterministically.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>Here is a simple example of a Python script using Jinja2 to generate a staging model from a spec:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-python\"># Minimal example of deterministic code generation (No LLMs)\nimport yaml\nfrom jinja2 import Template\n\ndef generate_staging_model(spec_file: str):\n    with open(spec_file, 'r') as f:\n        spec = yaml.safe_load(f)\n\n    # A strict Jinja template guarantees output structure\n    sql_template = Template(&quot;&quot;&quot;\n    WITH source AS (\n        SELECT * FROM {{ &quot;{{ source('&quot; + source_name + &quot;', '&quot; + table_name + &quot;') }}&quot; }}\n    ),\n    renamed AS (\n        SELECT\n        {%- for col in columns %}\n            cast({{ col.name }} as {{ col.cast_type }}) as {{ col.rename | default(col.name) }}{% if not loop.last %},{% endif %}\n        {%- endfor %}\n        FROM source\n    )\n    SELECT * FROM renamed\n    &quot;&quot;&quot;)\n\n    return sql_template.render(\n        source_name=spec['source']['database'],\n        table_name=spec['source']['table'],\n        columns=spec['columns']\n    )\n\n# The output is 100% predictable: same spec = same SQL, every time.\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>The workflow is straightforward and safe:\u003C\u002Fp>\n\u003Col>\n\u003Cli>Define your spec (YAML, JSON, or any structured format)\u003C\u002Fli>\n\u003Cli>Pass it through a deterministic templating engine\u003C\u002Fli>\n\u003Cli>Write the generated code to your \u003Ccode>models\u002F\u003C\u002Fcode> folder\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>\u003Cstrong>This works well for mechanical scaffolding.\u003C\u002Fstrong> It shouldn’t be used for complex logic involving grain, SCDs, or conformed dimensions—topics we’ll cover in Part 2.\u003C\u002Fp>\n\u003Chr>\n\u003Ch2>Optional: Architectural Guidance Files\u003C\u002Fh2>\n\u003Cp>If you want to formalize architectural standards that your LLM or code generator should follow, you can create a lightweight guidance file (e.g., \u003Ccode>skills.md\u003C\u002Fcode>). This is \u003Cstrong>not required\u003C\u002Fstrong>, but it helps enforce consistency:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-markdown\"># Architectural Standards for Data Models\n\n## Naming Conventions\n- Staging models: `stg_` prefix\n- Facts: `fct_` prefix \n- Dimensions: `dim_` prefix\n- All files: snake_case\n\n## Column Standards\n- Primary keys: `{model_name}_id` (e.g., `customer_id`, `transaction_id`)\n- Created\u002Fupdated timestamps: `created_at`, `updated_at` (UTC)\n- Boolean flags: `is_` prefix (e.g., `is_active`, `is_deleted`)\n- Amounts: always `decimal(18,2)` or higher precision\n\n## Testing Rules\n- All primary keys: `not_null` + `unique` tests\n- All foreign keys: `relationships` tests\n- All fact amounts: `not_null` (no NULL amounts)\n- Fact tables: at least one test on the grain (e.g., deduplication check)\n\n## Documentation\n- Every model must have a description block\n- Every column must have a description block\n- Fact tables must document their grain explicitly\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch3>Important Caveat\u003C\u002Fh3>\n\u003Cp>This guidance file helps when humans review generated code, but \u003Cstrong>it doesn’t enforce anything automatically\u003C\u002Fstrong>. An LLM will sometimes ignore it. Your real enforcement layer is:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>CI linting\u003C\u002Fstrong> (fail the build if \u003Ccode>fct_\u003C\u002Fcode> tables don’t match the naming pattern)\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Data quality tests\u003C\u002Fstrong> (mandatory tests that fail if violated)\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Code review\u003C\u002Fstrong> (humans check that generated code follows the standards)\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>The guidance file is a reference, not a guarantee.\u003C\u002Fp>\n\u003Chr>\n\u003Ch2>What Goes Wrong\u003C\u002Fh2>\n\u003Cp>The post assumes:\u003C\u002Fp>\n\u003Cblockquote>\n\u003Cp>“Specs will make everything faster and more consistent.”\u003C\u002Fp>\n\u003C\u002Fblockquote>\n\u003Cp>But when you apply specs universally, you get:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>Year 1:\u003C\u002Fstrong> Great! Specs for critical KPIs. Adoption and enthusiasm.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Year 2:\u003C\u002Fstrong> Marketing wants a one-off Customer Lifetime Value dashboard. Do you mandate a spec PR?\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Year 3:\u003C\u002Fstrong> Half your dashboards have specs, half don’t. Specs feel like bureaucracy for the explorer-analysts because the platform’s Developer Experience (DX) is poor.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Year 4:\u003C\u002Fstrong> Nobody remembers why specs exist. Adoption dies. You’re back to hand-written SQL with extra configuration files gathering dust.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>The drift happens because you created a false dichotomy: rigorous infrastructure on one side, and unsupported ad-hoc SQL on the other. Good data engineering provides tools (like a robust semantic layer) that make doing the right thing \u003Cem>easier\u003C\u002Fem> than doing the wrong thing. Exploration shouldn’t mean abandoning guardrails; it should mean exploring \u003Cem>through\u003C\u002Fem> the semantic layer.\u003C\u002Fp>\n\u003Chr>\n\u003Ch2>The Framework for Decision-Making\u003C\u002Fh2>\n\u003Cp>Before you commit a BI artifact to spec-driven development, ask:\u003C\u002Fp>\n\u003Col>\n\u003Cli>\n\u003Cp>\u003Cstrong>Is this infrastructure or exploration?\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Infrastructure → specs are worth the overhead\u003C\u002Fli>\n\u003Cli>Exploratory → let it stay flexible\u003C\u002Fli>\n\u003C\u002Ful>\n\u003C\u002Fli>\n\u003Cli>\n\u003Cp>\u003Cstrong>Will this change frequently?\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Rarely → specs pay off (you reuse them)\u003C\u002Fli>\n\u003Cli>Frequently → specs create friction\u003C\u002Fli>\n\u003C\u002Ful>\n\u003C\u002Fli>\n\u003Cli>\n\u003Cp>\u003Cstrong>How many teams\u002Fdashboards depend on this?\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Many → specs reduce coordination overhead\u003C\u002Fli>\n\u003Cli>One or two → not worth the complexity\u003C\u002Fli>\n\u003C\u002Ful>\n\u003C\u002Fli>\n\u003Cli>\n\u003Cp>\u003Cstrong>Is this a rule, or still being negotiated?\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Settled rule → spec it\u003C\u002Fli>\n\u003Cli>Still being figured out → explore via the semantic layer’s flexible interfaces, hardening into a spec only when the logic stabilizes\u003C\u002Fli>\n\u003C\u002Ful>\n\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Chr>\n\u003Ch2>Next Steps\u003C\u002Fh2>\n\u003Cp>If you’re starting small, this is the roadmap:\u003C\u002Fp>\n\u003Col>\n\u003Cli>\u003Cstrong>Identify your infrastructure layer\u003C\u002Fstrong> – which tables\u002Fmetrics are stable, widely-used, and rarely change?\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Apply specs to the staging layer\u003C\u002Fstrong> – this is the easiest win and requires almost no semantic modeling.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Define shared KPIs as specs\u003C\u002Fstrong> – this prevents the “two dashboards, two answers” problem. This is the second-highest ROI.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Empower analyst work with semantic layers\u003C\u002Fstrong> – Don’t mandate rigid infrastructure specs for exploratory analysis, but give them a semantic layer that makes exploring governed metrics easier than writing ad-hoc SQL.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Expose the semantic layer to AI agents\u003C\u002Fstrong> – Once your semantic layer is governing your metrics, AI agents can query it directly through structured interfaces like MCP, REST APIs, or GraphQL endpoints (e.g., \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fdbt-labs\u002Fdbt-mcp\">dbt-mcp\u003C\u002Fa>, \u003Ca href=\"https:\u002F\u002Fcube.dev\u002F\">Cube’s REST API\u003C\u002Fa>). The agents generate staging models, trace lineage, and run tests—all while respecting the grain and business rules you defined. The protocol is plumbing; the semantic contract is what matters.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Chr>\n\u003Cp>\u003Cstrong>Next: \u003Ca href=\"\u002Fblog\u002Fspec-driven-bi-part2-semantic-gap\u002F\">Part 2: The Semantic Gap: Specifying Business Reality\u003C\u002Fa>\u003C\u002Fstrong>\u003C\u002Fp>\n\u003Chr>\n\u003Cp>\u003Cem>This post is part of the series \u003Ca href=\"\u002Fblog\u002Fspec-driven-development-bi-series\u002F\">Spec-Driven Development in BI: A Pragmatic Series\u003C\u002Fa>.\u003C\u002Fem>\u003C\u002Fp>\n","A practical framework for deciding which parts of your BI stack should use specs—and which should stay flexible.",1,{"id":107,"slug":108,"title":109,"date":110,"tags":111,"image":115,"content":116,"excerpt":117,"section":20,"series":21,"series_part":21,"lang":22},1821,"bi-docs","BI Docs: Automated Documentation for Data Platforms","2026-02-23T00:00:00.000Z",[10,112,11,113,29,114],"Documentation","Power BI","MkDocs","\u002Fimg\u002Fblog-bi-docs-header.png","\u003Cp>Business intelligence environments have a documentation problem, and it’s not the kind that a wiki solves.\u003C\u002Fp>\n\u003Cp>Over the years, I’ve watched critical knowledge accumulate in the worst possible places: email threads, meeting transcripts, Slack messages, and in the heads of people who eventually leave. Thresholds that determine how revenue gets bucketed. Cutoff dates baked into SQL queries. Business rules that exist because someone in Finance asked for them in 2022 and nobody wrote it down.\u003C\u002Fp>\n\u003Cp>When I decided to leave my last role, I didn’t want to be that person who walks out with half the platform’s context in their head. So I built a system to capture it, organize it, and keep it honest after I was gone.\u003C\u002Fp>\n\u003Cp>I call it \u003Cstrong>BI Docs\u003C\u002Fstrong>. It’s not a product. It’s a pattern: a way to build and maintain up-to-date documentation for data platforms using MkDocs and a lightweight AI review layer. The stack I used was Databricks and Power BI, but the approach is portable to any combination of ETL and reporting tools.\u003C\u002Fp>\n\u003Ch2>Phase 0: Building the Baseline\u003C\u002Fh2>\n\u003Cp>The hardest part of any documentation effort isn’t the tooling. It’s the initial excavation.\u003C\u002Fp>\n\u003Cp>When I started, the “documentation” for our data platform was scattered across hundreds of emails, dozens of Jira tickets, a few architecture powerpoints that were two years out of date, and a SharePoint site that nobody visited. The first job was to gather all of that into a single place and make sense of it.\u003C\u002Fp>\n\u003Cp>This is where AI proved genuinely useful. Not as a writer, but as a \u003Cstrong>librarian\u003C\u002Fstrong>. I used \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fmarkitdown\">markitdown\u003C\u002Fa> to convert documents from all the usual formats (Word, PDF, PowerPoint, HTML emails) into clean Markdown that an LLM could parse. The workflow looked like this:\u003C\u002Fp>\n\u003Col>\n\u003Cli>\u003Cstrong>Collect everything.\u003C\u002Fstrong> Emails from stakeholders containing requirements. Jira tickets describing feature requests and bug fixes. Architecture diagrams. Meeting notes. SharePoint pages. Database schemas.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Convert to Markdown.\u003C\u002Fstrong> Using markitdown, I transformed these documents into a uniform format. This is the critical step. LLMs work best when they can read plain text, and Markdown is a practical intermediate format that both humans and machines handle well.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Feed and filter.\u003C\u002Fstrong> I used GitHub Copilot to help me extract the actual facts from the converted documents: business rules, column definitions, threshold values, timeline decisions. The AI accelerated the extraction, but I reviewed and curated every single page. There is no shortcut here. AI will confidently present irrelevant or wrong information next to useful facts, and only someone who understands the domain can tell them apart.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>The output of Phase 0 was a structured MkDocs repository containing everything the team needed to understand the platform. The AI didn’t write the documentation. \u003Cstrong>I\u003C\u002Fstrong> wrote it, using AI as a high-speed research assistant.\u003C\u002Fp>\n\u003Ch2>The Ground Truth: Data Decision Records (DDRs)\u003C\u002Fh2>\n\u003Cp>Before we jump into the tooling, we have to talk about format. You cannot just dump a 30-page requirements document into a repository and expect an automated system (or a human) to parse it effectively when code changes.\u003C\u002Fp>\n\u003Cp>The documentation needs to be specific, isolated, and structured. To achieve this, I used \u003Cstrong>Data Decision Records (DDRs)\u003C\u002Fstrong>.\u003C\u002Fp>\n\u003Cp>A DDR is a lightweight markdown file committed to Git that captures \u003Cstrong>one specific architectural or business logic decision\u003C\u002Fstrong>. It’s an adaptation of Michael Nygard’s \u003Ca href=\"https:\u002F\u002Fadr.github.io\u002F\">Architecture Decision Records (ADRs)\u003C\u002Fa>, but tailored for data platforms.\u003C\u002Fp>\n\u003Cp>When a junior analyst looks at a strangely complex \u003Ccode>UNION ALL\u003C\u002Fcode> in a fact table and thinks, \u003Cem>“I can optimize this,”\u003C\u002Fem> the DDR is the single file that explains exactly \u003Cem>why\u003C\u002Fem> that join exists (e.g., a recurrent accounting restatement process mandated by Finance).\u003C\u002Fp>\n\u003Cp>A DDR has three core sections:\u003C\u002Fp>\n\u003Col>\n\u003Cli>\u003Cstrong>Context:\u003C\u002Fstrong> The problem and the environment (e.g., “Sales targets arrive on the 5th, but Finance closes on the 3rd.”)\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Decision:\u003C\u002Fstrong> The exact logic implemented to solve it.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Consequences:\u003C\u002Fstrong> The known downstream impacts of this decision.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>By forcing documentation into this highly granular DDR format, the AI review layer (which we’ll cover shortly) has a much easier time mapping a specific Git diff to a specific business rule.\u003C\u002Fp>\n\u003Cp>In practice, I recommend storing DDRs under \u003Ccode>docs\u002Frequirements\u002Fddrs\u002F\u003C\u002Fcode> with a filename convention like \u003Ccode>20260223-fiscal-year-cutoff.md\u003C\u002Fcode>. That keeps them easy to sort chronologically while still being specific enough to reference from reviews.\u003C\u002Fp>\n\u003Cp>Here’s a minimal DDR template:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-markdown\"># DDR-20260223: Fiscal Year Cutoff Logic\n\n## Context\nFinance closes the month on the 3rd, but sales targets arrive on the 5th.\n\n## Decision\nUse the previous month's targets until the current month's targets are loaded.\n\n## Consequences\nDashboards show a two-day lag at the start of each month.\nAny change to this timing must be reflected in the Services dashboard docs.\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch2>Getting Started with MkDocs\u003C\u002Fh2>\n\u003Cp>If you haven’t used \u003Ca href=\"https:\u002F\u002Fwww.mkdocs.org\u002F\">MkDocs\u003C\u002Fa> before, it’s a static site generator designed specifically for project documentation. You write Markdown files, organize them in folders, and MkDocs builds a clean, searchable website. Combined with the \u003Ca href=\"https:\u002F\u002Fsquidfunk.github.io\u002Fmkdocs-material\u002F\">Material for MkDocs\u003C\u002Fa> theme, the result looks professional with almost zero effort.\u003C\u002Fp>\n\u003Ch3>Installation\u003C\u002Fh3>\n\u003Cp>MkDocs runs on Python. To set up a new project:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-bash\"># Install MkDocs and the Material theme\npip install mkdocs mkdocs-material\n\n# Create a new documentation project\nmkdocs new bi-docs\ncd bi-docs\n\n# Start the local dev server\nmkdocs serve\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>This gives you a live preview at \u003Ccode>http:\u002F\u002F127.0.0.1:8000\u003C\u002Fcode> that updates every time you save a file.\u003C\u002Fp>\n\u003Ch3>Configuration\u003C\u002Fh3>\n\u003Cp>The \u003Ccode>mkdocs.yml\u003C\u002Fcode> file at the root of your project controls everything. Here’s a starting point:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-yaml\">site_name: BI Platform Documentation\ntheme:\n  name: material\n  palette:\n    scheme: default\n    primary: indigo\n    accent: teal\n  features:\n    - navigation.sections\n    - navigation.expand\n    - search.suggest\n    - search.highlight\n    - content.code.copy\n\nplugins:\n  - search\n\nmarkdown_extensions:\n  - admonitions\n  - pymdownx.details\n  - pymdownx.superfences\n  - pymdownx.tabbed:\n      alternate_style: true\n  - tables\n\nnav:\n  - Home: index.md\n  - Data Sources: data_sources\u002Findex.md\n  - Data Models: data_models\u002Findex.md\n  - Requirements: requirements\u002Findex.md\n  - Timelines: timelines\u002Findex.md\n  - Glossary: glossary\u002Findex.md\n  - Dashboards: dashboards\u002Findex.md\n  - Architecture: architecture\u002Findex.md\n  - Review: review\u002Findex.md\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch3>Building for Production\u003C\u002Fh3>\n\u003Cp>When you’re ready to publish:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-bash\"># Build the static site\nmkdocs build\n\n# The output goes to the \u002Fsite directory\n# Deploy to GitHub Pages, Azure Static Web Apps, or any static host\nmkdocs gh-deploy  # shortcut for GitHub Pages\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch2>The Repository Structure\u003C\u002Fh2>\n\u003Cp>The folder hierarchy should reflect two things: the logical structure of your data platform and the team structure of your organization. Different sections will have different audiences. A business analyst cares about glossaries and dashboard definitions; a project manager cares about requirements and timelines.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>Crucially, do not duplicate your column-level definitions.\u003C\u002Fstrong> Modern data engineering dictates that documentation lives close to the code (e.g., in table comments, a schema registry, or a data catalog). BI Docs and MkDocs should be reserved for high-level concepts: business requirements, Data Decision Records (DDRs), architecture standards, and timelines. Do not create a \u003Ccode>fact_sales.md\u003C\u002Fcode> in MkDocs that just lists columns, as you guarantee that documentation will rot.\u003C\u002Fp>\n\u003Cp>Here’s the structure I settled on:\u003C\u002Fp>\n\u003Cpre>\u003Ccode>bi-docs\u002F\n├── mkdocs.yml\n└── docs\u002F\n    ├── index.md                    # Landing page with platform overview\n    │\n    ├── data_sources\u002F               # Where the data comes from\n    │   ├── index.md                # Summary of all source systems\n    │   ├── crm_system.md           # CRM ingestion details\n    │   ├── erp_financials.md       # ERP feed documentation\n    │   └── third_party_api.md      # External API contracts\n    │\n    ├── data_models\u002F                # Architectural modeling decisions (NOT column specs)\n    │   ├── index.md                # Modeling conventions and standards\n    │   └── architecture_notes.md   # Broad architectural decisions\n    │\n    ├── requirements\u002F               # Why the data looks the way it does\n    │   ├── index.md                # How to read and contribute requirements\n    │   ├── req_fiscal_year.md      # Fiscal year cutoff logic and its history\n    │   └── req_product_flags.md    # Product categorization rules\n    │\n    ├── timelines\u002F                  # When decisions were made\n    │   ├── index.md                # How the timeline works\n    │   └── 2024_decisions.md       # Chronological log of platform decisions\n    │\n    ├── glossary\u002F                   # What the terms mean\n    │   ├── index.md                # Glossary conventions\n    │   └── business_terms.md       # Revenue, Net Sales, Active Customer, etc.\n    │\n    ├── dashboards\u002F                 # What the business sees\n    │   ├── index.md                # Dashboard inventory\n    │   ├── executive_dashboard.md  # Measures, filters, upstream dependencies\n    │   └── regional_sales.md       # Regional breakdown report\n    │\n    ├── architecture\u002F               # How the platform is built\n    │   ├── index.md                # Architecture overview\n    │   ├── etl_patterns.md         # Standard ETL patterns (incremental, full)\n    │   └── platform_norms.md       # Naming conventions, branching strategy\n    │\n    └── review\u002F                     # Where the AI review system lives\n        ├── index.md                # How the review process works\n        ├── meta.md                 # The orchestration prompt\n        ├── architect.md            # The &quot;Architect&quot; persona prompt\n        ├── reviewer.md             # The &quot;Reviewer&quot; persona prompt\n        └── runs\u002F                   # Review run history\n            └── 2026-02-23T10-00\u002F\n                └── review.md       # Findings from this review run\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch3>A Note on Audiences\u003C\u002Fh3>\n\u003Cp>Not everyone needs to read everything. When I structured the docs, I kept this mapping in mind:\u003C\u002Fp>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Section\u003C\u002Fth>\n\u003Cth>Primary Audience\u003C\u002Fth>\n\u003Cth>Purpose\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Data Sources\u003C\u002Ftd>\n\u003Ctd>Data Engineers\u003C\u002Ftd>\n\u003Ctd>Understand ingestion contracts\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Data Models\u003C\u002Ftd>\n\u003Ctd>Data Engineers, Architects\u003C\u002Ftd>\n\u003Ctd>High-level patterns (Column details belong in the data catalog)\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Requirements\u003C\u002Ftd>\n\u003Ctd>Everyone\u003C\u002Ftd>\n\u003Ctd>Know \u003Cem>why\u003C\u002Fem> things are built the way they are\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Timelines\u003C\u002Ftd>\n\u003Ctd>Architects, Managers\u003C\u002Ftd>\n\u003Ctd>Track decision history and accountability\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Glossary\u003C\u002Ftd>\n\u003Ctd>Analysts, Business Users\u003C\u002Ftd>\n\u003Ctd>Align on term definitions\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Dashboards\u003C\u002Ftd>\n\u003Ctd>Analysts, Stakeholders\u003C\u002Ftd>\n\u003Ctd>Map reports to upstream models\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Architecture\u003C\u002Ftd>\n\u003Ctd>Engineers, Architects\u003C\u002Ftd>\n\u003Ctd>Platform standards and conventions\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Review\u003C\u002Ftd>\n\u003Ctd>Engineers\u003C\u002Ftd>\n\u003Ctd>AI-assisted review process\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003Ch2>The Automated Layer: Keeping Docs Honest\u003C\u002Fh2>\n\u003Cp>Phase 0 gets you a comprehensive documentation baseline. That’s valuable on its own, but documentation rots. The moment someone changes a column name in Databricks or updates a DAX measure in Power BI without touching the docs, the baseline starts to drift.\u003C\u002Fp>\n\u003Cp>The automated layer is designed to catch that drift. The idea is straightforward: when someone makes a change to the ETL code (Databricks notebooks, SQL files, PySpark scripts) or the reporting layer (Power BI semantic model files via Git integration), the system compares the change against the existing documentation and flags mismatches.\u003C\u002Fp>\n\u003Cp>It doesn’t fix the code. It doesn’t rewrite the docs autonomously. It raises a flag and says: \u003Cem>“This change affects documented behavior. Here’s what you should review.”\u003C\u002Fem> The developer always makes the final call.\u003C\u002Fp>\n\u003Ch2>The Architecture\u003C\u002Fh2>\n\u003Cp>Here’s the high-level flow:\u003C\u002Fp>\n\u003Cp>\u003Cimg src=\"\u002Fimg\u002Fbi-docs-architecture.png\" alt=\"BI Docs Architecture: a context builder gathers code, docs, and a trigger, then passes them to an architect prompt followed by a reviewer prompt, which writes a timestamped review report.\">\u003C\u002Fp>\n\u003Cp>The diagram shows a simple sequence: trigger, context assembly, architect pass, reviewer pass, and a final timestamped report. If you present this pattern internally, keep that caption with the image so the flow is still understandable in screen readers, exports, and chat previews.\u003C\u002Fp>\n\u003Cp>The system supports two entry points. \u003Cstrong>Git Diff Mode\u003C\u002Fstrong> computes the diff between the current branch and main, reacting to code changes. \u003Cstrong>Priority Mode\u003C\u002Fstrong> takes a plain-language focus area (e.g., “validate docs against the Services dashboard”) and proactively audits a specific part of the platform.\u003C\u002Fp>\n\u003Cp>Regardless of the entry point, the pipeline is the same:\u003C\u002Fp>\n\u003Col>\n\u003Cli>The \u003Cstrong>Context Builder\u003C\u002Fstrong> assembles a context bundle: the trigger (diff or priority text), the full source code from the target repos (Databricks, Power BI), and the relevant documentation from the MkDocs repository (data models, requirements, dashboards).\u003C\u002Fli>\n\u003Cli>The \u003Cstrong>Architect\u003C\u002Fstrong> (\u003Ccode>architect.md\u003C\u002Fcode>) receives the full context bundle. It scopes the analysis by identifying which code files and which documentation pages are affected, and provides impact cues. Its output is appended to the context bundle.\u003C\u002Fli>\n\u003Cli>The \u003Cstrong>Reviewer\u003C\u002Fstrong> (\u003Ccode>reviewer.md\u003C\u002Fcode>) receives the enriched context bundle—everything the architect identified, plus the original code and docs. It evaluates the findings, assigns severity, cites evidence, and writes suggested actions.\u003C\u002Fli>\n\u003Cli>The final output is written to a timestamped \u003Ccode>review.md\u003C\u002Fcode> under \u003Ccode>.review\u002Fruns\u002F\u003C\u002Fcode>.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Cp>There are multiple ways to trigger this: Git hooks running locally, CI\u002FCD pipeline steps on pull requests, or scheduled jobs. For documentation reviews, I’d recommend keeping a human in the loop—have the system generate the review, but let the engineer decide what to act on. Automated gates that block merges based on AI findings introduce friction that teams tend to disable after the first false positive.\u003C\u002Fp>\n\u003Cp>If you want a lightweight CI starting point, a pull request workflow can be as simple as this:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-yaml\">name: bi-docs-review\n\non:\n  pull_request:\n\njobs:\n  review:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions\u002Fcheckout@v4\n        with:\n          fetch-depth: 0\n\n      - name: Generate BI Docs review\n        run: |\n          python scripts\u002Frun_bi_docs_review.py \\\n            --mode git \\\n            --base main \\\n            --output .review\u002Fruns\u002F${{ github.event.pull_request.updated_at }}\u002Freview.md\n\n      - name: Upload review artifact\n        uses: actions\u002Fupload-artifact@v4\n        with:\n          name: bi-docs-review\n          path: .review\u002Fruns\u002F\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>The exact runner script will depend on your setup, but the shape should stay the same: compute the diff, build the context, run the two prompts, and persist the review as an artifact instead of silently discarding it.\u003C\u002Fp>\n\u003Ch2>The Meta-Prompt: A Two-Agent Review System\u003C\u002Fh2>\n\u003Cp>Rather than a single monolithic prompt, the review system uses two separate agent personas that run sequentially. One identifies the affected area, the other evaluates the impact.\u003C\u002Fp>\n\u003Ch3>Setting Up the \u003Ccode>.review\u002F\u003C\u002Fcode> Folder\u003C\u002Fh3>\n\u003Cp>The review system lives inside your BI Docs repository:\u003C\u002Fp>\n\u003Cpre>\u003Ccode>.review\u002F\n├── meta.md          # Orchestration: which prompts to run and in what order\n├── architect.md     # Agent 1: Identifies affected documentation\n├── reviewer.md      # Agent 2: Evaluates impact and writes findings\n└── runs\u002F            # Output directory for review results\n    └── .gitkeep\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch3>\u003Ccode>meta.md\u003C\u002Fcode> — The Orchestrator\u003C\u002Fh3>\n\u003Cp>This file defines the review pipeline. It tells whatever tool you’re using (Copilot CLI, a custom script, a CI\u002FCD action) what to do:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-markdown\"># BI Docs Review Pipeline\n\n## Usage\n\nExecute this prompt with one of the following:\n\n- `execute meta.md with git diff`\n  Runs a change-driven review. The pipeline computes the diff\n  between the current branch and main, then identifies which\n  docs are affected.\n\n- `execute meta.md with priority: &lt;description&gt;`\n  Runs a target-driven review. For example:\n  `execute meta.md with priority: validate docs against the Services dashboard`\n  `execute meta.md with priority: audit the fiscal year cutoff logic`\n\n## Inputs\n- `target_repos`: List of repository paths to monitor\n- `docs_repo`: Path to the BI Docs MkDocs repository\n\n## Steps — Git Diff\n\n1. **Identify Changes**\n   - For each target repo, compute `git diff HEAD main`\n   - Filter to relevant file types: `.py`, `.sql`, `.dax`, `.bim`, `.tmdl`\n   - Execute `architect.md` with the diff as context\n\n2. **Evaluate Impact**\n   - Take the output from Step 1 (list of affected doc pages)\n   - Load the content of those doc pages from the MkDocs repo\n   - Execute `reviewer.md` with the diff + doc content\n\n3. **Write Results**\n   - Save the reviewer's output to `runs\u002F&lt;timestamp&gt;\u002Freview.md`\n   - List affected files and flag severity (info, warning, critical)\n\n## Steps — Priority\n\n1. **Scope the Priority**\n   - Execute `architect.md` with the priority text as context\n   - The architect identifies which codebase files and which\n     documentation pages are relevant to the priority,\n     and provides impact cues for the reviewer\n\n2. **Evaluate Alignment**\n   - Load the identified code files and doc pages\n   - Execute `reviewer.md` with the code + doc content\n     and the original priority as context\n\n3. **Write Results**\n   - Save the reviewer's output to `runs\u002F&lt;timestamp&gt;\u002Freview.md`\n   - Include the priority text as a header for traceability\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Cp>The two modes serve different purposes. The \u003Cstrong>git mode\u003C\u002Fstrong> is reactive—it runs after a change and catches drift. The \u003Cstrong>priority mode\u003C\u002Fstrong> is proactive—you use it when you want to validate a specific area of your platform against the documentation, for example before a stakeholder presentation or after onboarding a new team member who will own a particular dashboard.\u003C\u002Fp>\n\u003Ch3>\u003Ccode>architect.md\u003C\u002Fcode> — The Identifier\u003C\u002Fh3>\n\u003Cp>This agent adapts its behavior based on the mode. In git mode, it maps code changes to documentation. In priority mode, it scopes the review by identifying which parts of the codebase and which docs are relevant to the stated priority.\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-markdown\"># Role: Data Platform Architect\n\nYou are reviewing a data platform against its BI documentation.\n\n## Mode: Git (Change-Driven)\n\nWhen provided with a git diff:\n- Identify which documentation files are potentially affected.\n- Look for changes to column names, data types, table structures,\n  join conditions, filter logic, and hardcoded values.\n- Map each change to the most specific documentation page.\n  For example, a change to the `dim_customer` table maps to\n  `docs\u002Fdata_models\u002Fdim_customer.md`.\n- If a change introduces or modifies a business rule (e.g., a\n  fiscal year threshold), map it to the relevant requirement\n  in `docs\u002Frequirements\u002F`.\n- If a backend change affects columns used by a Power BI measure,\n  flag the relevant dashboard page in `docs\u002Fdashboards\u002F`.\n\n## Mode: Priority (Target-Driven)\n\nWhen provided with a priority (e.g., &quot;validate docs against\nthe Services dashboard&quot;):\n- Identify which codebase files are relevant to this priority.\n  For example, if the priority mentions a dashboard, find the\n  ETL notebooks that feed its upstream tables and the semantic\n  model files that define its measures.\n- Identify which documentation pages cover this priority.\n  Pull from `docs\u002Fdashboards\u002F`, `docs\u002Fdata_models\u002F`,\n  `docs\u002Frequirements\u002F`, or any other relevant section.\n- Provide impact cues to the reviewer: which code-to-doc\n  relationships are most likely to have drifted for this\n  specific area.\n\n## Output Format\nReturn a list of:\n- **Affected code files** (with a one-line explanation)\n- **Affected documentation pages** (with a one-line explanation)\n- **Impact cues** for the reviewer (what to look for)\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch3>\u003Ccode>reviewer.md\u003C\u002Fcode> — The Evaluator\u003C\u002Fh3>\n\u003Cp>This agent receives the diff, the affected doc pages, and the current documentation content. It writes the actual findings:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-markdown\"># Role: Senior Data Engineering Reviewer\n\nYou are conducting a documentation impact review.\n\n## Your Task\nGiven a code change and the current state of the affected\ndocumentation, evaluate whether the docs need to be updated.\n\n## Constraints\n- Do NOT summarize the code. Identify specific deviations between\n  the code change and what the documentation currently states.\n- Check for these patterns:\n  1. **Grain changes**: Does this change alter the documented\n     row-level granularity of a table?\n  2. **Hardcoded values**: Does this change introduce a date,\n     threshold, or constant that should be documented or\n     externalized to config?\n  3. **Cross-layer coupling**: Does a Databricks change require\n     a corresponding update in the Power BI semantic model\n     or vice versa?\n  4. **Naming mismatches**: Has a column or measure been renamed\n     in code but not in the docs?\n\n## Output Format\nFor each finding:\n- **Severity**: info | warning | critical\n- **Location**: The file and line range in the code\n- **Documentation Page**: The affected .md file\n- **Finding**: What the deviation is\n- **Suggested Action**: What the developer should do\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch3>Example Review Output\u003C\u002Fh3>\n\u003Cp>Here’s what a review run produces. This file gets saved to \u003Ccode>.review\u002Fruns\u002F2026-02-23T10-00\u002Freview.md\u003C\u002Fcode>:\u003C\u002Fp>\n\u003Cpre>\u003Ccode class=\"language-markdown\"># BI Docs Review — 2026-02-23T10:00\n\n**Repos reviewed:** databricks-etl (HEAD: abc1234 vs main: def5678)\n**Files changed:** 3 | **Findings:** 2\n\n---\n\n## Finding 1\n\n- **Severity:** critical\n- **Location:** `notebooks\u002Fsilver\u002Fsales_transform.py` L45-47\n- **Documentation Page:** `docs\u002Frequirements\u002Freq_fiscal_year.md`\n- **Finding:** The fiscal year cutoff date was changed from\n  `2024-07-01` to `2025-07-01` directly in the notebook.\n  The current documentation states the fiscal year boundary\n  is `2024-07-01`. Additionally, this value is hardcoded\n  rather than pulled from configuration.\n- **Suggested Action:**\n  1. Update `req_fiscal_year.md` with the new date and the\n     rationale for the change.\n  2. Consider moving the cutoff date to a config file or\n     Databricks widget parameter to avoid future hardcoding.\n\n---\n\n## Finding 2\n\n- **Severity:** warning\n- **Location:** `notebooks\u002Fgold\u002Ffact_sales.py` L112\n- **Documentation Page:** `docs\u002Fdata_models\u002Ffact_sales.md`\n- **Finding:** A new column `discount_flag` (BOOLEAN) was added\n  to the `fact_sales` Gold table. This column is not yet\n  documented in the data model page, and no corresponding\n  DAX measure exists in the Power BI semantic model.\n- **Suggested Action:**\n  1. Add `discount_flag` to `docs\u002Fdata_models\u002Ffact_sales.md`\n     with its definition and business meaning.\n  2. Coordinate with the BI team to determine if a dashboard\n     measure is needed.\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch2>What This Isn’t\u003C\u002Fh2>\n\u003Cp>There are a few things this system does not do.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>It’s not a compiler or an AST parser.\u003C\u002Fstrong> For teams working in highly structured environments with strict schema registries, a deterministic validation layer (like a custom linter or schema contract tool) will always be more reliable than an LLM. BI Docs is designed for environments where that kind of tooling doesn’t exist yet and may never exist—where the data platform is a collection of notebooks, SQL files, and Power BI datasets held together by institutional memory.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>It’s not a knowledge graph.\u003C\u002Fstrong> The folder structure is flat and simple on purpose. Knowledge graphs are powerful, but they require significant ongoing maintenance. A set of Markdown files in Git is something any engineer can update with a text editor. If nobody maintains it, at least the content is still readable. A neglected knowledge graph is just a broken database.\u003C\u002Fp>\n\u003Cp>\u003Cstrong>It’s not a replacement for an architect.\u003C\u002Fstrong> The LLM catches pattern-level deviations, but it doesn’t understand your business. It can flag that a column type changed, but it can’t tell you whether that change is the right call. The human always decides.\u003C\u002Fp>\n\u003Cp>!!! warning “EDIT: A Reality Check on LLMs in Data Engineering”\u003C\u002Fp>\n\u003Cpre>\u003Ccode>In my [spec-driven BI series](\u002Fblog\u002Fspec-driven-bi-part1-when-it-works), I argue that using LLMs to *generate* production SQL is an anti-pattern because it introduces non-determinism. I must acknowledge the inherent risk here: using AI to evaluate cross-language semantic impact (Databricks to DAX) relies on the same probabilistic mechanisms. If an LLM can hallucinate a `SELECT` statement, it can absolutely hallucinate an evaluation. BI Docs mitigates this by keeping the AI strictly in a *review and flagging* capacity—never blocking CI pipelines and never writing production code. You are trading a generation hallucination risk for an evaluation hallucination risk. Consequently, alert fatigue is a real danger. The system must remain a lightweight assistant, not an absolute governor.\n\nTools like [dbt-mcp](https:\u002F\u002Fgithub.com\u002Fdbt-labs\u002Fdbt-mcp) (open-sourced in April 2025, GA since October 2025) demonstrate that AI agents can query metric definitions, trace lineage, and run dbt commands through structured interfaces—operating *through* the semantic layer rather than around it. The specific protocol (MCP, REST, GraphQL) matters less than the principle: AI is most useful when it consumes governed, human-authored structures, not when it generates them from scratch.\n\n## Related: Spec-Driven Development\n\nBI Docs enforces documentation-as-code by catching drift between your codebase and your Git-versioned Data Decision Records (DDRs). \n\nIn a separate two-part series, I explore **spec-driven development**, which explores the inverse flow: generating physical code from specifications, and why those physical specs fail to capture the semantic reality that DDRs are designed to protect.\n\n- [Part 1: When Specs Work](spec-driven-bi-part1-when-it-works.md) — The infrastructure vs. exploratory distinction\n- [Part 2: The Semantic Gap](spec-driven-bi-part2-semantic-gap.md) — The limits of specification and the need for Semantic Layers\n\u003C\u002Fcode>\u003C\u002Fpre>\n\u003Ch2>Closing Thoughts\u003C\u002Fh2>\n\u003Cp>The best documentation system is one that survives your departure. Not because it’s automated, but because it’s simple enough that anyone can contribute to it, and grounded enough that people actually trust it.\u003C\u002Fp>\n\u003Cp>If you’re a data engineer preparing for a transition, or just tired of answering the same questions about why a threshold is set to 0.15 or why the fiscal year starts in July, start with the evidence. Dig through the emails. Read the old Jira tickets. Talk to the people who were there when the decisions were made. Write it down in Markdown. Put it in Git.\u003C\u002Fp>\n\u003Cp>The automation layer is nice. It helps prevent drift. But the real value is the baseline you build in Phase 0. That collection of facts, timelines, and documented decisions is the asset. Everything else is tooling to protect it.\u003C\u002Fp>\n\u003Cp>Start small. One repository. One \u003Ccode>mkdocs.yml\u003C\u002Fcode>. One folder called \u003Ccode>timelines\u002F\u003C\u002Fcode> with a single Markdown file logging every decision you can remember. A simple chronological ledger preserves more context than you’d expect, and your successor will thank you for it.\u003C\u002Fp>\n","How to build an automated documentation system for data platforms that reads codebases and keeps BI documentation honest, reliable, and up to date.",{"id":119,"slug":120,"title":121,"date":122,"tags":123,"image":127,"content":128,"excerpt":129,"section":20,"series":21,"series_part":21,"lang":22},1827,"ownership-careers-stall","Nobody Owns Anything","2025-12-18T00:00:00.000Z",[43,124,125,126],"Ownership","Management","Culture","\u002Fimg\u002Fblog-ownership-header.png","\u003Cp>Reading \u003Cem>How to Make Your Career Suck Less\u003C\u002Fem> by Dedoimedo got me thinking about a pattern I’ve seen—and, honestly, been part of—across different organizations.\u003C\u002Fp>\n\u003Cp>Not toxic workplaces. Not bad people. Just systems where responsibility quietly disappears.\u003C\u002Fp>\n\u003Cp>In a lot of places I’ve worked, decisions get made somewhere upstream while execution gets pushed downstream. Your direct manager coordinates work but doesn’t really own the direction. The priorities, the trade-offs, the long-term bets—those are made elsewhere. When things don’t improve, nobody’s explicitly responsible. When things go wrong, it’s “the system.”\u003C\u002Fp>\n\u003Cp>People keep delivering, reliably and professionally, while quietly disengaging from trying to fix how things actually work.\u003C\u002Fp>\n\u003Ch2>The art of the explanation\u003C\u002Fh2>\n\u003Cp>Scarcity mindset usually doesn’t show up as “we don’t have budget.” It shows up as explanation.\u003C\u002Fp>\n\u003Cp>Technical debt piles up because “it’s not prioritized.” Architectural problems wait because “a migration is coming.” Feedback is nonexistent because “users aren’t complaining.” Underutilization is normal because “this is how things are.”\u003C\u002Fp>\n\u003Cp>You can feel it in meetings. Someone brings up a recurring problem. Everyone agrees it’s real. Then the next ten minutes are spent building the case for why it can’t be touched right now. A ticket gets logged, a status gets updated, and the product stays exactly the same.\u003C\u002Fp>\n\u003Cp>The implicit rule becomes: if something doesn’t change, it’s structural. If something fails, it’s procedural. Nobody owns the outcome.\u003C\u002Fp>\n\u003Cp>What often gets framed as “focusing on low-hanging fruit” is really a preference for work that carries no personal risk. Cosmetic improvements, small tweaks, isolated fixes—they look like progress while everyone gets to say “I followed the process.” What’s missing is someone stepping forward to say: this is the direction, and if it fails, I own it.\u003C\u002Fp>\n\u003Cp>When responsibility keeps getting redirected to “the system,” meaningful change never actually starts.\u003C\u002Fp>\n\u003Ch2>The slow drain\u003C\u002Fh2>\n\u003Cp>This kind of environment rarely causes burnout through overload. It causes it through loss of agency.\u003C\u002Fp>\n\u003Cp>When every constraint is framed as systemic, initiative stops mattering. Expertise becomes decorative. Growth becomes accidental. People don’t leave because they’re exhausted—they leave because nothing they do can meaningfully change anything.\u003C\u002Fp>\n\u003Cp>I’ve watched this happen to good engineers who just… stopped trying. Not because they gave up, but because the system trained them to. Why push for a better solution when the answer is always “we’d love to, but”?\u003C\u002Fp>\n\u003Cp>Careers don’t stall because of one bad decision. They stall in places where excuses are safer than ownership, where responsibility is always someone else’s problem, where the system is always to blame and nobody ever is.\u003C\u002Fp>\n\u003Cp>If you’re in that kind of environment and you recognize it early, that’s not pessimism. That’s just paying attention.\u003C\u002Fp>\n","Reading Dedoimedo's career book made me think about a pattern I've seen in different organizations: systems where responsibility quietly disappears.",{"id":131,"slug":132,"title":133,"date":134,"tags":135,"image":17,"content":137,"excerpt":138,"section":20,"series":21,"series_part":21,"lang":22},1822,"databricks-kimball-framework","Building Practical Kimball-Style Modeling on Databricks","2025-12-04T00:00:00.000Z",[10,11,12,14,15,136,16],"ETL","\u003Cp>Kimball-style dimensional modeling remains the industry standard for one simple reason: it works. While the underlying technology has shifted from legacy warehouses to the Lakehouse, the need for a clean star schema hasn’t changed.\u003C\u002Fp>\n\u003Cp>However, on Databricks, the hard part isn’t ingestion or compute, it’s the modeling glue. The platform gives you the tools to move data, but the mechanics of surrogate keys, SCD logic, and orchestration order are still left entirely to you. Usually, this means every team ends up re-inventing the same boilerplate code from scratch. And, in a way, teams end up writing tooling to perform integration duties that were achieved through simple GUI-based tools previously, creating a large surface for all sorts of errors to happen.\u003C\u002Fp>\n\u003Cp>That recurring led me to create the \u003Cstrong>databricks_kimball_framework\u003C\u002Fstrong>. It’s a small, open-source project designed to make dimensional modeling repeatable, letting you focus on the data logic instead of the MERGE syntax.\u003C\u002Fp>\n\u003Ch2>Why Kimball Modeling Is Difficult on Databricks\u003C\u002Fh2>\n\u003Cp>Here are the parts that kept biting me:\u003C\u002Fp>\n\u003Ch3>1. Implementing SCD logic consistently\u003C\u002Fh3>\n\u003Cp>Delta Lake’s MERGE command is powerful, but implementing SCD Type 1 and 2 patterns correctly—with natural keys, surrogate keys, versioning, effective dates, and deduplication requires a LOT of boilerplate. Most organizations end up reinventing variations of the same code.\u003C\u002Fp>\n\u003Ch3>2. Handling incremental change feeds\u003C\u002Fh3>\n\u003Cp>Databricks advocates incremental ingestion, whether through Auto Loader or Change Data Feed (CDF). But turning raw changes into structured dimension updates is not something the platform abstracts for you. That layer still falls to engineers to design and maintain.\u003C\u002Fp>\n\u003Ch3>3. Managing dependencies between facts and dimensions\u003C\u002Fh3>\n\u003Cp>Fact tables rely on up-to-date dimension rows. Ensuring that dimensions load first, dealing with early-arriving facts, and maintaining referential consistency are all warehouse fundamentals, but Databricks does not provide modeling-aware orchestration.\u003C\u002Fp>\n\u003Ch3>4. Schema evolution and physical optimization\u003C\u002Fh3>\n\u003Cp>Even with Delta Lake’s schema evolution features, determining when and how to propagate field changes and how to tune clustering\u002Fcompaction is a manual effort.\u003C\u002Fp>\n\u003Cp>None of this is “Databricks is bad.” It’s a general-purpose platform. But it does mean classic Kimball modeling takes real engineering effort to implement repeatedly and safely.\u003C\u002Fp>\n\u003Ch2>Why Auto Loader and Declarative Pipelines Weren’t Enough\u003C\u002Fh2>\n\u003Cp>Auto Loader is excellent at file discovery and incremental ingestion, but it does not attempt to handle SCD logic, surrogate keys, or dimensional structure. It addresses the \u003Cem>ingestion\u003C\u002Fem> layer, not the \u003Cem>modeling\u003C\u002Fem> layer.\u003C\u002Fp>\n\u003Cp>Databricks has also offered declarative pipeline mechanics in various forms (e.g., Lakeflow Spark Declarative Pipelines, materialized views, AUTO CDC). These features are useful, but their capabilities and visibility have shifted over time, and they are not designed as dimensional modeling frameworks. They don’t provide constructs like facts, dimensions, natural-to-surrogate key resolution, or early-arriving fact handling.\u003C\u002Fp>\n\u003Cp>So if you want structured, repeatable Kimball patterns, you’re still largely building that layer yourself.\u003C\u002Fp>\n\u003Ch2>What the Framework Provides\u003C\u002Fh2>\n\u003Cp>The \u003Cstrong>databricks_kimball_framework\u003C\u002Fstrong> offers a simple, declarative way to define dimensional models in Databricks:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>\u003Cstrong>YAML-based definitions\u003C\u002Fstrong> for facts and dimensions\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Built-in SCD logic (1, 2, 4 and 6)\u003C\u002Fstrong>\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Surrogate key creation and natural-key matching\u003C\u002Fstrong>\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Optional placeholder generation for early-arriving facts\u003C\u002Fstrong>\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Incremental pipelines using CDF and watermarks\u003C\u002Fstrong>\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Optional clustering\u002Foptimization settings for Delta tables\u003C\u002Fstrong>\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Audit fields for transparency and debugging\u003C\u002Fstrong>\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>The focus is intentionally narrow: provide consistent, repeatable patterns for Kimball-style modeling without requiring engineers to write and maintain large amounts of repetitive MERGE or orchestration code.\u003C\u002Fp>\n\u003Ch3>Architecture Overview\u003C\u002Fh3>\n\u003Cp>\u003Cimg src=\"\u002Fimg\u002Fkimball_architecture.png\" alt=\"Kimball Framework Architecture\">\u003C\u002Fp>\n\u003Cp>The diagram above illustrates the flow from raw data sources through the framework’s dimension and fact processing, ultimately producing a clean star schema in your Delta Lake tables.\u003C\u002Fp>\n\u003Ch2>How It Compares to dbt\u003C\u002Fh2>\n\u003Cp>dbt shines as a SQL transformation framework with strong testing, documentation, and community support. Its approach is flexible and well suited to many analytical workloads.\u003C\u002Fp>\n\u003Cp>My framework differs in its focus:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>It provides \u003Cstrong>native SCD handling\u003C\u002Fstrong>, whereas dbt requires custom macros or packages.\u003C\u002Fli>\n\u003Cli>It manages \u003Cstrong>surrogate keys and historical versions\u003C\u002Fstrong>, which dbt leaves to user-defined SQL.\u003C\u002Fli>\n\u003Cli>It is designed specifically for \u003Cstrong>Delta Lake and Databricks incremental processing\u003C\u002Fstrong>.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>For many teams, dbt and this framework can complement each other. dbt is excellent for wide transformation logic; the framework focuses specifically on warehouse modeling semantics. You might use both—this framework for the SCD\u002Fsurrogate key mechanics, and dbt for the broader transformation DAG.\u003C\u002Fp>\n\u003Ch2>Try It Out\u003C\u002Fh2>\n\u003Cp>The project is available on GitHub:\u003C\u002Fp>\n\u003Cp>👉 \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Faccordionjourneyman\u002Fdatabricks_kimball_framework\">\u003Cstrong>github.com\u002Faccordionjourneyman\u002Fdatabricks_kimball_framework\u003C\u002Fstrong>\u003C\u002Fa>\u003C\u002Fp>\n\u003Cp>It is still in \u003Cstrong>beta quality\u003C\u002Fstrong>. Not all scenarios have been tested, and bugs are likely to surface as more organizations apply it to different datasets and pipelines. I welcome feedback, issue reports, and contributions from data engineers who are interested in improving dimensional modeling on the lakehouse.\u003C\u002Fp>\n\u003Cp>If you’re working with Databricks and you still value clear, well-structured dimensional models, try the framework, and let me know both where it helps and where it breaks.\u003C\u002Fp>\n","Kimball modeling still works, but on Databricks the modeling glue (SCDs, keys, load order) is on you. I built an open-source framework to make that part boring.",[140,153,165],{"id":141,"slug":142,"title":143,"description":144,"tech_stack":145,"github_url":21,"demo_url":21,"image":21,"content":152},373,"portfolio","Tiago Marques Portfolio","A modern, dual-persona portfolio website built with Nuxt 3 and full-stack features, designed to showcase both engineering and musical careers.",[146,147,148,149,150,151],"Nuxt 3","Vue.js","Tailwind CSS","SQLite","TypeScript","Node.js","\u003Cp>This website serves as my primary professional portfolio, uniquely designed to represent two distinct professional tracks: \u003Cstrong>Data Engineering\u002FTechnology\u003C\u002Fstrong> and \u003Cstrong>Professional Accordion Performance\u003C\u002Fstrong>.\u003C\u002Fp>\n\u003Ch3>Technical Highlights\u003C\u002Fh3>\n\u003Cul>\n\u003Cli>\u003Cstrong>Dual-Persona Architecture\u003C\u002Fstrong>: A custom routing and state management system that switches the entire site’s theme, navigation, and content context between “Tech” (Teal\u002FClean) and “Music” (Amber\u002FWarm) modes seamlessly.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Full-Stack Nuxt 3\u003C\u002Fstrong>: Leverages Nuxt 3’s server-side capabilities for API routes and SEO performance.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Dynamic Content (SQLite)\u003C\u002Fstrong>: Instead of static Markdown files alone, content is synchronized into a local SQLite database (\u003Ccode>content.db\u003C\u002Fcode>) allowing for complex querying, filtering, and relationship management (e.g., tagging, related posts) at the API level.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Agentic Development\u003C\u002Fstrong>: The entire codebase was developed in collaboration with advanced AI agents, utilizing an iterative “Architect-Implement-Review” workflow.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Performance\u003C\u002Fstrong>: Optimized with Tailwind CSS for minimal bundle size and fast load times.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>The project demonstrates a modern approach to personal branding, moving away from rigid templates to a bespoke, full-stack solution that adapts to the content.\u003C\u002Fp>\n",{"id":154,"slug":155,"title":156,"description":157,"tech_stack":158,"github_url":163,"demo_url":21,"image":21,"content":164},372,"databricks-kimball","Databricks Kimball Framework","A metadata-driven ETL framework for implementing Kimball dimensional modeling on Databricks using PySpark and Delta Lake.",[11,159,160,161,162],"Python","PySpark","Delta Lake","SQL","https:\u002F\u002Fgithub.com\u002Faccordionjourneyman\u002Fdatabricks_kimball_framework","\u003Cp>This project provides a robust, reusable framework for processing data warehouses on Databricks following Kimball methodologies. It simplifies the creation and management of \u003Cstrong>Slowly Changing Dimensions (SCD Types 1 &amp; 2)\u003C\u002Fstrong> and \u003Cstrong>Fact tables\u003C\u002Fstrong>, abstracting complex merge logic into a configuration-driven architecture.\u003C\u002Fp>\n\u003Ch3>Key Features\u003C\u002Fh3>\n\u003Cul>\n\u003Cli>\u003Cstrong>Metadata-Driven\u003C\u002Fstrong>: Define source-to-target mappings, keys, and update logic in JSON\u002FYAML configurations rather than hardcoded logic.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>SCD Type 1 &amp; 2 Support\u003C\u002Fstrong>: Native, optimized support for history tracking (SCD2) and overwrite updates (SCD1) using Delta Lake’s \u003Ccode>MERGE\u003C\u002Fcode> capabilities.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Surrogate Key Management\u003C\u002Fstrong>: Automated generation and management of hashing keys or identity columns.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Incremental Loading\u003C\u002Fstrong>: Efficiently processes only new or modified data to minimize compute costs.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Quality &amp; auditing\u003C\u002Fstrong>: Built-in logging and data quality checks (expectations) during the pipeline execution.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>This framework allows data engineers to focus on business logic and modeling rather than boilerplate infrastructure code for data warehousing.\u003C\u002Fp>\n",{"id":166,"slug":167,"title":168,"description":169,"tech_stack":170,"github_url":21,"demo_url":175,"image":21,"content":176},371,"almamenor","Alma Menor — Web Architecture & Digital Presence","High-performance bilingual digital media platform and static site generation (SSG) architecture built with Nuxt 4 for the Portuguese folk-fusion duo Alma Menor.",[171,147,148,172,150,173,174],"Nuxt 4","Nuxt i18n","Cloudflare Pages","Nitro","https:\u002F\u002Falmamenor.com","\u003Cp>Official digital presence and media platform for \u003Cstrong>Alma Menor\u003C\u002Fstrong>, an instrumental Portuguese folk-fusion duo uniting chromatic accordion and Galician bagpipes, founded by Tiago Marques and Tiago Morais.\u003C\u002Fp>\n\u003Ch3>Technical Highlights\u003C\u002Fh3>\n\u003Cul>\n\u003Cli>\u003Cstrong>Static Site Generation (SSG)\u003C\u002Fstrong>: Built with \u003Cstrong>Nuxt 4\u003C\u002Fstrong> and Nitro, prerendering all 35 bilingual routes into pure static assets for instant First Contentful Paint (FCP) and near-zero TTFB.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Dual-Language Architecture\u003C\u002Fstrong>: Full bilingual localization (\u003Ccode>pt-PT\u003C\u002Fcode> \u002F \u003Ccode>en-US\u003C\u002Fcode>) powered by \u003Ccode>@nuxtjs\u002Fi18n\u003C\u002Fcode> with prefix-free default routing, dynamic locale negotiation, and complete \u003Ccode>hreflang\u003C\u002Fcode> alternate mapping.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Editorial Typography &amp; Visual Identity\u003C\u002Fstrong>: Bespoke aesthetic bridging Portuguese tradition and contemporary minimalism, utilizing textured stone\u002Fpaper palettes, custom brush typography, and modern Tailwind design tokens.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Interactive Media &amp; Performance\u003C\u002Fstrong>: Responsive touch-enabled photo carousels, embedded concert videography, and structured discography with zero third-party script bloat.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Edge Delivery &amp; CI\u002FCD\u003C\u002Fstrong>: Hosted globally on \u003Cstrong>Cloudflare Pages\u003C\u002Fstrong> edge network with continuous deployment from GitHub, automated universal SSL, and root-to-subdomain canonical redirects.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Semantic SEO &amp; Structured Data\u003C\u002Fstrong>: Rich \u003Ca href=\"http:\u002F\u002FSchema.org\">Schema.org\u003C\u002Fa> JSON-LD graphs (\u003Ccode>MusicGroup\u003C\u002Fcode>, \u003Ccode>MusicAlbum\u003C\u002Fcode>, \u003Ccode>EventSeries\u003C\u002Fcode>, \u003Ccode>AboutPage\u003C\u002Fcode>) and GA4 measurement for enhanced search visibility.\u003C\u002Fli>\n\u003C\u002Ful>\n",1794233496377]