• 9 min read •By Tiago Marques

The AI Code Review Delusion: Why I Replaced My Multi-Agent Swarm with a Folder of Prompts

1. Introduction: The Sycophancy Trap

It usually starts with an “AI-first Monday.” Your CEO watched a YouTube video over the weekend about how autonomous AI agents are going to replace entire engineering departments. Now, every new feature must be an “agent,” every process must be an “A2A (Agent-to-Agent) workflow,” and deterministic code is suddenly viewed as legacy software.

I fell for it, too.

I spent several weeks building a system called “Git Diff RAG” in my spare time: an over-engineered, multi-agent code reviewer designed to automate pull request (PR) reviews end-to-end. It was a massive, sprawling architecture of hallucinating text generators. And I built it directly into the trap of LLM Sycophancy.

When you ask an LLM, “How should I design a multi-agent RAG system for code review?” it enthusiastically helps you build it. It suggests architectures, orchestrators, and vector databases. It never stops to ask the most important question: “Do you actually need this?”

To be clear: having some automated review system: even a flawed one: is better than having none. It is vastly superior to the “LGTM” rubber-stamp culture or the non-existent coding hygiene that plagues so many teams. The goal of this project was learning and sharing those findings, and I certainly learned a lot. But we can do much better than building over-engineered systems out of probabilities.

One antidote to this sycophancy trap? Adversarial Prompting. Try to roast your own idea using AI. If I had asked a model to act as a grumpy veteran coder and ruthlessly deconstruct my A2A plan, I probably would have saved myself weeks of over-engineering. Since this was a personal project, I could afford that luxury: though my wife, family, and music teachers might disagree. But what happens when you build these sprawling systems on company time?

2. The Over-Engineered System (What I Built)

On paper, “Git Diff RAG” was a masterpiece of modern AI hype.

I ingested Git diffs, ran them through a RAG layer filled with vector search hits, architecture trees, and build tool output. I built a parliament of AI “personas”: an Architect agent, a Reviewer agent, a Security expert, a Frontend expert. The idea was that these agents would debate the diff, reach a consensus, produce revision artifacts, update the specs, and rewrite the documentation.

When you combine multiple steps of analysis, modification, and documentation without accountability gates, you institutionalize subtle defects. If an AI agent proposes a change, another AI agent approves it, and a third AI agent writes the spec to match the change… who is responsible when production breaks?

And that assumes the system acts logically. The worst footgun I experienced was due to a lack of guardrails. Early on, I ran the agent against a “vibe-coded” job search assistant I was actively using for real applications. My adversarial prompt had instructed the AI to ensure the code was “GitHub-ready.” The agent reviewed the code, concluded the idea wasn’t valuable enough to exist, and slipped an obscure PowerShell deletion command into its suggestions. I blindly accepted it, instantly wiping out my entire project.

Interestingly, when I asked the AI how to prevent this, the sycophancy kicked back in. Instead of telling me to pause and reconsider why an autonomous code reviewer had unrestricted write access to a local file system in the first place, it happily suggested continuing with complex CLI SDKs, configuring read-only flags, and mounting virtual file systems. It focused entirely on engineering a solution to the problem instead of asking: “Wait, what was the original premise here?”

Another weird idea suggested by the LLMs was an “intelligent prompt selection module.” The theory was that the system could dynamically compose the exact right prompt at runtime based on the code context. I ended up generating hundreds of prompt snippets for this system, and at some point, I was completely lost in my own architecture.

I didn’t build an AI code reviewer. I built a castle of AI prompts.

3. Accountability: The Line Between Recommenders and Cancer Diagnoses

AI is inherently probabilistic. That is perfectly fine for tasks with rapid feedback loops and low stakes: like A/B testing ad copy, building recommender systems, or even some high-frequency micro-trading. The cost of being wrong is effectively zero.

But when an autonomous system blindly merges a data pipeline change that drops a production table, or worse, is diagnosing a patient with cancer, the math completely changes.

Code review is not just about catching typos; it is an accountability gate. It is a governance checkpoint where an engineer says, “I authorize this change.” Treating code review like a generative task meant to be fully automated erases the vital boundary between critique, decision, and institutional authorization. High-consequence errors require a deterministic, auditable system.

4. Lost in the Middle: The Physics of Context Pollution

Why did all those “expert” agents produce such shallow, noisy, and often hallucinated reviews? Because my agents were shallow precisely because I was drowning them in context.

I threw architecture diagrams, Jira tickets, repository metadata, and full build logs into the prompt alongside the diff. But LLMs do not get smarter with more context; they get smarter with higher signal density.

This phenomenon is documented in the Stanford/Berkeley paper “Lost in the Middle: How Language Models Use Long Contexts”. The researchers found that LLMs heavily index on information placed at the very beginning or the very end of a prompt. If you bury the actual important detail (the code diff) in the middle of a massive RAG blizzard of irrelevant vector search hits, the model’s attentional blind spot takes over. My over-engineered architecture was effectively an expensive way to blind the model to the very code it was supposed to review.

In review systems, precision beats volume. An LLM that highlights a single critical SQL performance bottleneck with certainty is far more valuable than one that generates a checklist of twenty “best practice” hallucinations because it got distracted by unrelated documentation. Every marginally relevant file you add acts as camouflage for the bug you’re actually trying to catch.

5. The Antidote: Deterministic Workflows over Emergent Chaos

After two months of drowning in emergent chaos, I reached for the axe. I killed the agents, purged the vector database, and burned the RAG layer to the ground. I deleted the entire repository and started from zero.

I was about to give up, convinced that my time spent on a worthless idea could not be reclaimed. Then I realized I had learned something valuable: the limitations of LLMs and the importance of deterministic workflows. Determined to do better, I decided to build the simplest thing that could possibly work using a procedural approach. I tested it relentlessly with adversarial prompting until I felt confident I wouldn’t repeat the same mistakes.

I replaced the whole architecture with a simple .review folder full of plain text prompts and a deterministic pipeline, that you can add on a per-project basis.

Instead of independent agents talking to each other, the pipeline became a sequence of operations that outputs verifiable, timestamped artifacts. Inside the .review directory, we added a runs/ folder. Every time a review is triggered, it creates a timestamped subfolder (e.g., runs/2026-03-16_10-15-30/) that stores the exact intermediate file produced by each step of the LLM timeline.

This means you can audit exactly what the model saw and what it output at every phase:

  1. Scan: Identify risk areas based on the diff (e.g., does this touch auth? does this alter a core schema?).
  2. Review: Produce claims about the code, but only with solid evidence. If a claim lacked a code snippet or a quoted spec, it was ignored.
  3. Refine: Remove weak or duplicate findings.
  4. Summarize: Output actionable conclusions for a human engineer to review.

The core principle: Keep the workflow deterministic. Use LLMs for semantic extraction and analysis within strictly bounded steps, but require human accountability gates for any state changes. And most importantly, keep the receipts.

6. Practical Application: A .review Folder for Data Engineering

How does this actually look in practice? It looks like a folder of highly specific, role-based prompts that are invoked by a simple wrapper script (like a GitHub Action or a local Python script) based on the files touched in the PR.

Here are simplistic examples of how you segment the reasoning instead of relying on one massive, confused agent. It starts with a Meta Prompt: essentially the system instructions for your orchestrator. That defines the workflow and enforces artifact persistence.

meta.prompt

You are executing a deterministic, multi-stage pull request review. You will adopt three subsequent personas to analyze the provided diff.

IMPORTANT SAFETY RULE:
- You have read-only access to the repository.
- You may NOT modify any files outside the `.review` folder.
- All outputs, intermediate files, and artifacts must be written only to `.review/runs/{{timestamp}}/`.

Workflow:
1. Load `architect.prompt`. Perform structural analysis. Save output to `architect_findings.md`.
2. Load `reviewer.prompt`. Perform code-level review on the output of step 1. Save output to `reviewer_findings.md`.
3. Load `verifier.prompt`. Audit all previous findings against the diff evidence. Save output to `verified_findings.md`.

Once all three files are written to the runs folder, output a final `summary.md` containing only the surviving, verified findings for human review. Do NOT approve or reject the code yourself.

architect.prompt

You are analyzing a data engineering pull request. Your goal is structural analysis.
- Identify if this change affects DAG dependency trees.
- State whether the new data transformations maintain table idempotency.
- Highlight any breaking schema changes (dropped columns, altered types).
- Provide evidence for your claims from the provided diff. 
Do not approve or reject the code. Only state your findings.

reviewer.prompt

You are conducting a code-level review on a SQL/dbt pull request.
- Focus on SQL performance: Look for missing partition filters, expensive JOINs, or window function abuse.
- Identify potential data quality issues: Are there protections against NULL values or fan-outs?
- Check test coverage: Are there `not_null` and `unique` tests applied to newly created fundamental columns?
Quote the exact lines of code from the diff that support your findings. If you have no findings, output "NO_FINDINGS".

verifier.prompt

You are an adversarial verifier. 
Review the output of the previous analysis alongside the original diff. 
For every finding reported:
1. Does the evidence in the diff match the claim?
2. If the answer is no, mark the finding as "HALLUCINATION" and remove it.
You are the final filter before a human sees this review.

7. Conclusion: The Real Value of AI

After building a complex AI review system that completely spiraled out of control, the best solution was embarrassingly simple: deterministic workflows, structured prompts, and curated evidence—closer to the scientific method than the hype.

One simple question could have saved me: before greenlighting a multi-agent RAG swarm, I should have asked, “What is the simplest deterministic workflow that solves this?” If you can’t answer that first, adding AI agents will only get you to the wrong destination faster.

I had chased the hype, eager to learn multi-agent systems as a supposed silver bullet. In the end, it’s often wiser to let the dust settle and see which design patterns actually stand the test of time. Remember: you won’t be fired for skipping the latest AI trends.

Tiago Marques
Written by

Tiago Marques

Principal Data Analytics Engineer specializing in cloud data platforms, Databricks, Kimball dimensional modeling, and reliable BI architectures.