Work · Open source · 02

faultline

When a dbt pipeline breaks, what broke it, and where is the evidence?

In 30 seconds

Problem
When a dbt pipeline breaks, an engineer digs through tests, lineage and commits for hours to find what broke it.
Result
Named the exact root cause in 43 of 48 dev incidents (89.6%), with 0 confidently wrong answers and 0 false alarms on 30 clean runs. Held-out score: pending.
My role
Sole developer: fault injector, 192-incident benchmark, eval harness, monitors and triage.
  • measured 89.6% root cause found at top-1 on dev (43 of 48 incidents)
  • measured 0 of 48 confidently wrong answers on dev; 0 of 30 false alarms on clean runs
  • measured 192 injected incidents across 8 fault types, plus 70 clean runs
  • pending — held-out score: the frozen test split has not been run yet
Type
Open source
When
November 2026 · in progress
Role
Sole developer: injector, benchmark, eval harness, monitors, triage
Scale
4 dbt projects · 193 tests · 5 methods scored
Status
Milestone 5 of 8 · benchmark v1 released

01In plain language

Data teams build their reporting tables with dbt. When a number on a dashboard is suddenly wrong, someone has to find out why: bad data arriving, a column renamed upstream, a join that started duplicating rows, or a code change last Tuesday.

faultline does that search. It watches the tables, groups the symptoms into one incident, proposes possible causes, and tests each one against the recorded data. It only names a cause that passed its test. If none did, it says "not confirmed".

To know whether that works, I first built the exam: a tool that breaks real dbt projects on purpose, in eight different ways, and records the right answer for every break.

02The question

dbt's own tests catch some failures. How many root causes can a system name exactly, down to the table, column and commit, without ever pointing confidently at the wrong one?

03Why the obvious approach fails

dbt tests are the standard safety net. On the dev split they raised an alert for 22.9% of incidents and named the cause for none. A test tells you that a table is wrong, not why.

Pasting the logs into a language model is the popular shortcut. With a local 7B model, it named the right cause once in 48 incidents and a wrong cause seven times.

Both share the same gap: neither checks a guess against the data before saying it.

04The gate

Every decision goes through the same order: deterministic lineage and checks first, then statistics, and a language model last. The model's only planned job is to rank causes that have already passed a test and to write the report. No raw rows ever reach a model.

LayerJobState
LineageTable and column lineage from compiled SQL; what feeds whatBuilt
MonitorsSeven checks: schema, volume, freshness, null rate, distribution, duplicates, time shiftBuilt
TriageGroup symptoms, propose causes, test each one, keep only survivorsBuilt (deterministic)
Local LLMRe-rank survivors and write the reportNext milestone

The headline number on this page comes from the deterministic layers alone. The model has not been added yet, so it cannot have helped.

05Thought process

Left to right, the path one incident takes. Each card hands the next one less uncertainty.

  1. 01

    Gather

    • dbt artefacts and run logs
    • Monitor anomalies, failed tests
    • Lineage of this run, cached

    Symptoms, not guesses.

  2. 02

    Group

    • Symptoms joined by table lineage
    • One incident, not ten alerts
    • Stored in DuckDB

    One problem, one report.

  3. 03

    Propose

    • Data causes at seeds and sources
    • Code causes at changed models
    • With the commit that changed them

    Every place it could have started.

  4. 04

    Test

    • One check per cause
    • On aggregates, lineage, SQL, history
    • Failing causes are dropped

    Guesses must survive the data.

  5. 05

    Rank and report

    • Survivors only
    • "Not confirmed" if none survive
    • LLM re-ranker: next milestone

    Never a confident wrong answer.

The highlighted card is where the local model will sit. It will only ever see causes that already passed a test, never the data itself.

06Decisions, and what they cost

ChosenBuild the exam before the student

A fault injector and 192 scored incidents came before any diagnosis code.

Instead ofDemo on hand-picked examples

Hand-picked examples always work. A benchmark tells you how often.

ChosenA sealed held-out project

The jira split (58 runs) was generated by me on my own machine and never opened in the coding session. A CI guard blocks any code that names it.

Instead ofTune and test on the same incidents

Whoever writes the rules must never see the final exam.

ChosenScore at location level

A cause counts only if the table, the column (when there is one) and, for code faults, the commit are right.

Instead ofScore "right table"

The table-only score is reported, but only as a diagnostic, never as the headline.

ChosenCode faults must change something

A code mutation is kept only if the build fails or a table profile changes.

Instead ofKeep every mutation

A change with no symptom has no correct answer and would only add noise.

ChosenLineage follows the SQL

When compiled SQL reads a table dbt did not declare, the edge is kept and reported.

Instead ofTrust dbt's declared dependencies

The SQL is what runs. The Stripe project has two such hidden reads.

07What came out

$ faultline eval --split dev --baseline triage

| Fault type          | n  | Top-1          | Wrong top-1 |
|---------------------|----|----------------|-------------|
| schema_change       | 6  | 100.0% (6/6)   | 0.0% (0/6)  |
| late_or_missing     | 6  | 100.0% (6/6)   | 0.0% (0/6)  |
| duplicate_keys      | 6  | 100.0% (6/6)   | 0.0% (0/6)  |
| null_spike          | 6  | 100.0% (6/6)   | 0.0% (0/6)  |
| distribution_shift  | 6  |  66.7% (4/6)   | 0.0% (0/6)  |
| model_logic_bug     | 6  |  83.3% (5/6)   | 0.0% (0/6)  |
| broken_join         | 6  | 100.0% (6/6)   | 0.0% (0/6)  |
| timezone_error      | 6  |  66.7% (4/6)   | 0.0% (0/6)  |
| all faulted         | 48 |  89.6% (43/48) | 0.0% (0/48) |

Without the tests (every proposed cause ranked) top-1 would be 39.6% (19/48).
False alarms on clean runs: 0.0% (0/30).
Output of the tool, trimmed to three columns. Full table: eval/results/dev/triage.md (config hash 2d6665853400).

Five methods on the same 48 dev incidents. Teal is the share of root causes named exactly. Grey is the share named confidently and wrongly.

right cause at top-1 wrong cause at top-1
  • dbt testsalert only
    0.0% (0/48)
    0.0% (0/48)
  • dbt tests + guessalert plus naive cause
    0.0% (0/48)
    22.9% (11/48)
  • Raw logs → LLMlocal qwen2.5 7B
    2.1% (1/48)
    14.6% (7/48)
  • faultline monitorsanomalies, ranked upstream-first
    56.3% (27/48)
    33.3% (16/48)
  • faultline triagemonitors + tested causes
    89.6% (43/48)
    0.0% (0/48)
Dev split, 48 faulted incidents. Each method also had 0 of 30 false alarms on clean runs. Files: eval/results/dev/*.md.

The monitors alone already detect 89.6% of incidents, but they point at the wrong place a third of the time. Testing every proposed cause removes all of those wrong answers. The 5 incidents triage misses are ones the monitors never flagged: two distribution shifts, two timezone errors and one logic bug.

A one-time check on the train split gave a similar picture: 85 of 96 right at top-1 (88.5%), 3 wrong. Both results reproduce byte for byte, and a test checks that they still do.

Sources: triage, monitors, raw_log_llm, dbt_tests, dbt_tests_guess, train/triage.

08What it refuses to claim

A held-out score. Dev is where the method was developed, so 89.6% is an optimistic number. The held-out jira split will be run once, when the method is final. That number goes here, good or bad.

That abstaining works. "Not confirmed" never fired on dev or train. The behaviour exists in code but the benchmark has not tested it yet.

That the model helps. The LLM re-ranker is not built. Every number above is from deterministic code.

09What broke

A green build with empty tables

The first Stripe build passed 105 of 105 tests while every staging table was empty. dbt had read the sources before the seed data loaded. Now seeds load first, and a test fails if any model is empty.

The answers were in the logs

The saved git history still carried the injector's own commit messages, which described each fault. Any method could have read the answer. They are now removed at load time, and a leak test runs on every dev evaluation.

Realistic noise broke clean runs

Dropping a few percent of rows to simulate real data left orders without payments and failed tests on runs that were meant to be clean. Clean runs now vary only by 2% noise on numbers. The cost: volume false alarms became an easy test. That is listed as a known weakness.

10Still open

  • Ablations: no tests, no lineage, no model. What each layer is actually worth.
  • The local LLM re-ranker, run on my own MacBook.
  • One run on the sealed held-out split.
  • Incidents where "not confirmed" is the right answer, so abstaining gets tested.
  • court_cases is a synthetic project I wrote. Real public court data comes after a licence check.
  • Drafting fixes as pull requests (milestone 7) and a read-only MCP server (milestone 8).

11Reproduce it

Every number on this page comes from these commands, run in the repository.

git clone https://github.com/omkar-k0/faultline && cd faultline
make setup            # uv env, install, build the 4 dbt test projects
make check            # ruff, mypy --strict, dbt build, pytest
make fetch-benchmark  # loads the train and dev incidents
make eval-dev         # scores every method on dev, writes eval/results/dev/
FileWhat it holds
eval/results/dev/triage.mdThe 89.6% table, per fault type, with config hash
eval/results/dev/ablations.mdTriage without tests and without lineage
benchmark/manifest.jsonThe 192 incidents and 70 clean runs in release benchmark-v1
docs/EVAL.mdHow top-1, top-3 and "not confirmed" are counted

The held-out split ships sealed and stays unopened until the method is frozen.