Skip to main content

> Term

data lineage

The visual and technical map tracing the complete lifecycle of data from its source to its final consumption.

Detailed Explanation

Data lineage provides the 'audit trail' for data pipelines. It records exactly where data originated, every ETL transformation applied to it, and the dashboards or models consuming it.

It is heavily relied upon for debugging data errors, determining the impact of schema changes, and meeting compliance auditing requirements.

Why It Matters

When a CEO's dashboard shows wrong revenue numbers, data lineage is the only way an engineer can trace the error back to a broken upstream API.

Common Failure Mode

An upstream team alters a column format in a core database without knowing that four downstream machine learning models rely on it, causing silent model degradation.

Practical Example

Using dbt to map table dependencies and establish data lineage.

Production Manifestation

Tools like dbt, Apache Atlas, or OpenLineage mapping dependencies visually as Directed Acyclic Graphs (DAGs).

Frequently Asked Questions

What is data lineage in short?

The visual and technical map tracing the complete lifecycle of data from its source to its final consumption.

What is the most common failure mode?

An upstream team alters a column format in a core database without knowing that four downstream machine learning models rely on it, causing silent model degradation.

AI Summary

The visual and technical map tracing the complete lifecycle of data from its source to its final consumption. When a CEO's dashboard shows wrong revenue numbers, data lineage is the only way an engineer can trace the error back to a broken upstream API.