Great Expectations: Automated Data Quality Checks for ML Pipelines

Updated
7 min read
Great Expectations: Automated Data Quality Checks for ML Pipelines
TL;DR

Great Expectations lets you define what good data looks like, validate it automatically in your pipeline, and generate documentation - catching data issues before they corrupt your models.

Why Data Quality Matters for ML

Garbage in, garbage out is not a metaphor - it is the most common cause of ML model failures in production. A model trained on clean data and served corrupted data degrades silently. Great Expectations (GX) gives your pipeline a data quality layer that fails loudly when data does not meet your expectations.

Core Concepts

Expectation: a testable assertion about your data. "Column X should not be null." "Column Y mean should be between 100 and 200."

Expectation Suite: a collection of expectations for a dataset.

Checkpoint: runs an Expectation Suite against data and produces a validation result.

Data Docs: auto-generated HTML documentation showing your expectations and validation history.

Defining Expectations

python
import great_expectations as gx
import pandas as pd

context = gx.get_context()

df = pd.read_csv("orders.csv")
validator = context.sources.pandas_default.read_dataframe(df)

# Basic column expectations
validator.expect_column_values_to_not_be_null("order_id")
validator.expect_column_values_to_be_unique("order_id")
validator.expect_column_values_to_be_in_set("status", ["pending", "shipped", "delivered", "cancelled"])

# Statistical expectations
validator.expect_column_mean_to_be_between("total_usd", min_value=10.0, max_value=500.0)
validator.expect_column_min_to_be_between("total_usd", min_value=0.0, max_value=1.0)
validator.expect_column_values_to_be_between("total_usd", min_value=0, max_value=10000)

# Schema expectations
validator.expect_column_to_exist("created_at")
validator.expect_table_row_count_to_be_between(min_value=1000, max_value=10_000_000)

validator.save_expectation_suite(discard_failed_expectations=False)

Connecting to Real Databases

python
import great_expectations as gx

context = gx.get_context()

# PostgreSQL
pg_datasource = context.sources.add_postgres(
    name="production_db",
    connection_string="postgresql://user:pass@host:5432/dbname",
)
asset = pg_datasource.add_table_asset("orders", table_name="orders")

# BigQuery
bq_datasource = context.sources.add_bigquery(
    name="bigquery_prod",
    project="my-project",
)

Checkpoints for Pipeline Integration

python
checkpoint = context.add_or_update_checkpoint(
    name="daily_orders_checkpoint",
    validations=[
        {
            "batch_request": asset.build_batch_request(),
            "expectation_suite_name": "orders_suite",
        }
    ],
)

results = checkpoint.run()

if not results["success"]:
    raise ValueError("Data quality check failed  -  pipeline halted")

Embed this in your Airflow DAG or Prefect flow to halt pipelines on data quality failures.

Generating Data Docs

bash
great_expectations docs build
great_expectations docs serve  # opens browser with validation history

Data Docs show every expectation, when it was last validated, and the pass/fail rate over time. Share this URL with data consumers to document your SLAs.

GX Cloud vs Self-Hosted

GX Core (open source): all the above features, self-managed. Free.

GX Cloud: managed UI, team collaboration, alerting. Free tier for small teams.

GX vs Soda Core

Soda Core is an alternative that uses YAML-defined checks instead of Python. Simpler for non-engineers. GX is more powerful for complex statistical expectations and better Python integration.

Resources: Great Expectations, GitHub, docs.

#great-expectations#data-quality#validation#testing#mlops

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns, delivered to your inbox.

What should we test or explain next?

Comments

Have a question or something to add? Sign in to join the discussion.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

Software architect, full-stack engineer, and machine learning practitioner. Founder & CEO of Pristren, CTO at Venture Kick-backed PIN Sport Application GmbH (Basel, Switzerland), Software Development Lead at DailyFrame GmbH (building Dimetra patient transport software), and creator of Zlyqor. Focused on custom workflow automation, scalable web and mobile applications, and production AI/ML systems.