Apache Parquet: The Deep Dive into Columnar Storage, Compression, and Query Pruning

Updated
7 min read
Apache Parquet: The Deep Dive into Columnar Storage, Compression, and Query Pruning
TL;DR

Apache Parquet stores columns together instead of rows, enabling 10-100x faster analytics queries and 5-10x better compression than CSV - here is everything you need to know to use it effectively.

In modern data engineering and analytical architectures, file format selection is one of the most critical structural decisions you will make. Storing large-scale analytical datasets in row-oriented text formats like CSV or newline-delimited JSON introduces massive performance penalties: excessive disk utilization, CPU-intensive text parsing, and catastrophic I/O waste when queries scan gigabytes of irrelevant fields.

Apache Parquet solves these architectural bottlenecks. Created through a collaboration between Twitter and Cloudera, Parquet is an open-source, column-oriented data file format designed specifically for complex analytical queries. By organizing records by column rather than by row, Parquet enables extreme compression ratios, efficient encoding schemes, and aggressive query pruning via predicate pushdown.

Whether you are querying data lakes with Trino and Apache Spark, running local in-process analytics with DuckDB and Polars, or structuring open table formats like Apache Iceberg and Delta Lake, understanding the low-level physical anatomy of Apache Parquet is essential for optimizing query latency and cloud storage costs.

snippet
+-----------------------------------------------------------------------------------+
|                        PARQUET FILE ANATOMY (DISK LAYOUT)                         |
+-----------------------------------------------------------------------------------+
| 4-byte Magic Header: 'PAR1'                                                       |
+-----------------------------------------------------------------------------------+
| ROW GROUP 0 (~128MB - 512MB)                                                      |
|   - Column Chunk 0 (user_id): [Dictionary Page] -> [Data Page 0] -> [Data Page 1] |
|   - Column Chunk 1 (timestamp): [Data Page 0 (Delta Encoded)]                     |
|   - Column Chunk 2 (event_type): [Data Page 0 (RLE/Bit-Packed)]                   |
+-----------------------------------------------------------------------------------+
| ROW GROUP 1 (~128MB - 512MB)                                                      |
|   - Column Chunk 0 (user_id): ...                                                 |
|   - Column Chunk 1 (timestamp): ...                                               |
+-----------------------------------------------------------------------------------+
| FILE METADATA FOOTER (Self-describing Schema & Statistics)                        |
|   - File schema tree & column types                                               |
|   - Row Group 0 offsets & stats: min/max, null_count, distinct_count              |
|   - Row Group 1 offsets & stats: min/max, null_count, distinct_count              |
|   - Split-Block Bloom Filters (optional high-entropy filters)                     |
+-----------------------------------------------------------------------------------+
| 4-byte Footer Length (uint32)                                                     |
+-----------------------------------------------------------------------------------+
| 4-byte Magic Footer: 'PAR1'                                                       |
+-----------------------------------------------------------------------------------+

1. The Mechanics of Columnar Storage: Why Row-Oriented Layouts Fail at Scale

To understand why Apache Parquet dominates analytical workloads (OLAP), consider how data is laid out on physical storage media and how modern CPU hardware interacts with memory.

Row-Oriented Storage (CSV, JSON, Traditional RDBMS)

In a row-oriented file format (like CSV, JSON lines, or standard relational database heaps in PostgreSQL), all attributes belonging to a single record are stored contiguously on disk:

snippet
Row 1: [ID: 101, Timestamp: 1712000100, Country: "US", Amount: 42.50, UserAgent: "Mozilla..."]
Row 2: [ID: 102, Timestamp: 1712000102, Country: "CA", Amount: 19.99, UserAgent: "Chrome..."]
Row 3: [ID: 103, Timestamp: 1712000105, Country: "US", Amount: 110.00, UserAgent: "Safari..."]

This layout is ideal for transactional workloads (OLTP). When a web service looks up a specific customer by ID or inserts a single order, the storage engine reads or writes a single contiguous block of bytes. If you are designing OLTP backends with tools like PostgreSQL Guide for Developers or Drizzle ORM Guide, row storage is the natural paradigm.

However, analytical queries rarely access every column in a record. An analytical query typically computes aggregate metrics across millions of rows for only two or three attributes:

sql
SELECT Country, SUM(Amount) 
FROM transactions 
WHERE Timestamp >= 1712000000 
GROUP BY Country;

In a row-oriented layout, the query engine must scan the entire file from start to finish. To access Country and Amount, it must also load UserAgent, IPAddress, BillingDetails, and every other contiguous byte into memory, saturating disk I/O channels and polluting the CPU cache with discarded data.

Columnar Storage (Apache Parquet)

In a columnar format, all values for a given column are stored contiguously in memory and on disk:

snippet
Column ID:        [101, 102, 103]
Column Timestamp: [1712000100, 1712000102, 1712000105]
Column Country:   ["US", "CA", "US"]
Column Amount:    [42.50, 19.99, 110.00]
Column UserAgent: ["Mozilla...", "Chrome...", "Safari..."]

This structural inversion unlocks three distinct architectural advantages:

  1. Projection Pushdown (Extreme I/O Reduction): If a table has 80 columns but your SQL query only selects 3, the execution engine reads only the byte ranges corresponding to those 3 columns. The remaining 77 columns are never transferred over the network or read from disk.
  2. Superior Compression Ratios: Homogeneous data compresses drastically better than heterogeneous data. A contiguous sequence of 64-bit integers or ISO country codes has high mathematical predictability, allowing compression algorithms to achieve 70% to 90% reductions in size.
  3. CPU Vectorization (SIMD Execution): Modern processors support Single Instruction, Multiple Data (SIMD) vector instructions (AVX-512, NEON). Because column values exist as contiguous arrays of identical data types, CPU registers can evaluate filters and aggregates across 8 or 16 numbers simultaneously in a single clock cycle without branching hazards.
5 rows · Click column to sort
I/O OverheadReads 100% of bytes for all columnsReads only requested columns80% - 95% reduction in disk / network I/O
Compression Ratio2:1 to 3:1 (Generic gzip)6:1 to 15:1 (ZSTD / Snappy + Dictionary)Up to 85% storage cost savings on S3/GCS
CPU Cache EfficiencyPoor (L1/L2 cache filled with discarded attributes)Optimal (Contiguous arrays match CPU cache lines)Up to 10x faster aggregation loops
Schema EnforcementWeak or None (Parsing errors at runtime)Strict, self-describing binary metadataZero parse failures, type safety guaranteed
Write LatencyVery fast (Append row to end of file)Moderate (Requires buffering & encoding row groups)Parquet is optimized for analytical reads, not streaming appends

2. The Internal Anatomy of a Parquet File

A Parquet file is a self-describing binary file structured hierarchically into Row Groups, Column Chunks, and Pages.

snippet
+-----------------------------------------------------------------------------------+
| 4-byte Magic: 'PAR1'                                                              |
+-----------------------------------------------------------------------------------+
| ROW GROUP 0                                                                       |
|   +-----------------------------------------------------------------------------+ |
|   | Column Chunk: 'user_id' (INT64)                                             | |
|   |   [Dictionary Page] [Data Page 0] [Data Page 1] ...                         | |
|   +-----------------------------------------------------------------------------+ |
|   +-----------------------------------------------------------------------------+ |
|   | Column Chunk: 'country' (BYTE_ARRAY / DICT)                                 | |
|   |   [Dictionary Page] [Data Page 0] ...                                       | |
|   +-----------------------------------------------------------------------------+ |
+-----------------------------------------------------------------------------------+
| ROW GROUP 1                                                                       |
|   ...                                                                             |
+-----------------------------------------------------------------------------------+
| FILE METADATA FOOTER                                                              |
|   - Version & Schema Tree                                                         |
|   - Total Row Count: 5,000,000                                                    |
|   - Row Group 0 Metadata: Byte Offset, Compressed Size, Uncompressed Size         |
|       * user_id Stats: min=1, max=2500000, null_count=0                           |
|       * country Stats: min="AE", max="ZW", null_count=12                         |
|   - Row Group 1 Metadata: Byte Offset, Compressed Size, Uncompressed Size         |
|       * user_id Stats: min=2500001, max=5000000, null_count=0                     |
|   - Key-Value User Metadata (Application-specific tags)                           |
+-----------------------------------------------------------------------------------+
| 4-byte Footer Length: uint32LE                                                    |
+-----------------------------------------------------------------------------------+
| 4-byte Magic: 'PAR1'                                                              |
+-----------------------------------------------------------------------------------+

Notice a deliberate design choice: Parquet stores its primary metadata at the end of the file (Footer), not at the beginning.

When writing a large dataset to object storage (like AWS S3 or Google Cloud Storage), you often stream data as it is generated. If metadata had to reside in the header, the writer would have to know total row counts, column byte offsets, and statistical min/max ranges before writing the data - or seek back to byte offset 0 after finishing, which object storage APIs prohibit.

By writing the metadata into the footer, Parquet allows writers to emit data pages in a single streaming pass. Once all records are serialized, the writer writes the footer metadata, records the 4-byte footer length, and closes with the 4-byte magic string PAR1.

When a query engine reads a Parquet file, it performs an efficient initialization:

  1. It issues an HTTP Range request for the final few kilobytes of the file.
  2. It reads the last 4 bytes to verify the PAR1 signature.
  3. It reads the preceding 4 bytes to determine the exact byte length N of the Thrift-encoded file metadata.
  4. It decodes the footer metadata into memory.

With that single tiny read, the query engine knows the entire table schema, total row count, data types, physical offsets of every row group, and min/max statistics for every column chunk.

Row Groups: Horizontal Partitioning Inside the File

A Parquet file partitions its total records horizontally into one or more Row Groups. A row group contains a collection of rows (typically between 500,000 and 5,000,000 rows).

Within each row group, data is split vertically into Column Chunks - one chunk per column in the schema.

Sizing Row Groups: The Memory vs Pruning Tradeoff

Selecting the appropriate row group size requires balancing read pruning efficiency against write-time memory consumption:

  • Default Row Group Size: Typically 128MB to 512MB uncompressed (or 1,000,000 rows).
  • Too Small (< 32MB): Compression ratios deteriorate because dictionary encoding cannot capture enough global patterns. Metadata overhead swells, and query engines waste time scanning thousands of tiny row group headers.
  • Too Large (> 1GB): The writer must buffer hundreds of megabytes of raw uncompressed rows in RAM across all columns before flushing to disk, triggering Out-Of-Memory (OOM) errors in Spark or DuckDB. Furthermore, coarse row groups reduce predicate pruning opportunities.

Column Chunks, Pages, and Page Headers

Inside a Column Chunk, the data is partitioned into Pages. The Page is the atomic unit of compression and encoding in Apache Parquet. A page typically targets an uncompressed size of 1MB.

There are three primary page types:

  1. Data Page (v1 / v2): Stores the actual encoded column values, accompanied by definition levels (which track null values) and repetition levels (which track nested arrays/lists).
  2. Dictionary Page: Stores the distinct unique values when dictionary encoding is activated. Data pages that follow store compact integer indexes pointing back to this dictionary.
  3. Index Page: Optional pages containing min/max bounds and Bloom filters for fine-grained page-level skipping within a column chunk.

3. Encoding Schemes and Compression Algorithms

The high storage efficiency of Parquet is achieved through a two-stage pipeline: Encoding followed by Compression.

snippet
Raw Column Values
       |
       v
+-------------------------------------------------------+
| STAGE 1: LOGICAL ENCODING                             |
| (Dictionary, RLE, Bit-Packing, Delta Binary)          |
| Goal: Eliminate mathematical redundancy               |
+-------------------------------------------------------+
       |
       v
+-------------------------------------------------------+
| STAGE 2: PHYSICAL COMPRESSION CODEC                   |
| (Zstandard, Snappy, LZ4, GZIP)                        |
| Goal: Byte-level entropy reduction                    |
+-------------------------------------------------------+
       |
       v
Serialized Physical Disk Page

Many developers mistakenly conflate encoding with compression. Encoding transforms structured values into integer representations that exploit domain patterns. Compression operates on raw byte streams to eliminate general entropy.

1. Dictionary Encoding (PLAIN_DICTIONARY / RLE_DICTIONARY)

Dictionary encoding replaces repetitive values with integer keys. For example, consider a status column containing strings like "SUCCESS", "PENDING", and "FAILED".

Instead of repeating an 8-byte ASCII string millions of times:

  • The Dictionary Page stores the unique array: ["SUCCESS", "PENDING", "FAILED"]
  • The Data Page stores integer indexes: [0, 0, 1, 0, 2, 0, 1]

Using bit-packing, 3 unique states can be represented using only 2 bits per row instead of 8 bytes (64 bits), yielding an instant 96.8% size reduction before compression is even applied.

If a column's cardinality exceeds a threshold (for instance, a unique user_uuid column with millions of distinct values), Parquet dynamically falls back to raw PLAIN encoding to prevent the dictionary page from ballooning memory.

2. Run-Length Encoding (RLE) and Bit-Packing

Run-Length Encoding compresses consecutive repeating sequences into a value-and-count pair:

snippet
Uncompressed: [0, 0, 0, 0, 0, 1, 1, 1, 2, 2]
RLE Encoded:  [(0, count: 5), (1, count: 3), (2, count: 2)]

When data is sorted by a low-cardinality column (such as date or country), RLE reduces gigabytes of rows down to a negligible footprint. Bit-packing complements RLE by packing integers into the minimal bit width required (e.g., storing integers ranging from 0 to 7 in 3 bits instead of 32 bits).

3. Delta Encoding (DELTA_BINARY_PACKED)

Delta encoding is used for sequential integers, monotonically increasing auto-increment IDs, and timestamps. Instead of storing absolute values, it stores the delta between adjacent elements:

snippet
Raw Timestamps: [1712000000, 1712000005, 1712000012, 1712000015]
Delta Encoded:  [1712000000, +5, +7, +3]

These small integer deltas require far fewer bits to represent, creating dense arrays that compress efficiently under secondary compression codecs.

4. Compression Codec Comparison: Snappy vs Zstandard (ZSTD) vs GZIP

Once pages are encoded, they are processed through a block compression codec.

Click any column to sort
4 rows
Zstandard (ZSTD)Exceptional (10:1)Moderate (Configurable 1-22)Very Fast (1,200+ MB/s)Gold standard for data lakes & long-term storage
SnappyModerate (4:1)Ultra Fast (350+ MB/s)Fast (900 MB/s)Legacy default for high-throughput batch pipelines
LZ4Moderate (4:1)Extremely FastFastest (2,000+ MB/s)Ephemeral scratch storage and intermediate shuffle
GZIPHigh (8:1)Very SlowSlow (300 MB/s)Avoid for modern analytical workloads

Architectural Recommendation: Choose Zstandard (ZSTD) at compression level 3 as your default codec. ZSTD provides superior compression ratios compared to Snappy (typically saving an extra 20 - 35% of storage space) while matching or exceeding Snappy's decompression throughput during analytical scans.


4. Query Optimization: Predicate Pushdown and Row Group Pruning

The true power of Apache Parquet in engines like DuckDB, Trino, and Apache Spark is its ability to skip data without reading it.

How Predicate Pushdown Works

When you execute a query with a WHERE clause:

sql
SELECT customer_id, total_price 
FROM orders 
WHERE order_date >= '2026-06-01';

A naive database loads the entire table and evaluates the condition row by row. Parquet engines, however, push the filter predicate directly down into the file reader.

snippet
Incoming Filter: order_date >= '2026-06-01'
                          |
                          v
         +----------------------------------+
         | PARQUET FOOTER METADATA SCAN     |
         +----------------------------------+
                          |
       +------------------+------------------+
       |                                     |
       v                                     v
+-------------------------------+ +-------------------------------+
| ROW GROUP 0                   | | ROW GROUP 1                   |
| order_date Stats:             | | order_date Stats:             |
|   min = '2026-01-01'          | |   min = '2026-05-15'          |
|   max = '2026-03-31'          | |   max = '2026-07-30'          |
+-------------------------------+ +-------------------------------+
       |                                     |
       v                                     v
Condition: max < '2026-06-01'     Condition: max >= '2026-06-01'
DECISION: SKIP ENTIRE ROW GROUP!  DECISION: READ ROW GROUP 1
DISK BYTES READ: 0 BYTES          DISK BYTES READ: Only relevant chunks

Before issuing read commands for Row Group 0, the engine inspects its footer statistics. Because the maximum order_date in Row Group 0 is '2026-03-31', the engine mathematically proves that no row in that group can satisfy the condition. Row Group 0 is skipped entirely - no disk seek, no network transfer, and no decompression.

The Power of Data Sorting on Pruning

Predicate pushdown performance depends heavily on the physical sort order of your data during ingestion.

If rows are inserted in random order, every row group will have a broad min/max range (e.g., min = '2026-01-01', max = '2026-12-31'). In that scenario, min/max statistics cannot prune any row groups, forcing the engine to scan the entire dataset.

By sorting your dataset by primary query filters (e.g., ORDER BY event_date, tenant_id) prior to writing Parquet files, you establish tight, disjoint min/max bounds across row groups. This allows query engines to skip 90% or more of row groups on targeted queries.

Split-Block Bloom Filters

What happens when you filter by a high-cardinality, non-sequential field, such as a random UUID or email?

Min/max statistics are useless here: the minimum UUID in every row group starts with 0000... and the maximum starts with ffff....

To solve this, Parquet supports Split-Block Bloom Filters. A Bloom filter is a space-efficient probabilistic data structure stored alongside column chunk metadata. When querying for WHERE user_uuid = 'e3b0c44...', the query engine checks the row group's Bloom filter. If the Bloom filter returns false, the engine knows with 100% certainty that the UUID does not exist in that row group, skipping it entirely.


5. Partitioning Strategies and the "Small Files Problem"

While Parquet provides intra-file optimization through row groups, data lakes organize collections of Parquet files using directory partitioning schemes.

Hive-Style Directory Partitioning

The standard convention for data lake partitioning is Hive-style directory hierarchies:

snippet
s3://analytics-lake/events/
    ├── year=2026/
    │   ├── month=04/
    │   │   ├── day=01/
    │   │   │   ├── part-0001.parquet
    │   │   │   └── part-0002.parquet
    │   │   └── day=02/
    │   │       └── part-0001.parquet

When a SQL query includes WHERE year = 2026 AND month = 4 AND day = 1, the query engine performs Partition Pruning. It lists and reads only the files under day=01/, bypassing the thousands of other subdirectories. For guidance on structuring analytics pipelines and queries over large partitioned datasets, see SQL for Data Scientists: Window Functions, CTEs, and Query Optimization.

The "Small Files Problem" Anti-Pattern

Directory partitioning can backfire when applied too granularly. A common failure mode is partitioning across too many dimensions (e.g., /year/month/day/hour/tenant_id/country/).

When you over-partition, your data pipeline produces millions of tiny Parquet files (e.g., 2MB to 10MB each):

  1. Metadata Overhead: Cloud object storage systems (AWS S3) throttle requests when query engines execute hundreds of thousands of LIST and GET calls across tiny files.
  2. Loss of Columnar Efficiency: A 5MB Parquet file cannot build effective dictionaries or amortize footer metadata costs.
  3. Memory Exhaustion: Query engines allocate thread pools and connection overhead for every individual file, leading to Out-Of-Memory crashes.
snippet
+-----------------------------------------------------------------------------------+
|                        COMPACTION: THE RE-BINNING PATTERN                         |
+-----------------------------------------------------------------------------------+
| RAW INGESTION (Uncompacted Stream)                                                |
|   1,000 files x 2MB = 2GB total storage                                           |
|   Query Latency: 14.8 seconds (S3 metadata listing & connection thrashing)        |
+-----------------------------------------------------------------------------------+
                                         |
                                         v  Background Compaction Job (Daily)
+-----------------------------------------------------------------------------------+
| COMPACTED TARGET STATE                                                            |
|   8 files x 250MB = 2GB total storage                                             |
|   Query Latency: 0.6 seconds (Optimal HTTP range reads & vectorized scans)        |
+-----------------------------------------------------------------------------------+

Target File Size Rule: Target individual Parquet files between 128MB and 512MB. If streaming pipelines emit small files throughout the day, run a scheduled compaction task to merge them into consolidated 256MB Parquet files.


6. Hands-On Engineering: PyArrow, Polars, and DuckDB

Let's explore how to inspect internal Parquet metadata, write optimized files with PyArrow, and query them at high speed using Polars and DuckDB.

Inspecting Parquet Low-Level Metadata with PyArrow

You can inspect the row groups, column chunk encodings, and physical byte allocations of any Parquet file using pyarrow.parquet:

python
# inspect_parquet.py
import pyarrow.parquet as pq

def inspect_parquet_internals(file_path: str):
    parquet_file = pq.ParquetFile(file_path)
    metadata = parquet_file.metadata
    
    print(f"File: {file_path}")
    print(f"Format Version: {metadata.format_version}")
    print(f"Total Rows: {metadata.num_rows:,}")
    print(f"Total Row Groups: {metadata.num_row_groups}")
    print(f"Total Columns: {metadata.num_columns}")
    
    for rg_idx in range(metadata.num_row_groups):
        rg = metadata.row_group(rg_idx)
        print(f"\n--- Row Group {rg_idx} ({rg.num_rows:,} rows, {rg.total_byte_size / (1024*1024):.2f} MB uncompressed) ---")
        
        for col_idx in range(metadata.num_columns):
            col = rg.column(col_idx)
            stats = col.statistics
            print(f"  Col '{col.path_in_schema}':")
            print(f"    Type: {col.physical_type}, Compression: {col.compression}")
            print(f"    Encodings: {col.encodings}")
            print(f"    Compressed Size: {col.total_compressed_size:,} bytes")
            if stats:
                print(f"    Min: {stats.min}, Max: {stats.max}, Nulls: {stats.null_count}")

if __name__ == "__main__":
    inspect_parquet_internals("telemetry_events.parquet")

Writing Optimized Parquet with PyArrow

When writing Parquet files in data pipelines, configure your compression codecs, row group limits, and dictionary thresholds deliberately:

python
# write_optimized_parquet.py
import pyarrow as pa
import pyarrow.parquet as pq
import pandas as pd
import numpy as np

# Generate sample analytical dataset
np.random.seed(42)
n_rows = 2_000_000

df = pd.DataFrame({
    "event_id": np.arange(n_rows, dtype=np.int64),
    "user_uuid": [f"usr_{i % 50000:08x}" for i in range(n_rows)],
    "region": np.random.choice(["us-east-1", "us-west-2", "eu-west-1", "ap-southeast-1"], size=n_rows),
    "latency_ms": np.random.exponential(scale=45.0, size=n_rows).astype(np.float32),
    "status_code": np.random.choice([200, 201, 400, 404, 500], p=[0.90, 0.05, 0.02, 0.02, 0.01], size=n_rows),
})

# Sort by high-frequency query filter columns to maximize min/max pruning!
df = df.sort_values(by=["region", "status_code"]).reset_index(drop=True)

# Convert to Arrow Table
table = pa.Table.from_pandas(df)

# Write with Zstandard compression and controlled row group sizing
pq.write_table(
    table,
    "optimized_telemetry.parquet",
    compression="ZSTD",
    compression_level=4,
    row_group_size=500_000, # 500k rows per row group for targeted pruning
    use_dictionary=True,     # Activates dictionary encoding on 'region'
    write_statistics=True    # Emits min/max statistics into the footer
)
print("Successfully generated optimized Parquet file.")

If you are using Pandas for data manipulation, see Pandas for Software Developers: The Complete Guide to Data Manipulation in Python to master memory management and Arrow-backed dtypes.

Blazing Fast Querying with Polars and DuckDB

Modern analytical tools can query Parquet files without loading the entire dataset into memory.

1. Polars Lazy Scanning (Rust-Powered Vectorized Engine)

python
# polars_query.py
import polars as pl

# scan_parquet reads only metadata, building a lazy execution graph
lazy_query = (
    pl.scan_parquet("optimized_telemetry.parquet")
    .filter(pl.col("region") == "us-east-1")
    .filter(pl.col("status_code") >= 500)
    .select(["user_uuid", "latency_ms"])
)

# Explain prints the physical plan, proving predicate & projection pushdown
print(lazy_query.explain())

# Execute the query efficiently with SIMD vectorization
result = lazy_query.collect()
print(result.head())

2. DuckDB SQL Directly on Parquet Files

DuckDB allows you to execute complete analytical SQL queries directly against local files or S3 objects without launching a dedicated database server:

python
# duckdb_query.py
import duckdb

conn = duckdb.connect()

# Query directly from the Parquet file path
query = """
    SELECT 
        region,
        COUNT(*) AS total_errors,
        AVG(latency_ms) AS avg_error_latency
    FROM 'optimized_telemetry.parquet'
    WHERE status_code >= 400
    GROUP BY region
    ORDER BY total_errors DESC;
"""

df_summary = conn.execute(query).df()
print(df_summary)

For teams presenting operational metrics and analytical findings in internal dashboards, see Data Visualization in Python: When to Use Matplotlib, Seaborn, Plotly, and Altair.


7. Architectural Comparison: Parquet vs Avro vs ORC vs JSON/CSV

Choosing the right file format requires matching the format's physical strengths to your workload characteristics:

7 rows · Click column to sort
OrientationColumnarRow-orientedColumnarRow-oriented
Compression EfficiencyHigh to Extreme (ZSTD/Snappy)Moderate (Deflate/Snappy)High to Extreme (ZLIB/Snappy)Low
Schema EvolutionRobust (Add/Remove optional fields)Best-in-class (JSON Schema + Union types)RobustFragile (Breaks on column order changes)
Write ThroughputModerate (Buffers row groups in RAM)Very Fast (Streaming binary appends)ModerateFast
Analytical AggregationsFastest (Vectorized, SIMD, Pruning)Slow (Scans full rows)Very Fast (Optimized for Hive/Presto)Slowest
Streaming / Message QueuesPoor (Not designed for single-event appends)Industry Standard (Kafka payload standard)PoorCommon (JSON events)
Primary EcosystemSpark, DuckDB, Polars, Snowflake, IcebergKafka, Event Streams, RPC schemasHive, Trino, Presto, HadoopWeb APIs, Log drains, Human inspection

Summary Decision Matrix

  • Choose Apache Parquet for analytical querying (OLAP), data lakes, data warehousing, BI aggregation, and feature storage in machine learning pipelines.
  • Choose Apache Avro for event streaming platforms (Apache Kafka), transactional message buses, and row-oriented ingestion where schemas evolve continuously.
  • Choose CSV / JSON Lines only for small configuration exports or human-readable diagnostics where binary formats are impractical.

8. Parquet as the Foundation for Modern Table Formats

Individual Parquet files are immutable. You cannot modify a single row inside a Parquet file in place - to alter one record, you must rewrite the entire row group or the entire file.

To support ACID transactions, updates, deletes, and time-travel querying on top of immutable Parquet files, the data engineering industry created Open Table Formats:

  • Apache Iceberg
  • Delta Lake
  • Apache Hudi
snippet
+-----------------------------------------------------------------------------------+
| TABLE FORMAT METADATA LAYER (Apache Iceberg / Delta Lake)                         |
| - ACID Transaction Logs (AOM Commit Log)                                          |
| - Snapshot Manifests (Track which files belong to Snapshot v4)                    |
| - File-Level Partition Bounds & Column Statistics                                 |
+-----------------------------------------------------------------------------------+
                                         |
                                         v
+-----------------------------------------------------------------------------------+
| PHYSICAL STORAGE LAYER (Apache Parquet)                                           |
| s3://lake/data/001.parquet  s3://lake/data/002.parquet  s3://lake/data/003.parquet|
| (Immutable, ZSTD-compressed, dictionary-encoded columnar files)                   |
+-----------------------------------------------------------------------------------+

Table formats do not replace Parquet; they serve as a metadata management layer operating directly on top of immutable Parquet files. When an UPDATE or DELETE statement executes on an Iceberg or Delta table:

  1. The table format writes new Parquet data files containing the updated rows or positional delete files.
  2. It commits an atomic snapshot metadata file updating the pointer to the active set of Parquet files.
  3. Readers querying the snapshot continue reading immutable Parquet files with zero locks and complete snapshot isolation.

By decoupling the file format (Parquet) from the transaction management layer (Iceberg / Delta), modern data platforms achieve both blistering analytical scan performance and strict database consistency.


Conclusion and Engineering Best Practices

Apache Parquet is one of the most effective tools for reducing data lake storage costs while improving analytical query speeds. To get the most out of Parquet in production:

  1. Default to Zstandard (ZSTD): Use ZSTD compression level 3 or 4 to achieve 25% better compression ratios than Snappy without sacrificing decompression throughput.
  2. Sort Data Prior to Writing: Order your tables by high-frequency query filter columns before flushing to disk. Tight min/max bounds allow predicate pushdown to eliminate 80%+ of row groups.
  3. Control Row Group Sizing: Maintain row groups between 128MB and 512MB (500,000 to 2,000,000 rows) to balance memory footprint during writes with pruning efficiency during reads.
  4. Solve the Small Files Problem: Ensure individual files fall in the 128MB - 512MB range. Run background compaction jobs on streaming pipelines to prevent S3 metadata throttling.
  5. Leverage Pushdown-Aware Engines: Use engines like DuckDB, Polars, and Trino that take full advantage of Parquet's projection and predicate pushdown capabilities.

Frequently Asked Questions

Why is Parquet metadata located at the end of the file rather than the beginning?

Parquet stores file metadata in the footer to enable efficient single-pass streaming writes. When generating data files for object storage systems (like AWS S3), the writer cannot seek back to the beginning to update header statistics once row counts and byte offsets are finalized. Placing metadata at the end allows the writer to stream data pages sequentially and append the completed schema, row group offsets, and min/max statistics in a single final write.

What is the difference between predicate pushdown and partition pruning?

Partition pruning operates at the file system or directory level (e.g., Hive-style /year=2026/month=04/ directories), skipping entire files or folders before they are opened. Predicate pushdown operates inside individual Parquet files: the query engine reads the footer metadata and uses column statistics (min/max bounds and Bloom filters) to skip entire row groups within the file without decompressing or reading their data pages.

Which compression codec should you choose for Parquet: Snappy or Zstandard (ZSTD)?

For almost all modern analytical workloads, Zstandard (ZSTD) at compression level 3 is the recommended default. ZSTD typically achieves 20% to 35% higher compression ratios than Snappy, dramatically reducing cloud storage costs and network transfer overhead, while matching or exceeding Snappy's decompression throughput during analytical queries.

What is the ideal row group size for Apache Parquet files?

The recommended uncompressed row group size is typically between 128MB and 512MB (representing approximately 500,000 to 2,000,000 rows). Setting row groups smaller than 32MB swells metadata overhead and ruins dictionary compression efficiency, whereas setting them larger than 1GB increases write-time memory buffers and risks Out-Of-Memory (OOM) errors.

How does Parquet handle schema evolution when adding or renaming columns?

Parquet natively supports backward-compatible schema evolution. Adding a new optional column allows legacy readers to read existing files while treating the missing column as NULL. Renaming columns requires careful consideration: if readers resolve columns by name (field names), renaming breaks compatibility unless the engine supports field IDs (as enabled in Apache Iceberg and Parquet format v2).

Why shouldn't Parquet be used for OLTP transactional database systems?

Parquet files are physically immutable and organized by column. In transactional OLTP systems, workloads involve high-frequency single-row inserts, updates, and primary key lookups. Modifying or appending a single row in Parquet would require rewriting entire row groups, buffering column arrays in memory, and recomputing footer metadata, creating massive write amplification and unacceptable latency.

#parquet#data-engineering#columnar-storage#compression#duckdb#pyarrow#big-data#sql

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns, delivered to your inbox.

What should we test or explain next?

Comments

Have a question or something to add? Sign in to join the discussion.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

Software architect, full-stack engineer, and machine learning practitioner. Founder & CEO of Pristren, CTO at Venture Kick-backed PIN Sport Application GmbH (Basel, Switzerland), Software Development Lead at DailyFrame GmbH (building Dimetra patient transport software), and creator of Zlyqor. Focused on custom workflow automation, scalable web and mobile applications, and production AI/ML systems.