In modern data engineering and analytical architectures, file format selection is one of the most critical structural decisions you will make. Storing large-scale analytical datasets in row-oriented text formats like CSV or newline-delimited JSON introduces massive performance penalties: excessive disk utilization, CPU-intensive text parsing, and catastrophic I/O waste when queries scan gigabytes of irrelevant fields.
Apache Parquet solves these architectural bottlenecks. Created through a collaboration between Twitter and Cloudera, Parquet is an open-source, column-oriented data file format designed specifically for complex analytical queries. By organizing records by column rather than by row, Parquet enables extreme compression ratios, efficient encoding schemes, and aggressive query pruning via predicate pushdown.
Whether you are querying data lakes with Trino and Apache Spark, running local in-process analytics with DuckDB and Polars, or structuring open table formats like Apache Iceberg and Delta Lake, understanding the low-level physical anatomy of Apache Parquet is essential for optimizing query latency and cloud storage costs.
+-----------------------------------------------------------------------------------+
| PARQUET FILE ANATOMY (DISK LAYOUT) |
+-----------------------------------------------------------------------------------+
| 4-byte Magic Header: 'PAR1' |
+-----------------------------------------------------------------------------------+
| ROW GROUP 0 (~128MB - 512MB) |
| - Column Chunk 0 (user_id): [Dictionary Page] -> [Data Page 0] -> [Data Page 1] |
| - Column Chunk 1 (timestamp): [Data Page 0 (Delta Encoded)] |
| - Column Chunk 2 (event_type): [Data Page 0 (RLE/Bit-Packed)] |
+-----------------------------------------------------------------------------------+
| ROW GROUP 1 (~128MB - 512MB) |
| - Column Chunk 0 (user_id): ... |
| - Column Chunk 1 (timestamp): ... |
+-----------------------------------------------------------------------------------+
| FILE METADATA FOOTER (Self-describing Schema & Statistics) |
| - File schema tree & column types |
| - Row Group 0 offsets & stats: min/max, null_count, distinct_count |
| - Row Group 1 offsets & stats: min/max, null_count, distinct_count |
| - Split-Block Bloom Filters (optional high-entropy filters) |
+-----------------------------------------------------------------------------------+
| 4-byte Footer Length (uint32) |
+-----------------------------------------------------------------------------------+
| 4-byte Magic Footer: 'PAR1' |
+-----------------------------------------------------------------------------------+
1. The Mechanics of Columnar Storage: Why Row-Oriented Layouts Fail at Scale
To understand why Apache Parquet dominates analytical workloads (OLAP), consider how data is laid out on physical storage media and how modern CPU hardware interacts with memory.
Row-Oriented Storage (CSV, JSON, Traditional RDBMS)
In a row-oriented file format (like CSV, JSON lines, or standard relational database heaps in PostgreSQL), all attributes belonging to a single record are stored contiguously on disk:
Row 1: [ID: 101, Timestamp: 1712000100, Country: "US", Amount: 42.50, UserAgent: "Mozilla..."]
Row 2: [ID: 102, Timestamp: 1712000102, Country: "CA", Amount: 19.99, UserAgent: "Chrome..."]
Row 3: [ID: 103, Timestamp: 1712000105, Country: "US", Amount: 110.00, UserAgent: "Safari..."]
This layout is ideal for transactional workloads (OLTP). When a web service looks up a specific customer by ID or inserts a single order, the storage engine reads or writes a single contiguous block of bytes. If you are designing OLTP backends with tools like PostgreSQL Guide for Developers or Drizzle ORM Guide, row storage is the natural paradigm.
However, analytical queries rarely access every column in a record. An analytical query typically computes aggregate metrics across millions of rows for only two or three attributes:
SELECT Country, SUM(Amount)
FROM transactions
WHERE Timestamp >= 1712000000
GROUP BY Country;In a row-oriented layout, the query engine must scan the entire file from start to finish. To access Country and Amount, it must also load UserAgent, IPAddress, BillingDetails, and every other contiguous byte into memory, saturating disk I/O channels and polluting the CPU cache with discarded data.
Columnar Storage (Apache Parquet)
In a columnar format, all values for a given column are stored contiguously in memory and on disk:
Column ID: [101, 102, 103]
Column Timestamp: [1712000100, 1712000102, 1712000105]
Column Country: ["US", "CA", "US"]
Column Amount: [42.50, 19.99, 110.00]
Column UserAgent: ["Mozilla...", "Chrome...", "Safari..."]
This structural inversion unlocks three distinct architectural advantages:
- Projection Pushdown (Extreme I/O Reduction): If a table has 80 columns but your SQL query only selects 3, the execution engine reads only the byte ranges corresponding to those 3 columns. The remaining 77 columns are never transferred over the network or read from disk.
- Superior Compression Ratios: Homogeneous data compresses drastically better than heterogeneous data. A contiguous sequence of 64-bit integers or ISO country codes has high mathematical predictability, allowing compression algorithms to achieve 70% to 90% reductions in size.
- CPU Vectorization (SIMD Execution): Modern processors support Single Instruction, Multiple Data (SIMD) vector instructions (AVX-512, NEON). Because column values exist as contiguous arrays of identical data types, CPU registers can evaluate filters and aggregates across 8 or 16 numbers simultaneously in a single clock cycle without branching hazards.
| I/O Overhead | Reads 100% of bytes for all columns | Reads only requested columns | 80% - 95% reduction in disk / network I/O |
| Compression Ratio | 2:1 to 3:1 (Generic gzip) | 6:1 to 15:1 (ZSTD / Snappy + Dictionary) | Up to 85% storage cost savings on S3/GCS |
| CPU Cache Efficiency | Poor (L1/L2 cache filled with discarded attributes) | Optimal (Contiguous arrays match CPU cache lines) | Up to 10x faster aggregation loops |
| Schema Enforcement | Weak or None (Parsing errors at runtime) | Strict, self-describing binary metadata | Zero parse failures, type safety guaranteed |
| Write Latency | Very fast (Append row to end of file) | Moderate (Requires buffering & encoding row groups) | Parquet is optimized for analytical reads, not streaming appends |
2. The Internal Anatomy of a Parquet File
A Parquet file is a self-describing binary file structured hierarchically into Row Groups, Column Chunks, and Pages.
+-----------------------------------------------------------------------------------+
| 4-byte Magic: 'PAR1' |
+-----------------------------------------------------------------------------------+
| ROW GROUP 0 |
| +-----------------------------------------------------------------------------+ |
| | Column Chunk: 'user_id' (INT64) | |
| | [Dictionary Page] [Data Page 0] [Data Page 1] ... | |
| +-----------------------------------------------------------------------------+ |
| +-----------------------------------------------------------------------------+ |
| | Column Chunk: 'country' (BYTE_ARRAY / DICT) | |
| | [Dictionary Page] [Data Page 0] ... | |
| +-----------------------------------------------------------------------------+ |
+-----------------------------------------------------------------------------------+
| ROW GROUP 1 |
| ... |
+-----------------------------------------------------------------------------------+
| FILE METADATA FOOTER |
| - Version & Schema Tree |
| - Total Row Count: 5,000,000 |
| - Row Group 0 Metadata: Byte Offset, Compressed Size, Uncompressed Size |
| * user_id Stats: min=1, max=2500000, null_count=0 |
| * country Stats: min="AE", max="ZW", null_count=12 |
| - Row Group 1 Metadata: Byte Offset, Compressed Size, Uncompressed Size |
| * user_id Stats: min=2500001, max=5000000, null_count=0 |
| - Key-Value User Metadata (Application-specific tags) |
+-----------------------------------------------------------------------------------+
| 4-byte Footer Length: uint32LE |
+-----------------------------------------------------------------------------------+
| 4-byte Magic: 'PAR1' |
+-----------------------------------------------------------------------------------+
The File Header and Footer Architecture
Notice a deliberate design choice: Parquet stores its primary metadata at the end of the file (Footer), not at the beginning.
When writing a large dataset to object storage (like AWS S3 or Google Cloud Storage), you often stream data as it is generated. If metadata had to reside in the header, the writer would have to know total row counts, column byte offsets, and statistical min/max ranges before writing the data - or seek back to byte offset 0 after finishing, which object storage APIs prohibit.
By writing the metadata into the footer, Parquet allows writers to emit data pages in a single streaming pass. Once all records are serialized, the writer writes the footer metadata, records the 4-byte footer length, and closes with the 4-byte magic string PAR1.
When a query engine reads a Parquet file, it performs an efficient initialization:
- It issues an HTTP Range request for the final few kilobytes of the file.
- It reads the last 4 bytes to verify the
PAR1signature. - It reads the preceding 4 bytes to determine the exact byte length
Nof the Thrift-encoded file metadata. - It decodes the footer metadata into memory.
With that single tiny read, the query engine knows the entire table schema, total row count, data types, physical offsets of every row group, and min/max statistics for every column chunk.
Row Groups: Horizontal Partitioning Inside the File
A Parquet file partitions its total records horizontally into one or more Row Groups. A row group contains a collection of rows (typically between 500,000 and 5,000,000 rows).
Within each row group, data is split vertically into Column Chunks - one chunk per column in the schema.
Sizing Row Groups: The Memory vs Pruning Tradeoff
Selecting the appropriate row group size requires balancing read pruning efficiency against write-time memory consumption:
- Default Row Group Size: Typically 128MB to 512MB uncompressed (or 1,000,000 rows).
- Too Small (< 32MB): Compression ratios deteriorate because dictionary encoding cannot capture enough global patterns. Metadata overhead swells, and query engines waste time scanning thousands of tiny row group headers.
- Too Large (> 1GB): The writer must buffer hundreds of megabytes of raw uncompressed rows in RAM across all columns before flushing to disk, triggering Out-Of-Memory (OOM) errors in Spark or DuckDB. Furthermore, coarse row groups reduce predicate pruning opportunities.
Column Chunks, Pages, and Page Headers
Inside a Column Chunk, the data is partitioned into Pages. The Page is the atomic unit of compression and encoding in Apache Parquet. A page typically targets an uncompressed size of 1MB.
There are three primary page types:
- Data Page (v1 / v2): Stores the actual encoded column values, accompanied by definition levels (which track null values) and repetition levels (which track nested arrays/lists).
- Dictionary Page: Stores the distinct unique values when dictionary encoding is activated. Data pages that follow store compact integer indexes pointing back to this dictionary.
- Index Page: Optional pages containing min/max bounds and Bloom filters for fine-grained page-level skipping within a column chunk.
3. Encoding Schemes and Compression Algorithms
The high storage efficiency of Parquet is achieved through a two-stage pipeline: Encoding followed by Compression.
Raw Column Values
|
v
+-------------------------------------------------------+
| STAGE 1: LOGICAL ENCODING |
| (Dictionary, RLE, Bit-Packing, Delta Binary) |
| Goal: Eliminate mathematical redundancy |
+-------------------------------------------------------+
|
v
+-------------------------------------------------------+
| STAGE 2: PHYSICAL COMPRESSION CODEC |
| (Zstandard, Snappy, LZ4, GZIP) |
| Goal: Byte-level entropy reduction |
+-------------------------------------------------------+
|
v
Serialized Physical Disk Page
Many developers mistakenly conflate encoding with compression. Encoding transforms structured values into integer representations that exploit domain patterns. Compression operates on raw byte streams to eliminate general entropy.
1. Dictionary Encoding (PLAIN_DICTIONARY / RLE_DICTIONARY)
Dictionary encoding replaces repetitive values with integer keys. For example, consider a status column containing strings like "SUCCESS", "PENDING", and "FAILED".
Instead of repeating an 8-byte ASCII string millions of times:
- The Dictionary Page stores the unique array:
["SUCCESS", "PENDING", "FAILED"] - The Data Page stores integer indexes:
[0, 0, 1, 0, 2, 0, 1]
Using bit-packing, 3 unique states can be represented using only 2 bits per row instead of 8 bytes (64 bits), yielding an instant 96.8% size reduction before compression is even applied.
If a column's cardinality exceeds a threshold (for instance, a unique user_uuid column with millions of distinct values), Parquet dynamically falls back to raw PLAIN encoding to prevent the dictionary page from ballooning memory.
2. Run-Length Encoding (RLE) and Bit-Packing
Run-Length Encoding compresses consecutive repeating sequences into a value-and-count pair:
Uncompressed: [0, 0, 0, 0, 0, 1, 1, 1, 2, 2]
RLE Encoded: [(0, count: 5), (1, count: 3), (2, count: 2)]
When data is sorted by a low-cardinality column (such as date or country), RLE reduces gigabytes of rows down to a negligible footprint. Bit-packing complements RLE by packing integers into the minimal bit width required (e.g., storing integers ranging from 0 to 7 in 3 bits instead of 32 bits).
3. Delta Encoding (DELTA_BINARY_PACKED)
Delta encoding is used for sequential integers, monotonically increasing auto-increment IDs, and timestamps. Instead of storing absolute values, it stores the delta between adjacent elements:
Raw Timestamps: [1712000000, 1712000005, 1712000012, 1712000015]
Delta Encoded: [1712000000, +5, +7, +3]
These small integer deltas require far fewer bits to represent, creating dense arrays that compress efficiently under secondary compression codecs.
4. Compression Codec Comparison: Snappy vs Zstandard (ZSTD) vs GZIP
Once pages are encoded, they are processed through a block compression codec.
| Zstandard (ZSTD) | Exceptional (10:1) | Moderate (Configurable 1-22) | Very Fast (1,200+ MB/s) | Gold standard for data lakes & long-term storage |
| Snappy | Moderate (4:1) | Ultra Fast (350+ MB/s) | Fast (900 MB/s) | Legacy default for high-throughput batch pipelines |
| LZ4 | Moderate (4:1) | Extremely Fast | Fastest (2,000+ MB/s) | Ephemeral scratch storage and intermediate shuffle |
| GZIP | High (8:1) | Very Slow | Slow (300 MB/s) | Avoid for modern analytical workloads |
Architectural Recommendation: Choose Zstandard (ZSTD) at compression level 3 as your default codec. ZSTD provides superior compression ratios compared to Snappy (typically saving an extra 20 - 35% of storage space) while matching or exceeding Snappy's decompression throughput during analytical scans.
4. Query Optimization: Predicate Pushdown and Row Group Pruning
The true power of Apache Parquet in engines like DuckDB, Trino, and Apache Spark is its ability to skip data without reading it.
How Predicate Pushdown Works
When you execute a query with a WHERE clause:
SELECT customer_id, total_price
FROM orders
WHERE order_date >= '2026-06-01';A naive database loads the entire table and evaluates the condition row by row. Parquet engines, however, push the filter predicate directly down into the file reader.
Incoming Filter: order_date >= '2026-06-01'
|
v
+----------------------------------+
| PARQUET FOOTER METADATA SCAN |
+----------------------------------+
|
+------------------+------------------+
| |
v v
+-------------------------------+ +-------------------------------+
| ROW GROUP 0 | | ROW GROUP 1 |
| order_date Stats: | | order_date Stats: |
| min = '2026-01-01' | | min = '2026-05-15' |
| max = '2026-03-31' | | max = '2026-07-30' |
+-------------------------------+ +-------------------------------+
| |
v v
Condition: max < '2026-06-01' Condition: max >= '2026-06-01'
DECISION: SKIP ENTIRE ROW GROUP! DECISION: READ ROW GROUP 1
DISK BYTES READ: 0 BYTES DISK BYTES READ: Only relevant chunks
Before issuing read commands for Row Group 0, the engine inspects its footer statistics. Because the maximum order_date in Row Group 0 is '2026-03-31', the engine mathematically proves that no row in that group can satisfy the condition. Row Group 0 is skipped entirely - no disk seek, no network transfer, and no decompression.
The Power of Data Sorting on Pruning
Predicate pushdown performance depends heavily on the physical sort order of your data during ingestion.
If rows are inserted in random order, every row group will have a broad min/max range (e.g., min = '2026-01-01', max = '2026-12-31'). In that scenario, min/max statistics cannot prune any row groups, forcing the engine to scan the entire dataset.
By sorting your dataset by primary query filters (e.g., ORDER BY event_date, tenant_id) prior to writing Parquet files, you establish tight, disjoint min/max bounds across row groups. This allows query engines to skip 90% or more of row groups on targeted queries.
Split-Block Bloom Filters
What happens when you filter by a high-cardinality, non-sequential field, such as a random UUID or email?
Min/max statistics are useless here: the minimum UUID in every row group starts with 0000... and the maximum starts with ffff....
To solve this, Parquet supports Split-Block Bloom Filters. A Bloom filter is a space-efficient probabilistic data structure stored alongside column chunk metadata. When querying for WHERE user_uuid = 'e3b0c44...', the query engine checks the row group's Bloom filter. If the Bloom filter returns false, the engine knows with 100% certainty that the UUID does not exist in that row group, skipping it entirely.
5. Partitioning Strategies and the "Small Files Problem"
While Parquet provides intra-file optimization through row groups, data lakes organize collections of Parquet files using directory partitioning schemes.
Hive-Style Directory Partitioning
The standard convention for data lake partitioning is Hive-style directory hierarchies:
s3://analytics-lake/events/
├── year=2026/
│ ├── month=04/
│ │ ├── day=01/
│ │ │ ├── part-0001.parquet
│ │ │ └── part-0002.parquet
│ │ └── day=02/
│ │ └── part-0001.parquet
When a SQL query includes WHERE year = 2026 AND month = 4 AND day = 1, the query engine performs Partition Pruning. It lists and reads only the files under day=01/, bypassing the thousands of other subdirectories. For guidance on structuring analytics pipelines and queries over large partitioned datasets, see SQL for Data Scientists: Window Functions, CTEs, and Query Optimization.
The "Small Files Problem" Anti-Pattern
Directory partitioning can backfire when applied too granularly. A common failure mode is partitioning across too many dimensions (e.g., /year/month/day/hour/tenant_id/country/).
When you over-partition, your data pipeline produces millions of tiny Parquet files (e.g., 2MB to 10MB each):
- Metadata Overhead: Cloud object storage systems (AWS S3) throttle requests when query engines execute hundreds of thousands of
LISTandGETcalls across tiny files. - Loss of Columnar Efficiency: A 5MB Parquet file cannot build effective dictionaries or amortize footer metadata costs.
- Memory Exhaustion: Query engines allocate thread pools and connection overhead for every individual file, leading to Out-Of-Memory crashes.
+-----------------------------------------------------------------------------------+
| COMPACTION: THE RE-BINNING PATTERN |
+-----------------------------------------------------------------------------------+
| RAW INGESTION (Uncompacted Stream) |
| 1,000 files x 2MB = 2GB total storage |
| Query Latency: 14.8 seconds (S3 metadata listing & connection thrashing) |
+-----------------------------------------------------------------------------------+
|
v Background Compaction Job (Daily)
+-----------------------------------------------------------------------------------+
| COMPACTED TARGET STATE |
| 8 files x 250MB = 2GB total storage |
| Query Latency: 0.6 seconds (Optimal HTTP range reads & vectorized scans) |
+-----------------------------------------------------------------------------------+
Target File Size Rule: Target individual Parquet files between 128MB and 512MB. If streaming pipelines emit small files throughout the day, run a scheduled compaction task to merge them into consolidated 256MB Parquet files.
6. Hands-On Engineering: PyArrow, Polars, and DuckDB
Let's explore how to inspect internal Parquet metadata, write optimized files with PyArrow, and query them at high speed using Polars and DuckDB.
Inspecting Parquet Low-Level Metadata with PyArrow
You can inspect the row groups, column chunk encodings, and physical byte allocations of any Parquet file using pyarrow.parquet:
# inspect_parquet.py
import pyarrow.parquet as pq
def inspect_parquet_internals(file_path: str):
parquet_file = pq.ParquetFile(file_path)
metadata = parquet_file.metadata
print(f"File: {file_path}")
print(f"Format Version: {metadata.format_version}")
print(f"Total Rows: {metadata.num_rows:,}")
print(f"Total Row Groups: {metadata.num_row_groups}")
print(f"Total Columns: {metadata.num_columns}")
for rg_idx in range(metadata.num_row_groups):
rg = metadata.row_group(rg_idx)
print(f"\n--- Row Group {rg_idx} ({rg.num_rows:,} rows, {rg.total_byte_size / (1024*1024):.2f} MB uncompressed) ---")
for col_idx in range(metadata.num_columns):
col = rg.column(col_idx)
stats = col.statistics
print(f" Col '{col.path_in_schema}':")
print(f" Type: {col.physical_type}, Compression: {col.compression}")
print(f" Encodings: {col.encodings}")
print(f" Compressed Size: {col.total_compressed_size:,} bytes")
if stats:
print(f" Min: {stats.min}, Max: {stats.max}, Nulls: {stats.null_count}")
if __name__ == "__main__":
inspect_parquet_internals("telemetry_events.parquet")Writing Optimized Parquet with PyArrow
When writing Parquet files in data pipelines, configure your compression codecs, row group limits, and dictionary thresholds deliberately:
# write_optimized_parquet.py
import pyarrow as pa
import pyarrow.parquet as pq
import pandas as pd
import numpy as np
# Generate sample analytical dataset
np.random.seed(42)
n_rows = 2_000_000
df = pd.DataFrame({
"event_id": np.arange(n_rows, dtype=np.int64),
"user_uuid": [f"usr_{i % 50000:08x}" for i in range(n_rows)],
"region": np.random.choice(["us-east-1", "us-west-2", "eu-west-1", "ap-southeast-1"], size=n_rows),
"latency_ms": np.random.exponential(scale=45.0, size=n_rows).astype(np.float32),
"status_code": np.random.choice([200, 201, 400, 404, 500], p=[0.90, 0.05, 0.02, 0.02, 0.01], size=n_rows),
})
# Sort by high-frequency query filter columns to maximize min/max pruning!
df = df.sort_values(by=["region", "status_code"]).reset_index(drop=True)
# Convert to Arrow Table
table = pa.Table.from_pandas(df)
# Write with Zstandard compression and controlled row group sizing
pq.write_table(
table,
"optimized_telemetry.parquet",
compression="ZSTD",
compression_level=4,
row_group_size=500_000, # 500k rows per row group for targeted pruning
use_dictionary=True, # Activates dictionary encoding on 'region'
write_statistics=True # Emits min/max statistics into the footer
)
print("Successfully generated optimized Parquet file.")If you are using Pandas for data manipulation, see Pandas for Software Developers: The Complete Guide to Data Manipulation in Python to master memory management and Arrow-backed dtypes.
Blazing Fast Querying with Polars and DuckDB
Modern analytical tools can query Parquet files without loading the entire dataset into memory.
1. Polars Lazy Scanning (Rust-Powered Vectorized Engine)
# polars_query.py
import polars as pl
# scan_parquet reads only metadata, building a lazy execution graph
lazy_query = (
pl.scan_parquet("optimized_telemetry.parquet")
.filter(pl.col("region") == "us-east-1")
.filter(pl.col("status_code") >= 500)
.select(["user_uuid", "latency_ms"])
)
# Explain prints the physical plan, proving predicate & projection pushdown
print(lazy_query.explain())
# Execute the query efficiently with SIMD vectorization
result = lazy_query.collect()
print(result.head())2. DuckDB SQL Directly on Parquet Files
DuckDB allows you to execute complete analytical SQL queries directly against local files or S3 objects without launching a dedicated database server:
# duckdb_query.py
import duckdb
conn = duckdb.connect()
# Query directly from the Parquet file path
query = """
SELECT
region,
COUNT(*) AS total_errors,
AVG(latency_ms) AS avg_error_latency
FROM 'optimized_telemetry.parquet'
WHERE status_code >= 400
GROUP BY region
ORDER BY total_errors DESC;
"""
df_summary = conn.execute(query).df()
print(df_summary)For teams presenting operational metrics and analytical findings in internal dashboards, see Data Visualization in Python: When to Use Matplotlib, Seaborn, Plotly, and Altair.
7. Architectural Comparison: Parquet vs Avro vs ORC vs JSON/CSV
Choosing the right file format requires matching the format's physical strengths to your workload characteristics:
| Orientation | Columnar | Row-oriented | Columnar | Row-oriented |
| Compression Efficiency | High to Extreme (ZSTD/Snappy) | Moderate (Deflate/Snappy) | High to Extreme (ZLIB/Snappy) | Low |
| Schema Evolution | Robust (Add/Remove optional fields) | Best-in-class (JSON Schema + Union types) | Robust | Fragile (Breaks on column order changes) |
| Write Throughput | Moderate (Buffers row groups in RAM) | Very Fast (Streaming binary appends) | Moderate | Fast |
| Analytical Aggregations | Fastest (Vectorized, SIMD, Pruning) | Slow (Scans full rows) | Very Fast (Optimized for Hive/Presto) | Slowest |
| Streaming / Message Queues | Poor (Not designed for single-event appends) | Industry Standard (Kafka payload standard) | Poor | Common (JSON events) |
| Primary Ecosystem | Spark, DuckDB, Polars, Snowflake, Iceberg | Kafka, Event Streams, RPC schemas | Hive, Trino, Presto, Hadoop | Web APIs, Log drains, Human inspection |
Summary Decision Matrix
- Choose Apache Parquet for analytical querying (OLAP), data lakes, data warehousing, BI aggregation, and feature storage in machine learning pipelines.
- Choose Apache Avro for event streaming platforms (Apache Kafka), transactional message buses, and row-oriented ingestion where schemas evolve continuously.
- Choose CSV / JSON Lines only for small configuration exports or human-readable diagnostics where binary formats are impractical.
8. Parquet as the Foundation for Modern Table Formats
Individual Parquet files are immutable. You cannot modify a single row inside a Parquet file in place - to alter one record, you must rewrite the entire row group or the entire file.
To support ACID transactions, updates, deletes, and time-travel querying on top of immutable Parquet files, the data engineering industry created Open Table Formats:
- Apache Iceberg
- Delta Lake
- Apache Hudi
+-----------------------------------------------------------------------------------+
| TABLE FORMAT METADATA LAYER (Apache Iceberg / Delta Lake) |
| - ACID Transaction Logs (AOM Commit Log) |
| - Snapshot Manifests (Track which files belong to Snapshot v4) |
| - File-Level Partition Bounds & Column Statistics |
+-----------------------------------------------------------------------------------+
|
v
+-----------------------------------------------------------------------------------+
| PHYSICAL STORAGE LAYER (Apache Parquet) |
| s3://lake/data/001.parquet s3://lake/data/002.parquet s3://lake/data/003.parquet|
| (Immutable, ZSTD-compressed, dictionary-encoded columnar files) |
+-----------------------------------------------------------------------------------+
Table formats do not replace Parquet; they serve as a metadata management layer operating directly on top of immutable Parquet files. When an UPDATE or DELETE statement executes on an Iceberg or Delta table:
- The table format writes new Parquet data files containing the updated rows or positional delete files.
- It commits an atomic snapshot metadata file updating the pointer to the active set of Parquet files.
- Readers querying the snapshot continue reading immutable Parquet files with zero locks and complete snapshot isolation.
By decoupling the file format (Parquet) from the transaction management layer (Iceberg / Delta), modern data platforms achieve both blistering analytical scan performance and strict database consistency.
Conclusion and Engineering Best Practices
Apache Parquet is one of the most effective tools for reducing data lake storage costs while improving analytical query speeds. To get the most out of Parquet in production:
- Default to Zstandard (ZSTD): Use ZSTD compression level 3 or 4 to achieve 25% better compression ratios than Snappy without sacrificing decompression throughput.
- Sort Data Prior to Writing: Order your tables by high-frequency query filter columns before flushing to disk. Tight min/max bounds allow predicate pushdown to eliminate 80%+ of row groups.
- Control Row Group Sizing: Maintain row groups between 128MB and 512MB (500,000 to 2,000,000 rows) to balance memory footprint during writes with pruning efficiency during reads.
- Solve the Small Files Problem: Ensure individual files fall in the 128MB - 512MB range. Run background compaction jobs on streaming pipelines to prevent S3 metadata throttling.
- Leverage Pushdown-Aware Engines: Use engines like DuckDB, Polars, and Trino that take full advantage of Parquet's projection and predicate pushdown capabilities.

Comments
Have a question or something to add? Sign in to join the discussion.