Using Parquet Futures Data in Python: pandas, polars, and DuckDB

Start with columns, rows, and one explicit file

Parquet stores data in columns and carries schema metadata, so a reader can often retrieve only the fields needed for a calculation. That matters for tick histories: selecting timestamp and price is much cheaper than loading every field merely to compute a price change. Start with one clearly identified file and inspect its schema before scaling to directories.

With pandas, an illustrative single-file read is:

import pandas as pd

file = "ES_2024_ticks.parquet"
ticks = pd.read_parquet(
    file,
    columns=["ts", "price", "volume"],
    filters=[("ts", ">=", pd.Timestamp("2024-03-01", tz="UTC"))],
)
print(ticks.dtypes)
print(ticks.head())

The filename is an example, not a required layout. Column projection reduces memory; a filter can reduce rows when the reader and file layout support predicate pushdown. A filter argument is not a license to assume that every possible Python expression executes inside Parquet. Keep pushdown filters simple, then apply any complex logic after reading.

For an interactive first pass, reading a small representative file is safer than calling read_parquet on a directory containing years of ticks. Inspect row counts, timestamp ranges, and nulls before writing a longer query. The documentation defines the available derived datasets and their column meanings.

Select directories deliberately

A directory reader may interpret every matching Parquet file below a path as one dataset. That is convenient for a bounded year of data and dangerous when the root contains an entire market history. Enumerate the intended inputs explicitly when the question concerns only a few periods:

files = [
    "ticks/ES/year=2023/data.parquet",
    "ticks/ES/year=2024/data.parquet",
]
ticks = pd.read_parquet(files, columns=["ts", "price", "volume"])

These illustrative paths show year partitioning; use the names actually present in the downloaded derived dataset. Avoid constructing a broad recursive glob and relying on a later pandas filter to rescue memory usage. If selection happens only after materialization, the full row set has already consumed memory.

DuckDB can query selected files without first creating a pandas DataFrame:

import duckdb

con = duckdb.connect()
result = con.execute('''
    SELECT ts, price, volume
    FROM read_parquet([
        'ticks/ES/year=2023/data.parquet',
        'ticks/ES/year=2024/data.parquet'
    ])
    WHERE ts >= TIMESTAMPTZ '2024-03-01 00:00:00+00'
      AND ts <  TIMESTAMPTZ '2024-04-01 00:00:00+00'
''').df()

Use a half-open interval such as March 1 inclusive to April 1 exclusive. It avoids boundary duplication and works for monthly, daily, or session windows without guessing the final timestamp in a file.

Make UTC and dtypes explicit

Futures histories cross daylight-saving transitions and exchange session boundaries. Per-contract derived ticks use UTC ts; recv_time equals ts, and event_time is null. Do not localize a timestamp that is already timezone-aware. Convert for display only after filtering and ordering in UTC.

ticks["ts"] = pd.to_datetime(ticks["ts"], utc=True)
ticks = ticks.sort_values("ts")
chicago_time = ticks["ts"].dt.tz_convert("America/Chicago")

The converted display column does not change the underlying instants. Avoid calling tz_localize("UTC") indiscriminately: localization is for naive timestamps, while conversion is for aware ones. Mixing the two can shift data or raise errors.

Check dtypes rather than assuming that Parquet and every reader produce identical Python representations. In per-contract ticks, price is float64 and volume is uint32. bid_sz and ask_sz describe executed volume at bid and ask, not resting order-book sizes; oi is zero because open interest is unavailable. Continuous derived ticks additionally contain price_raw and back_adjustment, with price = price_raw + back_adjustment.

Downcasting can save memory, but it should be a measured, reversible choice. A float32 price loses precision, and a signed integer volume changes the source schema. Preserve the original files and document any conversion used by an analysis.

Keep polars queries lazy until collection

Polars separates query construction from execution through lazy frames. Build the projection and filters first, inspect the plan if useful, and call collect only after the query is bounded:

import polars as pl

query = (
    pl.scan_parquet("ticks/ES/year=2024/data.parquet")
    .filter(
        (pl.col("ts") >= pl.datetime(2024, 3, 1, time_zone="UTC"))
        & (pl.col("ts") < pl.datetime(2024, 4, 1, time_zone="UTC"))
    )
    .select("ts", "price", "volume")
    .sort("ts")
)
march = query.collect(streaming=True)

Engine options vary by polars version, so use the streaming API supported by the installed release. Laziness itself does not guarantee low memory: a broad query, a large sort, or a high-cardinality group-by can still require substantial resources. It does give the optimizer an opportunity to push projections and predicates into the scan before materialization.

Use explain to check whether filters and selected columns appear near the scan. Then validate the practical result with row counts and process memory measurements. A plan is evidence about execution strategy, not a substitute for observing the query on representative inputs.

Set a memory budget before scanning history

Estimate the materialized result, not the compressed files on disk. Parquet compression makes storage size a poor proxy for RAM. A rough budget is selected rows multiplied by in-memory bytes per row, plus overhead for Python objects, indexes, intermediate expressions, sorts, and copies. DataFrames can require considerably more than the sum of their simplest dtype sizes.

Reduce the result before optimizing the reader

• Project only columns used by the calculation.
• Restrict the time range and files before collection.
• Process years or sessions independently when the algorithm permits.
• Aggregate in DuckDB or a lazy polars query instead of collecting raw ticks.
• Retain headroom for the operating system and subsequent feature calculations.

For example, if the goal is daily volume, request a grouped daily result rather than millions of ticks followed by a local group-by. If the goal is a custom tick-by-tick signal, iterate over bounded periods and write intermediate results with their parameters. Do not concatenate every period merely because each one fits individually.

Partition by year without inviting full scans

Year partitions provide useful administrative boundaries: a query for one year can target one directory, and reprocessing a year does not require rewriting the entire history. They do not automatically make every query efficient. A filter on a non-partition column still needs to inspect relevant data pages, and a query across twenty year directories remains a twenty-year scan.

Keep paths and predicates aligned. For a year-partitioned layout, select the necessary year directories and also apply timestamp bounds. The directory selection limits files; the timestamp predicate limits rows and helps within selected files. Avoid a root-level recursive scan when only one partition is needed.

Be cautious with readers that infer hive-style partition columns from paths. Confirm that an inferred year has the expected type and that it agrees with ts. Partition naming is metadata, not proof that every row belongs there. Verification should compare minimum and maximum timestamps per file or partition and investigate rows outside the expected year.

Resample with session and bar semantics in mind

Resampling ticks is not just choosing a frequency string. Decide the timezone, bucket origin, label side, closed side, and aggregation rule. A common OHLCV aggregation uses first price for open, maximum for high, minimum for low, last price for close, sum for volume, and count for trade count. Sort by UTC timestamp first, and ensure duplicate events are understood rather than accidentally reordering them.

Intraday derived bars in this dataset are epoch-aligned in UTC. Available intervals are 10s, 1m, 2m, 5m, 30m, 1h, 4h, and 1D. Empty buckets are omitted, so a missing row can mean no trades rather than a missing file. Daily bars follow the Chicago 17:00 session with daylight-saving handling; they are not simply midnight-to-midnight UTC groups.

Derived bars are built from back-adjusted continuous ticks. Resampling per-contract ticks will not necessarily reproduce them, especially across rolls. For continuous inputs, preserve the adjustment convention and verify price = price_raw + back_adjustment before aggregation. The tick data versus OHLCV bars guide explains when prebuilt bars are sufficient and when tick-level work is necessary.

Verify results and make runs reproducible

Validate at several scales. For each input, record file names, row counts, schema, minimum and maximum UTC timestamps, and null counts. After filtering, confirm that every returned timestamp lies in the requested interval. After resampling, check that high is at least open, close, and low; low is no greater than the other prices; and volume and trade counts behave sensibly.

Cross-check a small period against the corresponding derived bars. Compare bucket boundaries as well as values, accounting for omitted empty buckets and the daily session rule. Around continuous-contract rolls, inspect adjustment identity and point differences rather than mistaking a contract-reference gap for an ordinary return.

For reproducibility, save the exact query, selected file list, package versions, parameters, and a dataset manifest or checksums where available. Use a fresh process to rerun a small golden example and compare its output. Record the reader too: pandas, polars, and DuckDB can differ in timestamp display, integer handling, and query execution even when they read the same Parquet bytes.

A reliable pipeline fails loudly on missing columns, unexpected dtypes, unsorted timestamps, or an input file count that changed unexpectedly. Those checks are cheaper than debugging a plausible chart produced from an accidental full-history scan or a subtly shifted session boundary.