Float statistics¶
In short
Parquetry writes the min/max statistics of float columns in the type-defined order by default, and the readers tested open such files and use the statistics to skip data. Pruning agrees with the per-row comparison on NaN and on the two zeros. IEEE 754 total order is available on request, for files read by readers known to support it.
This page explains how Parquetry writes and reads the min/max statistics of FLOAT, DOUBLE and FLOAT16 columns, what it does with NaN and the two zeros, and when to ask for IEEE 754 total order. It is for developers writing float columns for other readers, and for developers filtering on float columns.
The pruning tiers fed by these statistics are described in The read path.
The guiding rule: the readers tested open the default output and prune it like the files of other writers; IEEE 754 total order is a choice for files with known readers.
1. Why float bounds need rules¶
Min and max statistics let a reader skip a row group or a page without decoding it. Two kinds of floating-point values break the plain "smallest and largest value" contract:
NaNis unordered:<,<=,>,>=and=are false betweenNaNand any value,NaNincluded. Bounds computed overNaNcells hide the numbers of the column, and readers disagree on whetherNaNsorts above or below the numbers.-0.0and+0.0are equal under IEEE 754 comparison and differ as bit patterns. A minimum of+0.0hides a-0.0cell from a reader ordering the two.
The Parquet format answers with a nan_count statistic and with a column order declared per column in the footer. The order tells a reader how the bounds of the column were computed.
2. What Parquetry writes¶
Float columns declare the type-defined order by default. A write can ask for IEEE 754 total order instead (section 4). Both orders record a NaN count.
| Type-defined order (default) | IEEE 754 total order | |
|---|---|---|
nan_count | chunk statistics, page headers, column index | the same |
| Bounds of a chunk or page | its smallest and largest number; NaN cells are left out | the same |
| A zero bound | -0.0 as a minimum, +0.0 as a maximum | the zero of the cell: -0.0 orders before +0.0 |
Chunk or page of only NaN | no bounds | its smallest and largest NaN, sign and payload bits included |
Chunk with a page of only NaN | no column index, as required by the format | keeps its column index |
Deprecated min / max | written for FLOAT and DOUBLE | the same |
A FLOAT or DOUBLE column annotated UNKNOWN keeps the type-defined order in either case.
Dictionary pages keep the bit patterns of NaN cells and of the two zeros: a cell reads back with its sign and payload intact.
3. How Parquetry reads them¶
FLOAT and DOUBLE predicates compare as IEEE 754 numbers: <, <=, > and >= never match NaN, and -0.0 equals +0.0. NaN literals are the one extension: x = NaN and IN with a NaN match the NaN cells, and x <> NaN matches the other non-null cells. NOT over a comparison reads as the complementary operator: NOT (x > 5) is x <= 5, and matches no NaN. Pruning has to agree with that row by row. FLOAT16 cells compare as bytes, and their bounds and NaN counts do not prune.
- Bounds prune comparisons with numbers. A
NaNbound is ignored, and a zero bound matches either zero. - NaN counts rule on the
NaNcells: x = NaNskips a row group or page with a NaN count of zero;- a comparison with a number, and
x <> NaN, skip a row group or page holding nothing butNaN, known from NaN and null counts adding up to its values; - a row group with a NaN count of zero and no null can match a comparison as a whole, and count then answers from metadata.
- Without NaN counts (files from older writers)
NaNcells may sit anywhere:x = NaNprunes nothing, and no float row group matches as a whole.
The rules are the same in both column orders: what is known of NaN cells comes from the counts, never from a NaN bound. Only a total-order column index lists a page of only NaN (section 2), and only such an index lets a read skip that page.
par explain shows these decisions per row group and tier.
4. Asking for IEEE 754 total order¶
Total order is the order recommended by the format for float columns: it leaves no ambiguity on NaN and on the zeros, and a chunk with a page of only NaN keeps its column index. The cost is compatibility: the order entered the format in parquet-format 2.13.0 (June 2026), and several readers in use today cannot open a file written in it or do not prune its float columns (section 5).
Choose the order of float statistics shows how to ask for it.
5. Reader compatibility¶
This table was measured in October 2026, with the reader versions named in each row. Releases after those versions were not measured. The probe ran on par cp copies of a 1M-row file with float32 columns, NaN cells and one row group of only NaN, against the same queries on the source file written by pyarrow. "Prunes" means the reader skipped the row groups ruled out by a filter on a float column. The probe overwrote the column chunks of those row groups with garbage. A reader that still returned the right rows skipped them. The parquet-java row comes from the Parquetry conformance tests, not from this probe.
| Reader | Type-defined order (default) | IEEE 754 total order |
|---|---|---|
| Arrow C++ / pyarrow 16 to 24 | reads, prunes | reads, prunes |
| Arrow C++ / pyarrow 25.0 | reads, prunes | reads, does not prune float columns |
| arrow-rs / DataFusion 40 to 50 | reads, prunes | cannot open the file |
| arrow-rs / DataFusion 51 to 54 | reads, prunes | reads, prunes |
| Polars 1.0 to 1.40 | reads | cannot open the file |
| Polars 1.44 | reads | reads |
| DuckDB 0.10 to 1.5 | reads, prunes | reads, prunes |
| parquet-java 1.18 (conformance tests) | reads | reads |
Notes:
- GDAL and geopandas read through Arrow C++. Their bbox filters on float32 covering columns were not measured separately.
- The Polars rows say "reads" alone: Polars 1.30 and later did not skip the row groups of this query on the pyarrow source either. Polars 1.0 and 1.20 skipped them on the default copy as on the source.
- parquet-java 1.18 is the oracle of the Parquetry conformance tests. For the same cells it writes the same bounds and NaN counts in either order, and the same column indexes in total order. In the type-defined order it writes no column index for a chunk holding a
NaN; Parquetry keeps the index unless a page holds onlyNaN. Its filtered reads of Parquetry files return the same rows with and without pruning. - parquet-java 1.17 and earlier ignore the statistics of a column in an order unknown to them, except for a chunk with equal min and max (per its sources; not run here): rows stay correct, with little pruning on those columns.
- pyarrow 25.0, Polars 1.44, DataFusion 54 and DuckDB 1.5 write the type-defined order for float columns, as read back from a file written by each. parquet-java 1.18 writes total order by default.
6. Where to look¶
Paths are relative to the io/tileverse/parquetry package of parquetry-core, in its main sources, and in its test sources for the conformance tests.
| Concern | Start in |
|---|---|
| Write option | data/WriteOptions.java (FloatColumnOrder) |
| Order of a column's bounds | internal/write/BoundsOrder.java |
| Statistics rules per order | internal/write/StatisticsAccumulator.java |
| Footer column orders | internal/write/FooterColumnOrders.java |
| Row-group pruning with NaN counts | internal/filter/StatsEvaluator.java, internal/filter/NaNCells.java |
| Page pruning with NaN counts | internal/filter/ColumnIndexEvaluator.java |
| Conformance with parquet-java (tests) | internal/write/conformance/FloatStatisticsConformanceIT.java, FloatTotalOrderConformanceIT.java |
Scope: the statistics of floating-point columns in a single Parquet / GeoParquet file. The proof of a whole row group matching, and counting from it, is in Counting. The pruning tiers are in The read path.