Multi-file datasets¶
You have a directory, a bucket prefix or a Hive-partitioned tree of Parquet files sharing one schema, and want to query them as one table. parquetry-catalog reads them as one dataset: one count, one stream of rows, and the files unable to match the filter left unopened. This guide reads a directory of two files, then shows remote files, partitions, and one dataset per file.
It needs parquetry-catalog, with the BOM imported as in Getting started:
<dependency>
<groupId>io.tileverse.parquetry</groupId>
<artifactId>parquetry-catalog</artifactId>
</dependency>
parquetry-catalog uses Java 25 preview features: start the JVM with --enable-preview, as listed in Getting started.
Read a directory¶
The directory cities holds two files written as in Write files, with an id and a name column. part-0.parquet holds Rosario and Montevideo, with the ids 1 and 2. part-1.parquet holds Asuncion and Santiago, with the ids 3 and 4. The program counts and prints the cities with an id of 3 or more:
import static io.tileverse.parquetry.filter.Pred.col;
import io.tileverse.parquetry.catalog.CatalogOptions;
import io.tileverse.parquetry.catalog.FilesetCatalog;
import io.tileverse.parquetry.data.ReadOptions;
import io.tileverse.parquetry.dataset.ParquetDataset;
import io.tileverse.parquetry.filter.Predicate;
import io.tileverse.parquetry.filter.Projection;
import io.tileverse.parquetry.io.FileSource;
import io.tileverse.parquetry.record.ParquetRecord;
import io.tileverse.parquetry.schema.ColumnPath;
import java.nio.file.Path;
import java.util.stream.Stream;
public class ReadDirectory {
private static final ColumnPath NAME = ColumnPath.of("name");
public static void main(String[] args) {
Path directory = Path.of("cities");
Predicate filter = col("id").gtEq(3L);
try (FileSource files = FileSource.directory(directory, "*.parquet");
FilesetCatalog catalog = FilesetCatalog.open(files, CatalogOptions.defaults())) {
String name = catalog.datasets().getFirst();
ParquetDataset dataset = catalog.dataset(name);
long matching = dataset.count(filter, ReadOptions.DEFAULTS);
System.out.println(name + ": " + matching + " matching rows");
try (Stream<ParquetRecord> rows = dataset.read(filter, Projection.ALL, ReadOptions.DEFAULTS)) {
rows.forEach(row -> System.out.println(row.getString(NAME)));
}
}
}
}
It prints:
FileSource.directorylists the files of the directory matching the glob.*.parquetmatches the files of the directory itself, and**/*.parquetthe files of its subdirectories too.FilesetCatalog.openmerges the listed files into one dataset, named after the directory. Opening the catalog reads no file. The first call todataset(name)reads the footers of the files.- The dataset has the read methods of a single file:
read,readBatches,count,boundsandexplain, with the same predicates, projections and read options. - Closing the catalog closes the files and the source.
The files must have equal schemas. A file with another schema fails dataset(name) with a ParquetSchemaException naming the two files.
A read drains up to eight files at once, and the rows of different files come in no fixed order.
Skip files¶
A dataset skips a file when the statistics in its footer rule it out, the way a single file skips row groups. explain shows the decisions on the files without reading the rows:
import io.tileverse.parquetry.dataset.explain.DatasetExplainPlan;
DatasetExplainPlan plan = dataset.explain(filter, Projection.ALL, ReadOptions.DEFAULTS);
System.out.println(plan.toAsciiTable());
For the filter of the sample, the plan starts with the files:
1 kept, 1 skipped (2 rows pruned without opening a file)
location outcome rows reason
part-0.parquet SKIP 2 GtEq id: value outside [min, max]
part-1.parquet KEEP 2 kept
The ids of part-0.parquet lie between 1 and 2, and the read never opens it. A kept file is read like a single file, skipping row groups and pages, and the plan lists these decisions after the files.
Read files from object storage¶
FileSource.open takes the URI of a directory or a bucket prefix, and a glob:
import java.net.URI;
import java.util.Properties;
URI cities = URI.create("s3://my-bucket/cities/");
FileSource files = FileSource.open(cities, "**/*.parquet", new Properties());
A file: URI, or a URI without a scheme, names a local directory. Another scheme opens a tileverse Storage with the backend modules of Cloud storage, configured by the storage.* entries of the properties. The byte-range cache stays off unless the properties set storage.caching.enabled=true.
FileSource.openObject(uri, properties) serves one known object without listing its prefix. A credential allowed to read the object, and not to list the bucket, is enough.
Skip partitions¶
A Hive-partitioned tree puts partition keys and their values in the names of its directories, as key=value:
FileSource.directory(directory, "**/*.parquet") lists both files, and the catalog reads them as one dataset. A filter on the key skips the files of the other partitions without opening them. With col("country").eq("UY"), the plan shows:
location outcome rows reason
country=AR/part-0.parquet SKIP 2 Eq country: value outside [min, max]
country=UY/part-0.parquet KEEP 2 kept
The files of the sample have no country column. The dataset then adds one, after the columns of the files. For the rows of a file, the added column holds the value written in the path of that file. Its type comes from the values in the paths: whole numbers, decimal numbers, dates written as yyyy-MM-dd, or strings otherwise. A file with a country column of its own keeps it, and the path value prunes that column.
Such an added column cannot hold null partitions: a path segment country=__HIVE_DEFAULT_PARTITION__ fails dataset(name) with an IllegalStateException.
One dataset per file¶
FilesetCatalog.openPerFile(files) makes each listed file a dataset of its own, named after the file without its extension. The files need not share a schema. Over a directory holding rivers.parquet and lakes.parquet, catalog.datasets() returns [lakes, rivers].
Where to go next¶
- Filter rows builds the predicates, which work the same on a dataset.
- Cloud storage lists the backends and their modules.
- Read columnar batches reads column vectors, with
dataset.readBatches.
Under the hood¶
- The read path follows the read of one file, as run by a dataset on the kept files.