Prepare Data for DL Training

This topic describes two methods for preparing data for distributed training:

Petastorm

Petastorm is an open source data access library that enables directly loading data stored in Apache Parquet format. This is convenient for Databricks and Spark users because Parquet is the recommended data format.

The following topic describes and illustrates this use case:

TFRecord

You can also use TFRecord format as the data source for distributed DL. TFRecord format is a simple record-oriented binary format that many TensorFlow applications use for training data.

tf.data.TFRecordDataset is the TensorFlow dataset, which is comprised of records from TFRecords files. For more details about how to consume TFRecord data, see the TensorFlow guide Consuming TFRecord data.

The following topics describe and illustrate the recommended ways to save your data to TFRecord files:

The following topic describes and illustrates the recommended way to load TFRecord files: