partitions (DataSourceStreamReader)

Returns a list of InputPartition objects given the start and end offsets.

Each InputPartition represents a data split that can be processed by one Spark task. When called with an empty offset range where start == end, this method should return an empty sequence.

Added in Databricks Runtime 15.2

Syntax

partitions(start: dict, end: dict)

Parameters

Parameter	Type	Description
start	dict	The start offset of the microbatch to plan partitioning.
end	dict	The end offset of the microbatch to plan partitioning.

Parameter	Type	Description
start	dict	The start offset of the microbatch to plan partitioning.
end	dict	The end offset of the microbatch to plan partitioning.

Returns

Sequence[InputPartition]

A sequence of partitions for this data source. Each partition value must be an instance of InputPartition or a subclass of it.