readBetweenOffsets (SimpleDataSourceStreamReader)

Reads all available data between a specific start offset and end offset.

This method is invoked during failure recovery to re-read a batch deterministically.

Added in Databricks Runtime 15.3

Syntax

readBetweenOffsets(start: dict, end: dict)

Parameters

Parameter	Type	Description
start	dict	The start offset to read from.
end	dict	The offset at which reading stops.

Parameter	Type	Description
start	dict	The start offset to read from.
end	dict	The offset at which reading stops.

Returns

Iterator[Tuple]

All records between the start offset and end offset.