Test the mapGroupsWithState
update function
The TestGroupState
API enables you to test the state update function used for Dataset.groupByKey(...).mapGroupsWithState(...)
and Dataset.groupByKey(...).flatMapGroupsWithState(...)
.
The state update function takes the previous state as input using an object of type GroupState
. See the Apache Spark GroupState reference documentation. For example:
import org.apache.spark.sql.streaming._
import org.apache.spark.api.java.Optional
test("flatMapGroupsWithState's state update function") {
var prevState = TestGroupState.create[UserStatus](
optionalState = Optional.empty[UserStatus],
timeoutConf = GroupStateTimeout.EventTimeTimeout,
batchProcessingTimeMs = 1L,
eventTimeWatermarkMs = Optional.of(1L),
hasTimedOut = false)
val userId: String = ...
val actions: Iterator[UserAction] = ...
assert(!prevState.hasUpdated)
updateState(userId, actions, prevState)
assert(prevState.hasUpdated)
}