STREAMING_REAL_TIME_MODE error condition
Streaming real-time mode has the following limitation:
ARBITRARY_STATEFUL_OPERATIONS_NOT_SUPPORTED
The arbitrary stateful streaming operator, <op>, is not currently supported. Please see the Real-Time Mode User Guide for a list of supported operators.
ASYNC_PROGRESS_TRACKING_NOT_SUPPORTED
Async progress tracking is not supported in real-time mode. Set option asyncProgressTrackingEnabled to false and retry your query.
ASYNC_STATE_CHECKPOINT_NOT_SUPPORTED
Async state checkpointing is not supported in real-time mode. Set spark.databricks.streaming.statefulOperator.asyncCheckpoint.enabled to false and retry your query.
CHECKPOINT_V2_REQUIRED
The real-time mode requires checkpoint v2. Set spark.sql.streaming.stateStore.checkpointFormatVersion to 2 and retry your query.
CONCURRENT_STAGE_DAG_SCHEDULER_REQUIRED
The real-time mode requires the concurrent stage DAG scheduler. Restart your spark cluster and set spark.databricks.dagScheduler.type to ConcurrentStageDAGScheduler.
DROP_DUPLICATES_WITHIN_WATERMARK_NOT_SUPPORTED
dropDuplicatesWithinWatermark is not supported. To work around this limitation, you can use dropDuplicates instead. Remember to pass your event-time column to dropDuplicates so that state can be evicted.
EVENT_TIME_BASED_TIMERS_IN_TRANSFORM_WITH_STATE_NOT_SUPPORTED
Event time based timers in transformWithState operator are not supported yet. If you would like to use the transformWithState operator, please specify processing time in your transformWithState operator and retry your query.
MULTI_SHUFFLE_MANAGER_REQUIRED
The real-time mode requires the multi-shuffle manager. Set spark.shuffle.manager to org.apache.spark.shuffle.streaming.MultiShuffleManager and restart your spark cluster.
OUTPUT_MODE_NOT_SUPPORTED
The output mode <outputMode> is not supported. To work around this limitation, set the output mode to Update. In the future, <outputMode> may be supported.
ROCKSDB_STATE_STORE_PROVIDER_REQUIRED
The real-time mode requires the RocksDB state store provider or providers that extends it. Set spark.sql.streaming.stateStore.providerClass to org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider and retry your query.
SESSION_WINDOWS_NOT_SUPPORTED
Session windows are not yet supported. The only windowed aggregations allowed are over tumbling and sliding windows.
SPECULATIVE_EXECUTION_NOT_SUPPORTED
Speculative execution is not supported in real-time mode. Set spark.speculation to false and restart your spark cluster.
STREAM_STREAM_JOIN_NOT_SUPPORTED
No type of streaming joins are supported yet.