STREAMING_REAL_TIME_MODE error condition
Streaming real-time mode has the following limitation:
ARBITRARY_STATEFUL_OPERATIONS_NOT_SUPPORTED
The arbitrary stateful streaming operator, <op>, is not currently supported. Please see the Real-Time Mode User Guide for a list of supported operators.
ASYNC_PROGRESS_TRACKING_NOT_SUPPORTED
Async progress tracking is not supported in real-time mode. Set option asyncProgressTrackingEnabled to false and retry your query.
ASYNC_STATE_CHECKPOINT_NOT_SUPPORTED
Async state checkpointing is not supported in real-time mode. Set spark.databricks.streaming.statefulOperator.asyncCheckpoint.enabled to false and retry your query.
BATCH_UNION_NOT_SUPPORTED
Real-time mode does not support union with batch data sources. Ensure that the union streaming query only contains streaming data sources.
CLUSTER_CONFIGURATION_NOT_SUPPORTED
The following configuration(s) are not supported:
<invalidReasons>. Set spark.databricks.streaming.realTimeMode.enabled to true in your cluster configuration, and remove <confNames> from your cluster configuration if present, and then restart your Spark cluster.
DROP_DUPLICATES_WITHIN_WATERMARK_NOT_SUPPORTED
dropDuplicatesWithinWatermark is not supported. To work around this limitation, you can use dropDuplicates instead. Remember to pass your event-time column to dropDuplicates so that state can be evicted.
EVENT_TIME_BASED_TIMERS_IN_TRANSFORM_WITH_STATE_NOT_SUPPORTED
Event time based timers in transformWithState operator are not supported yet. If you would like to use the transformWithState operator, please specify processing time in your transformWithState operator and retry your query.
IDENTICAL_SOURCES_IN_UNION_NOT_SUPPORTED
Real-time mode does not support union on two or more identical streaming data sources in a single query. This includes scenarios such as referencing the same source DataFrame more than once, or using two data sources with identical configurations for some sources. For Kafka, avoid reusing the same DataFrame and create different ones. For Kinesis, in addition to duplicating data frames, ensure that each DataFrame sets a unique value for the "consumerName" option. Sources provided in the query: <sources>
INPUT_STREAM_NOT_SUPPORTED
The input stream <className> is not supported. See the Real-Time Mode User Guide for a list of supported input streams.
OPERATOR_OR_SINK_NOT_IN_ALLOWLIST
The <errorType>(s): <message> not in the <errorType> allowlist for Real-Time Mode. To bypass this check, set spark.databricks.streaming.realTimeMode.allowlistCheck to false. By changing this, you agree to run the query at your own risk.
OUTPUT_MODE_NOT_SUPPORTED
The output mode <outputMode> is not supported. To work around this limitation, set the output mode to Update. In the future, <outputMode> may be supported.
SESSION_WINDOWS_NOT_SUPPORTED
Session windows are not yet supported. The only windowed aggregations allowed are over tumbling and sliding windows.
SHUFFLE_MORE_THAN_ONCE
There are <shuffleCount> org.apache.spark.sql.execution.exchange.ShuffleExchangeExec operators in the physical plan. Likely, you are calling the operations that require shuffle more than once, and RTM does not support this. To run the query, set spark.databricks.streaming.realTimeMode.allowlistCheck to false. By changing this, you agree to run the query at your own risk.
SINK_NOT_SUPPORTED
The <className> sink is currently not supported. See the Real-Time Mode User Guide for a list of supported sinks.
SQL_CONFIGURATION_NOT_SUPPORTED
The following configuration(s) are not supported:
<invalidReasons>. Set spark.databricks.streaming.realTimeMode.enabled to true in your cluster configuration, and remove <confNames> from your SparkSession configuration if present, and then restart your Spark cluster.
STATEFUL_OPERATORS_BEFORE_UNION_NOT_SUPPORTED
Streaming queries in real-time mode cannot include stateful operators (e.g. aggregate, deduplicate, transformWithState) before a union. Please restructure your query to apply the union before any stateful operations.
STREAM_STREAM_JOIN_NOT_SUPPORTED
No type of streaming joins are supported yet.