Skip to main content

Lakehouse Real-Time

Beta

This feature is in Beta. Reach out to your Databricks account team to enable this feature in your account.

Lakehouse//RT is in active development. Performance characteristics and the supported feature set can change.

Lakehouse Real-Time (Lakehouse//RT) is a serverless SQL warehouse type built for low-latency, high-concurrency workloads, such as serving analytical data to custom applications and AI agents, running operational analytics, or powering BI dashboards that require sub-second responses for hundreds to thousands of concurrent users.

Lakehouse//RT offers sub-second latency on SQL read queries against your Unity Catalog tables that use Delta Lake or Apache Iceberg formats in cloud storage. It queries your tables in place, so you don't need to copy data into a separate low-latency serving database to get fast responses. You create and manage Lakehouse//RT much like you do other SQL warehouses. A workspace admin or privileged user creates one or more per workspace and assigns permissions to users.

When to use Lakehouse//RT​

Lakehouse//RT fits workloads that send many concurrent, selective read queries and need each one to return in under a second. Use the following table to choose compute for a workload:

Workload

Recommended compute

Selective SELECT queries that need sub-second responses at high concurrency, such as application backends and AI agent tools

Lakehouse//RT

Long-running queries, joins across large tables, extract, transform, and load (ETL) jobs, writes, or queries that rely on non-ANSI SQL behavior

Serverless SQL warehouse

Transactional workloads with frequent row-level inserts, updates, and deletes

Lakebase

Workload

Recommended compute

Selective SELECT queries that need sub-second responses at high concurrency, such as application backends and AI agent tools

Lakehouse//RT

Long-running queries, joins across large tables, extract, transform, and load (ETL) jobs, writes, or queries that rely on non-ANSI SQL behavior

Serverless SQL warehouse

Transactional workloads with frequent row-level inserts, updates, and deletes

Lakebase

Requirements​

To use Lakehouse//RT, you must:

  • Be in a supported region.
  • Have the Lakehouse//RT Beta enabled in your workspace.

Enable Lakehouse//RT in your workspace​

Workspace admins can enable the Lakehouse//RT Beta in your workspace:

  1. In your workspace menu (top-right corner), go to Previews.
  2. Search for Lakehouse RT.
  3. Enable the preview.

After you enable the preview, the Real-Time warehouse type becomes available in the SQL warehouse creation flow for your workspace.

Create a Lakehouse//RT warehouse​

To create a Lakehouse//RT warehouse:

  1. Go to Compute > SQL Warehouses > Create SQL Warehouse.
  2. Select Real-Time.
  3. Select a Query size: Small, Medium, Large, or X-Large, depending on the performance your queries require.
  4. Configure the Autoscaling to determine the maximum DBUs you can be billed per hour.
  5. Set Auto stop to control when an idle warehouse stops.
  6. Enter a name for the warehouse.
  7. Click Create.

To assign permissions, grant Can use, Can monitor, or Can manage to users and groups, the same as a SQL warehouse. See SQL warehouse ACLs. For the settings that apply to every SQL warehouse type, see Create a SQL warehouse.

note

You cannot upgrade an existing SQL warehouse to Lakehouse//RT or downgrade an existing Lakehouse//RT warehouse to another warehouse type.

Size and scale a Lakehouse//RT warehouse​

Lakehouse//RT has settings that control performance and cost:

  • Query size controls the compute available to a single query.
  • Autoscaling controls how much the warehouse scales out to serve concurrent queries.
  • Auto stop controls how long an idle warehouse keeps running before it stops.

Query size and autoscaling work differently from the sizing and scaling settings on a serverless SQL warehouse.

Query size​

The query size sets the maximum compute a single query can use, which determines how fast an individual query can run. It also sets the minimum compute the warehouse runs on, and the minimum you are billed for, while the warehouse is up.

You set the query size when you create the warehouse. Choose Small, Medium, Large, or X-Large. A larger size gives each query more compute for faster results, at a higher minimum cost.

This setting is similar in concept to the size of a serverless SQL warehouse, which also sets the compute available to a single query.

Autoscaling​

Autoscaling lets a warehouse add compute to serve more concurrent queries, then release it when demand drops. You set the maximum compute the warehouse can scale up to, measured in DBUs. You can set the maximum up to 960 DBUs per hour.

Autoscaling on Lakehouse//RT differs from cluster scaling on a serverless SQL warehouse in a few ways:

  • It is measured in DBUs, not clusters.
  • The maximum is independent of query size. See Size for concurrency.
  • Lakehouse//RT autoscaling adds and removes just the compute needed, instead of in increments of clusters.

To see how much a warehouse is scaling, use its monitoring page.

Size for concurrency​

To serve many concurrent queries, scale out with autoscaling rather than choosing a larger query size:

  1. Choose the smallest query size that meets your latency target for a single query.
  2. Set the autoscaling maximum high enough to serve your peak number of concurrent queries.
  3. Watch the queued queries on the monitoring page. If queries queue at peak load, raise the autoscaling maximum.

A single Lakehouse//RT warehouse with a high autoscaling maximum can scale to serve many concurrent queries without you adding warehouses. If you need higher concurrency, contact your Databricks account team.

Auto stop​

Auto stop terminates the warehouse after it's idle for a set number of minutes, so an idle warehouse doesn't keep accruing charges. You set the idle timeout when you create the warehouse.

Auto stop on Lakehouse//RT works the same as on a serverless SQL warehouse. For the default and minimum values, see Configure SQL warehouse settings.

Monitor Lakehouse//RT activity​

You can monitor Lakehouse//RT warehouses and the queries they run the same as any other SQL warehouse:

Evaluate your workload​

Before you move a workload to Lakehouse//RT, test it with your own queries and compare latency against your current warehouse:

  1. Pick the queries that your application or dashboard runs most often, and confirm that they run in a few seconds on a serverless SQL warehouse.
  2. Create a Lakehouse//RT warehouse and run the same queries on it. Use the same tables, so no data movement is required.
  3. Fix any query that fails because it uses an unsupported feature or exceeds a guardrail.
  4. Compare latency between the two warehouses in the query history system table.

To compare median and 90th percentile latency between the two warehouses, run the following query and replace the placeholder IDs with your warehouse IDs. You need the SELECT privilege on the system.query.history table:

SQL
SELECT
compute.warehouse_id,
count(*) AS queries,
percentile(total_duration_ms, 0.5) AS p50_ms,
percentile(total_duration_ms, 0.9) AS p90_ms
FROM system.query.history
WHERE compute.warehouse_id IN ('<serverless-warehouse-id>', '<real-time-warehouse-id>')
AND execution_status = 'FINISHED'
AND start_time >= current_timestamp() - INTERVAL 1 DAY
GROUP BY compute.warehouse_id;

The query compares every finished query on each warehouse over the last day, so the results are comparable only when both warehouses ran the same workload. To limit the comparison to your test queries, set a query tag on them and filter on the query_tags column. See Query history system table reference.

Guardrails​

To protect low latency for all queries, Lakehouse//RT enforces per-query resource limits. A query that exceeds a guardrail returns an error instead of running slowly, so design queries to stay well within these limits.

Guardrail

Limit

Error

Query runtime

2 minutes

QUERY_RESOURCE_LIMIT_EXCEEDED with the QUERY_RUNTIME subclass

Statement timeout

1 minute by default. You can configure it up to the 2-minute query runtime limit.

QUERY_EXECUTION_TIMEOUT_EXCEEDED

Shuffle bytes written per stage

300 GB

QUERY_RESOURCE_LIMIT_EXCEEDED with the SHUFFLE_SIZE subclass

Spill size

30 GB, when spill is enabled

QUERY_RESOURCE_LIMIT_EXCEEDED with the SPILL_SIZE subclass

Files listed after pruning

10,000 files, adjustable up to 50,000

QUERY_RESOURCE_LIMIT_EXCEEDED with the FILE_LIMIT subclass

Guardrail

Limit

Error

Query runtime

2 minutes

QUERY_RESOURCE_LIMIT_EXCEEDED with the QUERY_RUNTIME subclass

Statement timeout

1 minute by default. You can configure it up to the 2-minute query runtime limit.

QUERY_EXECUTION_TIMEOUT_EXCEEDED

Shuffle bytes written per stage

300 GB

QUERY_RESOURCE_LIMIT_EXCEEDED with the SHUFFLE_SIZE subclass

Spill size

30 GB, when spill is enabled

QUERY_RESOURCE_LIMIT_EXCEEDED with the SPILL_SIZE subclass

Files listed after pruning

10,000 files, adjustable up to 50,000

QUERY_RESOURCE_LIMIT_EXCEEDED with the FILE_LIMIT subclass

Databricks can change these limits. The QUERY_RESOURCE_LIMIT_EXCEEDED error returns SQLSTATE 53400.

Use the following approaches to stay within the limits:

  • Query runtime: A query that approaches the limit doesn't meet the sub-second target even if it completes. Filter more or pre-aggregate.
  • Shuffle bytes: Wide joins or aggregations across large tables can exceed the limit. Pre-aggregate with materialized views instead of joining large tables at query time.
  • Files listed: Filter on partition or liquid clustering columns so that the query reads only the files it needs.

Data freshness​

To keep latency low, Lakehouse//RT reuses recently cached table metadata. Query results can lag the latest commits to a table by a few seconds.

Prepare your workloads for Lakehouse//RT​

Prepare your workloads before you move them to Lakehouse//RT. To test a workload end to end, see Evaluate your workload.

  • Use Unity Catalog managed tables. Managed tables with predictive optimization and liquid clustering keep your data clustered for your query patterns.
  • Verify queries are selective. For sub-second latency, verify that your queries scan less data. Filter early with WHERE clauses, select only the columns you require, and lean on aggregations. Joining across tables is supported, but if you find your query getting complex or slow, consider using materialized views that pre-aggregate your data for faster latencies.
  • Check SQL coverage. Lakehouse//RT supports ANSI-compliant read queries and CREATE TABLE AS SELECT (CTAS) only. Confirm that your workloads are ANSI compliant and avoid the unsupported statements, functions, and data types listed under Limitations.

Supported features​

This section lists the tools, table types, and connection methods that Lakehouse//RT supports. For what it does not support, see Limitations.

Tools and interfaces​

You can select Lakehouse//RT from the compute picker for any of the following Databricks features:

To build AI/BI dashboards on a Lakehouse//RT warehouse, including guardrails and dashboard-specific limitations, see Build AI/BI dashboards on Lakehouse Real-Time.

Table types​

Lakehouse//RT queries Unity Catalog data only. For best performance, use Unity Catalog managed tables, which provide the engine with the data layout it needs for low latency.

Lakehouse//RT supports the following table types:

Connectivity​

Lakehouse//RT only accepts connections that use the Statement Execution API. It does not support the legacy Thrift protocol. A driver that sends queries to a Lakehouse//RT warehouse over Thrift receives an error with SQLSTATE KP001.

To connect a custom application or AI agent, see Build applications and agents on Lakehouse Real-Time.

You can connect to a Lakehouse//RT warehouse in the following ways:

  • Statement Execution API: Call the API directly from external applications. See Statement Execution API: Run SQL on warehouses. Two versions are available:

    • Statement Execution API version 2: Use this version for the best Lakehouse//RT performance. It is optimized for Lakehouse//RT warehouses, with lower latency and higher concurrency, and takes warehouse_id as a path parameter on every call. It currently supports Lakehouse//RT warehouses only.
    • Statement Execution API version 1: Works with all SQL warehouse types. You can also run statements on Lakehouse//RT warehouses with version 1, but version 2 is optimized for them.
  • Databricks drivers: Point the driver's HTTP path at your Lakehouse//RT warehouse and use at least the version listed for the driver. From these versions, the driver detects Lakehouse//RT automatically and uses the connection path optimized for it, with no setting required. Earlier versions can connect with the setting listed, but with lower performance. Databricks recommends the latest driver version for full features and best performance.

    note

    A driver explicitly set to use Thrift, with use_kernel=False, useKernel: false, WithUseKernel(false), or UseThriftClient=1, fails against a Lakehouse//RT warehouse with SQLSTATE KP001.

  • Supported tools: The following tools connect to Lakehouse//RT through their own connectors:

Geospatial​

Lakehouse//RT supports the GEOGRAPHY and GEOMETRY data types and the st_ and h3_ functions, except those listed under Limitations.

Fine-grained access control​

Lakehouse//RT supports read queries on tables, materialized views, and streaming tables protected by row filters and column masks, including Unity Catalog built-in filters and masks. The filters and masks can be assigned directly to a table or applied through attribute-based access control (ABAC) policies. See Row filters and column masks and Attribute-based access control in Unity Catalog.

Use compute that supports policy management, such as a serverless SQL warehouse, to create or change row filters, column masks, and ABAC policies. Lakehouse//RT applies existing policies when reading tables. It does not support policy management statements or writes to tables with policies. For query restrictions, see Fine-grained access control limitations.

Column mask result types​

Lakehouse//RT casts the result of a column mask back to the column's declared data type using ANSI semantics. This applies to both table-level masks and ABAC masks. If the result cannot be cast to the column's type, the query fails.

For example:

  • A mask that returns the string 'REDACTED' for a BOOLEAN column causes the query to fail because the string cannot be cast to BOOLEAN.
  • A mask that returns FALSE for an INT column produces the integer 0.

On Databricks Runtime, a table-level column mask exposes the mask's return type to the query. Moving the same query to Lakehouse//RT can change its result type or cause a cast error. ABAC masks cast results back to the column type on both Databricks Runtime and Lakehouse//RT. Databricks recommends masking functions that return the same type as the column.

Query plans and profiles for protected tables​

For queries that use row filters or column masks, Lakehouse//RT limits the details shown in EXPLAIN output and query profiles to protect information about the underlying data and policy logic:

  • EXPLAIN output hides query steps that read protected data or apply policies. It also omits plan statistics.
  • Query profiles hide these query steps and omit statistics for protected tables.

Troubleshooting​

The following table lists common errors and how to resolve them:

Error

Cause

Resolution

Thrift protocol error (SQLSTATE KP001)

The driver connects over Thrift.

Configure the driver to use the Statement Execution API. See Connectivity.

UNSUPPORTED_FEATURE (SQLSTATE 0A000)

The query uses a statement, function, data type, or table type that Lakehouse//RT doesn't support. The message names the feature.

Rewrite the query to avoid the feature, or run it on a serverless SQL warehouse. See Limitations.

QUERY_RESOURCE_LIMIT_EXCEEDED (SQLSTATE 53400)

The query exceeds a guardrail. The error subclass (QUERY_RUNTIME, SHUFFLE_SIZE, SPILL_SIZE, or FILE_LIMIT) names the limit.

Filter earlier, select fewer columns, or pre-aggregate with materialized views. See Guardrails.

QUERY_EXECUTION_TIMEOUT_EXCEEDED

The query runs longer than the statement timeout.

Optimize the query, or raise the statement timeout up to the query runtime limit.

A cast error on a table with a column mask

The mask returns a value that can't be cast to the column's type.

Use a masking function that returns the column's data type. See Column mask result types.

403 Forbidden when reading storage

A storage firewall blocks serverless compute.

Configure your firewall for serverless compute. See Serverless compute firewall configuration.

Error

Cause

Resolution

Thrift protocol error (SQLSTATE KP001)

The driver connects over Thrift.

Configure the driver to use the Statement Execution API. See Connectivity.

UNSUPPORTED_FEATURE (SQLSTATE 0A000)

The query uses a statement, function, data type, or table type that Lakehouse//RT doesn't support. The message names the feature.

Rewrite the query to avoid the feature, or run it on a serverless SQL warehouse. See Limitations.

QUERY_RESOURCE_LIMIT_EXCEEDED (SQLSTATE 53400)

The query exceeds a guardrail. The error subclass (QUERY_RUNTIME, SHUFFLE_SIZE, SPILL_SIZE, or FILE_LIMIT) names the limit.

Filter earlier, select fewer columns, or pre-aggregate with materialized views. See Guardrails.

QUERY_EXECUTION_TIMEOUT_EXCEEDED

The query runs longer than the statement timeout.

Optimize the query, or raise the statement timeout up to the query runtime limit.

A cast error on a table with a column mask

The mask returns a value that can't be cast to the column's type.

Use a masking function that returns the column's data type. See Column mask result types.

403 Forbidden when reading storage

A storage firewall blocks serverless compute.

Configure your firewall for serverless compute. See Serverless compute firewall configuration.

If an issue persists, collect the following information and contact Databricks support:

  • The error message.
  • An export of the query profile, including the raw query text.
  • If the query runs in an AI/BI dashboard, a link to the dashboard.
  • Confirmation that the same query runs on a serverless SQL warehouse.

Pricing​

For pricing information, see the Databricks pricing page.

Limitations​

When a query uses an unsupported feature, Lakehouse//RT returns an error naming the feature. To successfully run the query, use a serverless SQL warehouse instead.

Tools and features​

Lakehouse//RT does not support the following features:

  • Genie, including Genie Agents
  • Jobs tasks

Table types​

Lakehouse//RT does not support the following table types:

  • Tables in Unity Catalog default storage
  • External tables in Unity Catalog
  • Hive metastore tables (managed or external)
  • Foreign tables and query federation (Lakehouse Federation)
  • Delta Sharing tables
  • Temporary tables
  • Tables that use other data formats (CSV, JSON, Avro, Parquet, ORC, and text)

Fine-grained access control limitations​

Queries involving row filters or column masks have the following limitations:

  • Change data feed reads using table_changes are not supported on protected tables.
  • ABAC policies attached to views are not supported.
  • Multiple DISTINCT argument sets in an aggregate inside a policy function fail with UNSUPPORTED_FEATURE. Aggregates that share the same DISTINCT arguments and aggregates in the outer query remain supported.

Drivers and connectors​

Lakehouse//RT does not support the Simba Spark ODBC Driver or the Databricks ODBC Driver. Use the Databricks OSS ODBC Driver instead.

SQL language​

Lakehouse//RT runs SQL read queries in ANSI mode only. It evaluates all implicit type coercions and casts according to strict ANSI SQL rules, and this behavior cannot be turned off. Under ANSI semantics, queries that relied on non-ANSI behavior might:

  • Raise a runtime error instead of silently producing NULL. For example, casting a non-numeric string to a number.
  • Raise an analysis-time error when there is no safe common type. For example, COALESCE, CASE, IN, or set operations across incompatible types.
  • Return a different result type than legacy mode, because ANSI string promotion and numeric widening choose safe, lossless types.

To get predictable results, use explicit CAST expressions when ANSI mode implicit conversions don't produce the behavior you expect.

Lakehouse//RT does not support the following:

  • Functions: AI functions, Python UDFs, and XPath and XML functions.
  • Spatial functions: st_buffer, st_concavehull, st_simplify, st_envelope_agg, and st_union_agg.
  • H3 functions: h3_tessellateaswkb, h3_try_tessellateaswkb, and h3_getpentagoncellids.

Lakehouse//RT is for read (SELECT) queries and CREATE TABLE AS SELECT (CTAS). Other write and ETL commands are not supported, including:

  • Write operations: INSERT, UPDATE, DELETE, and MERGE.
  • DDL: CREATE (other than CTAS), ALTER, DROP, and other statements that create or modify objects.
  • Security statements: GRANT and REVOKE.
  • Scripting, stored procedures, temporary tables, and multi-statement transactions.
  • Delta Lake maintenance: OPTIMIZE, ANALYZE, VACUUM, and REFRESH.

Compliance​

Lakehouse//RT does not support compliance security profiles.