Build applications and agents on Lakehouse Real-Time
Lakehouse Real-Time (Lakehouse//RT) is in Beta. Reach out to your Databricks account team to enable this feature in your account.
Lakehouse//RT is in active development. Performance characteristics and the supported feature set can change.
Connect a custom application or AI agent to a Lakehouse//RT warehouse to serve sub-second query results from your Unity Catalog tables. Applications and agents query the tables in place, so they don't need a separate serving database that holds a copy of your data.
Requirements
To connect an application or agent to Lakehouse//RT, you need:
- A Lakehouse//RT warehouse. See Create a Lakehouse//RT warehouse.
- A service principal with an OAuth secret for the application to authenticate as. See Authorize Databricks service principal access to Databricks with OAuth.
- The Can use permission on the warehouse for the service principal. See SQL warehouse ACLs.
- The
USE CATALOGandUSE SCHEMAprivileges on the parent catalog and schema, and theSELECTprivilege on the tables the application queries, granted to the service principal. Grant access only to the tables the application needs. - Queries that fit the Lakehouse//RT limitations and guardrails.
Connect from an application
Lakehouse//RT accepts connections only through the Statement Execution API. Call the API directly, or use a driver configured for it. For the list of drivers and their settings, see Connectivity.
To connect from a Python application with the Databricks SQL Connector for Python:
-
Install the connector with the
kernelextra. The extra requires Python 3.10 or above. Version 4.9.0 and later detects a Lakehouse//RT warehouse automatically when thekernelextra is installed.Shellpip install "databricks-sql-connector[kernel]" -
Set the following environment variables:
DATABRICKS_SERVER_HOSTNAME: the Server Hostname value for the warehouse. See Get connection details for a Databricks compute resource.DATABRICKS_HTTP_PATH: the HTTP Path value for the warehouse.DATABRICKS_CLIENT_ID: the service principal's application ID.DATABRICKS_CLIENT_SECRET: the service principal's OAuth secret.
-
Connect and run parameterized queries. The following example sets
use_kernel=Trueto select the Statement Execution API path explicitly, which also works on versions earlier than 4.9.0. It authenticates as the service principal, sets a statement timeout for the session, and passes user input as a query parameter instead of building the SQL string:Pythonimport osfrom databricks import sqlconnection = sql.connect(server_hostname=os.getenv("DATABRICKS_SERVER_HOSTNAME"),http_path=os.getenv("DATABRICKS_HTTP_PATH"),oauth_client_id=os.getenv("DATABRICKS_CLIENT_ID"),oauth_client_secret=os.getenv("DATABRICKS_CLIENT_SECRET"),use_kernel=True,session_configuration={"STATEMENT_TIMEOUT": "30"}, # seconds)def get_account_summary(account_id):with connection.cursor() as cursor:cursor.execute("SELECT region, sum(revenue) AS revenue ""FROM main.sales.account_daily ""WHERE account_id = ? GROUP BY region",parameters=[account_id],)return cursor.fetchall()Calling
get_account_summaryreturns the matching rows, and the query appears in the warehouse's query history.With
use_kernel=True, pass the service principal's credentials asoauth_client_idandoauth_client_secret. The connector doesn't accept a customcredentials_provideron this path.
Reuse connections
Create the connection one time when your application starts, and reuse it for every request instead of connecting for each query. Opening a new connection for each request adds latency to every query. Open a cursor for each query and close it when the query finishes.
Set a statement timeout
Set a statement timeout so that a slow query fails fast instead of holding a request open. A query that runs longer than the statement timeout returns a QUERY_EXECUTION_TIMEOUT_EXCEEDED error. The default timeout is 1 minute. Set it for the session with the STATEMENT_TIMEOUT parameter, which takes a value in seconds (the previous example sets 30 seconds), or with a SET STATEMENT_TIMEOUT statement. The value can't exceed the query runtime guardrail. See STATEMENT_TIMEOUT.
Query from an AI agent
To give an AI agent fast access to your data, define a tool that the agent calls, and have the tool run SQL on the Lakehouse//RT warehouse through a driver or the Statement Execution API. Genie doesn't run on Lakehouse//RT. To set up the tool:
- Write a function that takes the agent's inputs as arguments and runs a parameterized query, like
get_account_summaryin the previous example. - Register the function as a tool in your agent framework. See Define local Python function tools.
- Run the agent as a service principal that has
SELECTonly on the tables the tool needs.
Agents can send many queries for each conversation, so size the warehouse for concurrency. See Size for concurrency.