Skip to main content

Build applications and agents on Lakehouse Real-Time

Beta

Lakehouse Real-Time (Lakehouse//RT) is in Beta. Reach out to your Databricks account team to enable this feature in your account.

Lakehouse//RT is in active development. Performance characteristics and the supported feature set can change.

Connect a custom application or AI agent to a Lakehouse//RT warehouse to serve sub-second query results from your Unity Catalog tables. Applications and agents query the tables in place, so they don't need a separate serving database that holds a copy of your data.

Requirements​

To connect an application or agent to Lakehouse//RT, you need:

Connect from an application​

Lakehouse//RT accepts connections only through the Statement Execution API. Call the API directly, or use a driver configured for it. For the list of drivers and their settings, see Connectivity.

To connect from a Python application with the Databricks SQL Connector for Python:

  1. Install the connector with the kernel extra. The extra requires Python 3.10 or above. Version 4.9.0 and later detects a Lakehouse//RT warehouse automatically when the kernel extra is installed.

    Shell
    pip install "databricks-sql-connector[kernel]"
  2. Set the following environment variables:

    • DATABRICKS_SERVER_HOSTNAME: the Server Hostname value for the warehouse. See Get connection details for a Databricks compute resource.
    • DATABRICKS_HTTP_PATH: the HTTP Path value for the warehouse.
    • DATABRICKS_CLIENT_ID: the service principal's application ID.
    • DATABRICKS_CLIENT_SECRET: the service principal's OAuth secret.
  3. Connect and run parameterized queries. The following example sets use_kernel=True to select the Statement Execution API path explicitly, which also works on versions earlier than 4.9.0. It authenticates as the service principal, sets a statement timeout for the session, and passes user input as a query parameter instead of building the SQL string:

    Python
    import os
    from databricks import sql

    connection = sql.connect(
    server_hostname=os.getenv("DATABRICKS_SERVER_HOSTNAME"),
    http_path=os.getenv("DATABRICKS_HTTP_PATH"),
    oauth_client_id=os.getenv("DATABRICKS_CLIENT_ID"),
    oauth_client_secret=os.getenv("DATABRICKS_CLIENT_SECRET"),
    use_kernel=True,
    session_configuration={"STATEMENT_TIMEOUT": "30"}, # seconds
    )

    def get_account_summary(account_id):
    with connection.cursor() as cursor:
    cursor.execute(
    "SELECT region, sum(revenue) AS revenue "
    "FROM main.sales.account_daily "
    "WHERE account_id = ? GROUP BY region",
    parameters=[account_id],
    )
    return cursor.fetchall()

    Calling get_account_summary returns the matching rows, and the query appears in the warehouse's query history.

    With use_kernel=True, pass the service principal's credentials as oauth_client_id and oauth_client_secret. The connector doesn't accept a custom credentials_provider on this path.

Reuse connections​

Create the connection one time when your application starts, and reuse it for every request instead of connecting for each query. Opening a new connection for each request adds latency to every query. Open a cursor for each query and close it when the query finishes.

Set a statement timeout​

Set a statement timeout so that a slow query fails fast instead of holding a request open. A query that runs longer than the statement timeout returns a QUERY_EXECUTION_TIMEOUT_EXCEEDED error. The default timeout is 1 minute. Set it for the session with the STATEMENT_TIMEOUT parameter, which takes a value in seconds (the previous example sets 30 seconds), or with a SET STATEMENT_TIMEOUT statement. The value can't exceed the query runtime guardrail. See STATEMENT_TIMEOUT.

Query from an AI agent​

To give an AI agent fast access to your data, define a tool that the agent calls, and have the tool run SQL on the Lakehouse//RT warehouse through a driver or the Statement Execution API. Genie doesn't run on Lakehouse//RT. To set up the tool:

  1. Write a function that takes the agent's inputs as arguments and runs a parameterized query, like get_account_summary in the previous example.
  2. Register the function as a tool in your agent framework. See Define local Python function tools.
  3. Run the agent as a service principal that has SELECT only on the tables the tool needs.

Agents can send many queries for each conversation, so size the warehouse for concurrency. See Size for concurrency.

Additional resources​