Skip to main content
Last updated on

Capture telemetry to the Lakehouse

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

Both Insights and Genie depend on an observability configuration: an object that sits above individual projects. That's deliberate: attaching one configuration to several projects lets you reuse a single setup and lets Insights show findings across all of them in one workspace-wide list, instead of every project owner configuring their own from scratch.

For what Insights and Genie do with the telemetry after it's flowing, see AI-assisted troubleshooting.

Prerequisites​

  • A Lakebase Autoscaling project on AWS or Azure. See Get a Postgres database.
  • A Unity Catalog catalog and schema to receive telemetry. The schema must already exist, and the catalog can't use Default storage.
  • A decision about the identity that runs the telemetry export. Lakebase writes telemetry to Unity Catalog as this identity, so it determines which destination catalogs and schemas the export can write to and where the Workspace folder of Insights findings is saved. Decide it before you start: a shared service principal is the durable choice for a configuration you'll reuse or mark as default, while your own user credentials are faster for testing. You set it in Step 1, and if you use a service principal it needs the permissions in Grant the service principal access.

Grant the service principal access​

If you choose Use service principal as the identity in Step 1, grant the principal the permissions below before you rely on the configuration. They're independent, and each is granted in a different place. The configuration saves whether or not they're in place, and if any is missing, telemetry silently never lands and no insights are generated, with no error to point you at the cause. If a configuration you've set up produces no telemetry, check each of them here first.

Grant CAN USE and Workspace access in Settings > Identity and access > Service principals, then select the principal.

Workspace access​

On the principal's Configurations tab, under Entitlements, select Workspace access. This lets the service principal act in the workspace at all. Without it, Lakebase writes telemetry as an identity that can't do anything, so every telemetry table stays empty even when everything else is set up correctly.

Workspace admin​

Add the service principal to the workspace admins group. In Settings > Identity and access > Groups, open the admins group and add the principal as a member. The background agent currently requires admin access to write telemetry and generate insights. Without it, the pipeline runs but produces nothing.

note

Requiring admin access is a current Beta constraint and may be narrowed in a later release. Grant it for now so that telemetry and insights work.

CAN USE on the service principal​

On the principal's Permissions tab, grant CAN USE. This is what lets you select the principal in the configuration wizard. Creating or managing a service principal grants CAN MANAGE, which doesn't include CAN USE, so you have to grant it explicitly. See Roles for managing service principals.

Unity Catalog privileges on the destination​

Grant the service principal the privileges it needs to create and write the telemetry tables in the destination catalog and schema:

  • USE CATALOG on the catalog
  • USE SCHEMA on the schema
  • CREATE TABLE on the schema

You can grant these in Catalog Explorer or with SQL:

SQL
GRANT USE CATALOG ON CATALOG <catalog> TO `<service-principal-id>`;
GRANT USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<service-principal-id>`;
GRANT CREATE TABLE ON SCHEMA <catalog>.<schema> TO `<service-principal-id>`;

Create the configuration​

Open the wizard from either of two places:

  • From Lakebase > Insights > Manage configurations > Add new config
  • From a project's own Settings > Observability config > Create new config

Both open the same wizard. Complete its three steps in order, clicking Next between them.

Step 1: Configure​

Name the configuration and select where its telemetry lands. Two choices here are worth pausing on:

  • Make this a default config for new projects automatically assigns this configuration to every new project created in the workspace, with no action from the project's creator. Turn it on only when you want one workspace-wide default. Otherwise, leave it off and assign the configuration to specific projects in Step 3.
  • Identity (under Authorize Lakebase to write your data) is the account Lakebase writes telemetry as. Use service principal is the durable choice: it doesn't depend on any one person's account, so it's right for a configuration you'll share across projects or mark as default. Use user-level credentials is faster for testing on your own. It relies on your existing permissions, so there's no principal to provision, but it's scoped to your access and isn't meant to outlive your testing.

If you select Use service principal, it must have the permissions described in Grant the service principal access. Only workspace admins can select a service principal. Other users select Use user-level credentials.

Under Send telemetry to Unity Catalog, choose the Catalog and Schema the tables land in, and optionally a table prefix. You're choosing a destination, not designing one: Lakebase creates the tables listed in How it works the first time it writes telemetry. The schema must already exist, and the catalog can't use Default storage. Because Use default storage is selected by default when you create a catalog in Catalog Explorer, be sure to clear it for the catalog you use here.

Step 2: Enable insights​

Make sure Enable insights is on. It's what runs the background Insights agent. Without it, a configuration writes telemetry to Unity Catalog but surfaces nothing proactively. Two related fields appear:

  • Instructions is free text the agent reads before each analysis pass, for example "Don't generate insights about table size." Treat it as guidance, not a guaranteed filter: like any instruction given to an LLM, it's usually followed, not strictly enforced.
  • Analysis frequency controls how often Insights reasons over your telemetry (for example, every hour). This is separate from how quickly raw telemetry lands in Unity Catalog, which happens continuously regardless.

Insights saves its findings to a Workspace location, distinct from the Unity Catalog destination above: raw telemetry lands as Delta tables, but the insights themselves are files in your own Databricks Workspace.

Step 3: Assign to projects​

Select the Lakebase projects this configuration applies to, then save.

After saving, confirm the compute is running. If you enabled observability on an instance that was already running, restart the instance for the change to take effect. During the Beta, telemetry doesn't begin flowing on an already-running instance until it restarts.

Next steps​

On this page