Configure observability
This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.
An observability configuration captures deep Lakebase Postgres telemetry as Delta tables in Unity Catalog. Insights, Genie, and the dashboards all run on this data; a project has none of them until a configuration is attached.
A configuration is a single object that can cover one or many projects. There are two ways to attach an identity to it, and they coexist by design:
- A workspace admin sets a service-principal configuration as the workspace default, so every new project gets observability automatically.
- An individual project owner configures their own project with user-identity credentials.
A workspace typically ends up with a mix: most projects on the shared default, some with their own configs. This page covers the full setup for either path. To set up a single project you own, see the quickstart.
Prerequisites
- The Lakebase Advanced Postgres Telemetry preview enabled for your workspace. A workspace admin turns it on from the Previews page.
- A Lakebase Autoscaling project on AWS or Azure. See Get a Postgres database.
- A Unity Catalog catalog and schema to receive telemetry. The schema must already exist, and the catalog can't use Default storage.
Set up the configuration
Choose an identity. The configuration writes telemetry to Unity Catalog as an identity you select in the wizard. That choice determines how far the configuration reaches and how long it lasts:
- Choose a service principal for a durable configuration that isn't tied to any one person's account. This is the right fit for standardizing observability: set it as the workspace default, and assign it to existing projects. Selecting a service principal is workspace-admin-only, and you provision it first (step 1 below).
- Choose user-level credentials to configure a project under your own access, with nothing to provision. This suits an individual project you own. It's scoped to your permissions, so skip the service-principal provisioning step and start at Open the configuration wizard.
Provision the service principal
If you're using user-level credentials, skip to Open the configuration wizard.
Grant the service principal the four permissions below before you create the configuration, so it's ready to select in the wizard. They're independent, and each is granted in a different place. If any is missing, the configuration still saves, but telemetry silently never lands and no insights are generated, with no error to point you at the cause. If a configuration produces no telemetry, check these first.
Grant | Where to set it | Why it's needed |
|---|---|---|
Workspace access | The principal's Configurations tab, under Entitlements | Lets the service principal act in the workspace at all. Without it, Lakebase writes telemetry as an identity that can't do anything, so every telemetry table stays empty. |
Workspace admin | Settings > Identity and access > Groups: open the admins group and add the principal | The background agent currently requires admin access to write telemetry and generate insights. |
CAN USE | The principal's Permissions tab | Lets you select the principal in the wizard. Creating a service principal grants CAN MANAGE, which doesn't include CAN USE, so you grant it explicitly. See Roles for managing service principals. |
Unity Catalog privileges | The destination catalog and schema (see the SQL below) | Lets the principal create and write the telemetry tables: |
Grant Workspace access and CAN USE in Settings > Identity and access > Service principals. Grant the Unity Catalog privileges in Catalog Explorer or with SQL:
GRANT USE CATALOG ON CATALOG <catalog> TO `<service-principal-id>`;
GRANT USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<service-principal-id>`;
GRANT CREATE TABLE ON SCHEMA <catalog>.<schema> TO `<service-principal-id>`;
Requiring workspace admin is a current Beta constraint and may be narrowed in a later release. Grant it for now so that telemetry and insights work.
Open the configuration wizard
Open the wizard from either place. Both open the same wizard:
- From Lakebase > Insights > Manage configurations > Add new config
- From a project's own Settings > Observability config > Create new config
Complete the three wizard steps that follow in order, clicking Next between them.
Configure
Name the configuration and select where its telemetry lands. Two choices here are worth pausing on:
- Make this a default config for new projects makes this the workspace default: every new project created in the workspace is assigned this configuration automatically, with no action from its creator. Turn this on to standardize observability workspace-wide. A workspace has at most one default, and the default applies only to projects created afterward, so you also assign it to existing projects in the Assign to projects step. Leave it off if you want this configuration to apply to specific projects rather than the whole workspace.
- Identity (under Authorize Lakebase to write your data) is the account Lakebase writes telemetry as, following the choice you made in Choose an identity. Select Use service principal and choose the principal you provisioned, or select Use user-level credentials to run as yourself. Only workspace admins can select a service principal.
Under Send telemetry to Unity Catalog, choose the Catalog and Schema the tables land in, and optionally a table prefix. You're choosing a destination, not designing one: Lakebase creates the tables listed in What gets captured the first time it writes telemetry. The schema must already exist, and the catalog can't use Default storage. Because Use default storage is selected by default when you create a catalog in Catalog Explorer, be sure to clear it for the catalog you use here.
Enable insights
Make sure Enable insights is on. It's what runs the background Insights agent. Without it, a configuration writes telemetry to Unity Catalog but surfaces nothing proactively. Two related fields appear:
- Instructions is free text the agent reads before each analysis pass, for example "Don't generate insights about table size." Treat it as guidance, not a guaranteed filter: like any instruction given to an LLM, it's usually followed, not strictly enforced.
- Analysis frequency controls how often Insights evaluates your telemetry (for example, every hour). This is separate from how quickly raw telemetry lands in Unity Catalog, which happens continuously regardless.
Insights saves its findings to a Workspace location, distinct from the Unity Catalog destination above: raw telemetry lands as Delta tables, but the insights themselves are files in your own Databricks Workspace.
Assign to projects
Select the existing Lakebase projects this configuration applies to, then save. Setting it as the workspace default in the previous step covers projects created from now on; assigning it here brings your existing projects under the same configuration.
After saving:
- Confirm the compute is running.
- If you enabled observability on an instance that was already running, restart the instance for the change to take effect. During the Beta, telemetry doesn't begin flowing on an already-running instance until it restarts.
Next steps
- Find and resolve issues with Insights: turn on proactive monitoring and learn how to read what it finds.
- Diagnose and fix issues with Genie: investigate a problem conversationally and walk through a recovery end to end.