Skip to main content
Last updated on

Genie ZeroOps

Beta

This feature is in Beta. To use it, a workspace admin must turn on Genie ZeroOps from the Previews page. See Manage Databricks previews.

Genie ZeroOps is a background agent that monitors your jobs, pipelines, and tables, detects failures and data quality issues, identifies the root cause, and proposes a fix. It is part of the Genie family of AI experiences in Databricks. Unlike Genie Code, which responds when you prompt it, Genie ZeroOps runs continuously without a prompt and surfaces issues for you to act on.

To set up Genie ZeroOps and work with the insights it generates, see Get started with Genie ZeroOps.

What Genie ZeroOps does​

Data teams spend significant time finding the source of a failure or a data quality issue. A job fails, a table goes stale, or an upstream schema change breaks a downstream table, and an engineer traces the problem across lineage, pipeline graphs, and run logs to understand what happened.

Genie ZeroOps does this work for you. It runs in the background across the assets you are responsible for, following three steps to help you find and fix issues:

  1. Detection: Finds failing jobs and pipelines, jobs that succeed but produce stale or incomplete tables, and tables that are stale or missing data.
  2. Root cause analysis: For each issue, identifies the underlying cause, such as a code change, a schema change, or an upstream failure.
  3. Resolution: Proposes a fix and links you to the source, such as the notebook, where you make the change. For a job that runs code from a Git repository, Genie ZeroOps can also push the fix to a new branch when you ask, so that you can open a pull request.

How Genie ZeroOps monitors assets and reports insights​

You configure Genie ZeroOps on a workspace and select what it monitors, such as the jobs or schemas you own. It can monitor the jobs, pipelines, and tables that you have OWNER, CAN MANAGE, or CAN VIEW permission on. You can change this scope at any time.

Genie ZeroOps surfaces findings as insights. Insights appear in an inbox, where you review them, follow the proposed fix, and mark insights as resolved. You can also set up notifications for new insights.

Each insight contains:

  • A description of what was detected.
  • The root cause of the issue.
  • A proposed fix and a link to the source where you make the change.

Notifications​

Genie ZeroOps can notify you when an insight first appears in your inbox. It supports multiple destination types, such as Slack and PagerDuty. See Configure notifications.

note

Notifications send information to systems outside Databricks and contain AI-generated text with the insight details. Copies stored in external systems are outside Databricks retention and deletion controls.

Git integration​

A job can run code from a Git repository through a Git source or a bundle deployed from Git. When such a job fails, Genie ZeroOps reads the repository to find the root cause. When the fix is a code change, Genie ZeroOps prepares the change. When you ask, it pushes the change to a new branch so that you can open a pull request. See Create a pull request from a fix.

Genie ZeroOps reads the repository with your Git credential. To add a Git credential, see Add Git credentials.

Data processing​

When you select the assets for Genie ZeroOps to monitor, you are choosing to have Databricks process them using a large language model on a continuous, autonomous basis until you disable Genie ZeroOps. For a job that runs code from a Git repository, the data that Genie ZeroOps processes includes the contents of the repository. During investigations, Genie ZeroOps may also query tables that you have permission to access.

Genie ZeroOps is powered by Anthropic and OpenAI models on Databricks, using endpoints hosted by Databricks within the Databricks security perimeter.

Permissions and visibility​

Genie ZeroOps produces insights only for assets that the configuring user has access to. By default, those insights are visible only to the user who configures Genie ZeroOps.

You can share your Genie ZeroOps configuration with other users in your workspace so that they can work on its insights. Users you share with see the code excerpts, suggested fixes, and sample data rows included in those insights, even if they don't have access to the underlying assets. See Share your Genie ZeroOps configuration.

Guardrails​

Genie ZeroOps doesn't change production data or code without human review. It proposes fixes and links you to the source, and you apply the fix yourself, prompt Genie Code, or, for jobs that run code from a Git repository, ask Genie ZeroOps to push it.

For a job that runs code from a Git repository, Genie ZeroOps doesn't write to your repository in the background. It pushes a fix only after you explicitly ask it to and confirm the push. Genie ZeroOps then commits the fix and pushes it to a new branch. It gives you a link to open a pull request in your Git provider, and the change goes through your regular review process.

Limitations​

  • When you set up Genie ZeroOps, the resulting configuration belongs to your account. You can share it with up to 25 individual workspace users so that they can work on its insights. You can't share it with groups or service principals. See Share your Genie ZeroOps configuration.
  • Genie ZeroOps monitors jobs, pipelines, and tables. It does not monitor other asset types, such as models or apps.
  • Genie ZeroOps detects job and pipeline failures and data quality issues. It does not detect performance or cost regressions.
  • Genie ZeroOps operates at the workspace level, not the account level.
  • Genie ZeroOps is not HIPAA compliant.
  • Detecting data quality issues on your tables requires separately billed data quality monitoring.
On this page