Skip to main content
Last updated on

Introduction to Lakewatch

Preview

Lakewatch is in Private Preview. To request access, reach out to your Databricks account team.

Lakewatch is a Databricks feature that brings security information and event management (SIEM) capabilities directly into your lakehouse. It helps security teams detect threats, investigate incidents, and respond to security events.

What does Lakewatch do?​

Lakewatch gives security teams a single place to:

  • Ingest security data from sources like CrowdStrike, Okta, AWS CloudTrail, and more, using managed connectors, cloud storage paths, or existing tables in your workspace.
  • Schematize data into governed tables with a consistent set of standard fields, using parsers, so you can write detection rules and queries that work across different sources of data.
  • Detect threats with SQL-based detection rules that run on a schedule.
  • Investigate incidents through alerts that automatically group related signals, track entity risk scores, and link to the events and entities involved.
  • Query security data with SQL directly in the Lakewatch UI, with support for time-range filtering, no-code refinements, and detailed row inspection.
  • Investigate with AI using Genie, a context-aware AI assistant that helps you write queries, build detection rules, and investigate threats using natural language.

How it works​

Lakewatch uses datasources to bring security data into your workspace. A datasource combines an ingestion method (how raw data is read) with one or more parsers that schematize the raw data into governed Unity Catalog tables:

  • Raw data: Unprocessed data read from external sources by the datasource's ingestion method.
  • Parsed tables: Schematized tables written by the datasource's parsers. Each parser describes the output columns of a log source and adds the Lakewatch-managed lw_id, lw_meta, and lw_observables columns.

Parsed data is ready for security analysis:

  • Detection rules run SQL queries against your parsed tables on a schedule. When a rule's conditions match, it generates a signal: a record of what was detected, when it happened, and which entities were involved.
  • Lakewatch automatically correlates related signals into alerts for investigation. Analysts can assign, classify, and track alerts through resolution, and export them to Jira, ServiceNow, or Tines.
  • The query page lets you run SQL queries across your security data for ad hoc investigation, with support for time-range filtering, no-code refinements, and detailed row inspection.

Lakewatch also monitors its own pipelines and generates system events when it detects processing problems or configuration errors. These are tracked in separate system errors.

The Overview page​

When you open Lakewatch, you land on the Overview page, your homepage. It summarizes ingestion, detection, and alert activity across your workspace, and gives you a fast path into Genie for natural-language investigation. In a new workspace, the panels start empty and populate as you create datasources and detection rules, and as detections generate alerts.

Additional resources​

On this page