Skip to main content
Last updated on

Walk through Lakewatch end to end

Beta

Lakewatch is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage workspace-level previews.

This guide walks through the core Lakewatch workflow end-to-end, from ingesting data to running detection rules, investigating alerts, and monitoring system health.

Before you begin​

Before you start

Lakewatch is set up one time for each workspace. The first time anyone opens Lakewatch, a guided setup flow creates its environment, which requires CREATE CATALOG on the metastore, or MANAGE and CREATE SCHEMA on an existing catalog. See Set up Lakewatch for your workspace.

You also need:

  • Access to the folders you work in. If Lakewatch was set up with the Recommended configuration, all account users already have access. Otherwise, ask an administrator to assign you permission groups on those folders.
  • A SQL warehouse selected in the warehouse picker in the top navigation bar. See SQL warehouse.

Ingest and schematize data​

Lakewatch ingests data using datasources, which combine an ingestion method (how raw data is read) with one or more parsers (which schematize the raw data into governed tables). You can create and manage datasources in the UI.

Ingest raw data using one of three methods:

  • Cloud storage: Point to files in a supported object store (for example, an S3 bucket or a /Volumes/ path).
  • Existing table: Ingest a Unity Catalog table or view that already exists in your workspace, without copying the data.
  • API connectors: Use an API connector to pull data directly from software as a service (SaaS) applications like CrowdStrike, Okta, and others.

Schematize the raw data with parsers. Each parser describes the output columns of a log source and writes a parsed table with the Lakewatch-managed lw_id, lw_meta, and lw_observables columns. Incoming events are classified to the first matching parser; unmatched events are routed to a dead-letter sentinel. The parsed tables are ready for security analysis with detection rules and queries.

To learn more, see Create and manage datasources.

Create and run detection rules​

Go to Detection rules to create detection rules. You can categorize rules using the MITRE ATT&CK framework or a custom framework, set a processing schedule, and define signal output.

Lakewatch detection rules are authored in SQL or SQL pipe syntax.

To learn more, see Detection rules.

Investigate alerts​

Detection rules generate signals when events match their criteria. Lakewatch automatically correlates related signals into alerts. Go to Alerts to view, assign, and manage alerts. Open an alert to review its signals and the detection rules that triggered them.

You can also export alerts to ServiceNow, Tines, a webhook, or Jira, or triage an alert with Genie.

To learn more, see Alerts.

Query data​

Go to Query to run SQL queries over your security data. The query page supports time-range filtering, no-code refinements, a timeline visualization, and detailed row inspection.

To learn more, see Query data.

Investigate with Genie​

Genie can help you investigate security events using natural language.

  • Genie Code: The AI assistant available across the Lakewatch UI. Use it to write SQL queries, build detection rules, explore datasources, and more. Click Genie in the left-hand menu, or click Genie Code in the top bar to open it alongside any page. To learn more, see Genie Code example prompts.

  • Genie Agents: A dedicated environment for deep investigative work with the Genie Research Agent. Connect your parsed tables and the signals and system_events views in your catalog's __lakewatch schema to a Genie Agent, then use multi-step reasoning, SQL generation, and MITRE ATT&CK mapping to investigate threats. To learn more, see Configure a Genie Agent for threat intel.

Monitor system health​

When a detection rule or datasource fails, or Lakewatch finds a configuration problem in one, Lakewatch opens a system error and groups repeated failures into it. Go to System errors to view, assign, and manage system errors, and open a system error's Errors tab to see each failure.

To learn more, see System errors.

The Overview page​

The Overview page is your Lakewatch homepage, summarizing ingestion, detection, and alert activity with a fast path into Genie.

On this page