Investigate with Genie

Preview

Lakewatch is in Private Preview. To request access, reach out to your Databricks account team.

Databricks Genie brings natural-language data assistance to Lakewatch. You'll mainly use two parts of the Genie family:

• Genie Code is an AI assistant embedded across the Lakewatch UI. Use it for quick, context-aware help: writing SQL, building detection rules, and exploring datasources. Click Genie in the left-hand menu, or the Genie icon in the upper-right corner, to open it alongside any page.
• Genie Agents (formerly Genie spaces) are dedicated environments for deep, multi-step investigations powered by the Genie Research Agent.

This page covers using both for Lakewatch security workflows. For what Genie is, how it's billed, and general setup, follow the links above.

Triage alerts with Genie

Genie can triage a security alert for you from the alert's detail page. When you select Triage with Genie on an alert, Genie works like a senior analyst: it builds an investigation plan, gathers the relevant context and evidence, reaches a verdict, and updates the alert with its findings.

The Triage with Genie button is available on security alerts only. It doesn't appear on system errors.

How Genie triages an alert

When you ask Genie to triage a security alert, it works through the following phases:

1. Plans the investigation. Genie states a short plan before it runs anything, including a few benign explanations to test alongside the malicious one, so you can follow its reasoning.
2. Gathers context. Genie reads the detection rule that produced the alert, reviews how often that rule has fired before to gauge how noisy it is, and examines the signals on the alert.
3. Investigates the evidence. Genie runs queries against your security data to test each explanation, pivoting on the entities in the alert, such as users, IP addresses, and hosts.
4. Reaches a verdict. Genie classifies the alert as a true positive, false positive, or benign positive, based on the evidence it gathered.
5. Updates the alert. Genie posts an investigation comment and sets the alert's classification, state, and assignee.

Triage an alert

To triage a security alert with Genie:

1. Go to Explore > Alerts and open the security alert you want to investigate.
2. Select Triage with Genie at the top of the alert. Genie opens and starts the investigation. If you have triaged this alert before, the button reads Resume triage instead and reopens the earlier investigation.
3. Review the plan Genie proposes and follow along as it gathers context and runs its investigation queries.
4. Review the verdict and the comment Genie adds to the alert. Genie sets the classification, state, and assignee based on its findings, so confirm they match your assessment before you close the alert.

What Genie produces

Genie records its investigation as a comment on the alert, structured for a reader who did not run the queries:

• Summary: The verdict and the primary root cause, in a few sentences.
• Assessment: The classification (true, false, or benign positive) and the resulting alert state.
• Risk-based evidence: A table of each finding that shaped the verdict, with the time frame and the data it came from.
• Detection quality assessment: An evaluation of the rule that fired, including any gap that caused a false positive and a recommended fix.
• Next steps: Recommended follow-up actions, separated into immediate and deferred work.

Genie also sets the alert's classification, state, and assignee, and saves the triage conversation to the alert so you can reopen it later to review or continue the investigation.

Limitations

• Triage runs on security alerts only. The Triage with Genie button isn't available on system errors.
• Genie needs a running SQL warehouse to read the alert's events and run its investigation queries.
• Genie pivots on the alert's signals and the entities they reference, such as users, IP addresses, and hosts.

Configure a Genie Agent for threat intel

The Genie Research Agent (RA) analyzes threat intelligence feeds and surfaces potential threats from your security data. This section walks through configuring a Genie Agent for Lakewatch threat-intel workflows.

Prerequisites

• Parsed tables ingested and available in your catalog
• A threat intel feed loaded into Databricks (for example, a TAXII/STIX feed mapped to a table)
• Research Agent enabled for your workspace

Step 1: Create a Genie Agent with your security data

Create a Genie Agent and connect the data you want Genie to query:

• All relevant parsed tables
• System tables like signals and operational_alerts
• Your threat intel table

note

A Genie Agent supports a maximum of 30 tables. For larger environments, create topic-specific agents (for example, "Endpoint RA").

Step 2: Turn on the Research Agent

In your new Genie Agent, switch the query mode from Chat to Research Agent. This enables Genie to perform multi-step analysis across your data rather than answering single questions.

Step 3: Add instructions

Good instructions help Genie understand your data model and where to look when answering questions. Go to Configure > Instructions > Text and add context covering the following areas.

General guidance:

• Focus queries on parsed tables unless the user explicitly asks about raw data.
• Lakewatch uses Apache Spark for running detection rules on a fixed schedule.

Core terminology:

• Signal: An alert generated when a detection rule matches its conditions.
• Entity: Users, hosts, applications, or network devices.
• Asset: Protected resources tracked for security status.
• Detection rule: A saved SQL query that searches event data for specific patterns.
• Datasource: A data pipeline that ingests security data from external sources (for example, AWS CloudTrail, Akamai WAF), existing Unity Catalog tables, or Lakeflow Connect connectors.

Schema context:

• Parsed tables carry your declared fields plus Lakewatch-managed columns (lw_id, lw_meta, lw_observables).
• Severity levels: 1=Informational, 2=Low, 3=Medium, 4=High, 5=Critical, 6=Fatal.

Threat intel feed context:

• The mitre_enterprise_attack_gold table contains recent threat feeds. Users can use this table to understand threats they should be considering.

tip

Explicitly naming the threat intel table in your instructions is important. Without this context, Genie might answer threat questions using other tables (like signals) rather than your intel feed.

Step 4: Query for threats

Once your agent is configured, ask Genie questions about your threat landscape.

If Genie doesn't reference your threat intel table automatically, either:

• Explicitly name the table in your question: "Using the mitre_enterprise_attack_gold table, what threats should I be concerned about?"
• Add instructions pointing Genie to the table for threat-related queries (see Step 3).

Example queries

Query	What Genie returns
"What threats should I be concerned about?"	Summary of active threats, pulling from signals
"Using the mitre_enterprise_attack_gold table, what threats should I be concerned about?"	Analysis of MITRE ATT&CK techniques, threat actor groups, and malware families from your intel feed
"Are there any new threats I should be considering or looking for?"	Recently updated attack patterns, emerging malware families, and active campaigns
"Find detection coverage gaps for command and control tactics."	Gap analysis across MITRE ATT&CK tactics
"Can you review all activity from IP 5.161.151.88?"	Investigation summary for a specific entity
"Walk me through what happened with the corporate S3 bucket defacement."	Phased attack summary with next steps

Query	What Genie returns
"What threats should I be concerned about?"	Summary of active threats, pulling from signals
"Using the mitre_enterprise_attack_gold table, what threats should I be concerned about?"	Analysis of MITRE ATT&CK techniques, threat actor groups, and malware families from your intel feed
"Are there any new threats I should be considering or looking for?"	Recently updated attack patterns, emerging malware families, and active campaigns
"Find detection coverage gaps for command and control tactics."	Gap analysis across MITRE ATT&CK tactics
"Can you review all activity from IP 5.161.151.88?"	Investigation summary for a specific entity
"Walk me through what happened with the corporate S3 bucket defacement."	Phased attack summary with next steps

Troubleshooting

Genie isn't using the threat intel table

Add explicit instructions in the Genie Agent configuration pointing to the table name and its purpose. Genie defaults to whichever table seems most relevant to the question. Without a hint, it might use signals instead.

The threat intel feed isn't in Databricks

The feed must be ingested and structured as a table before Genie can query it. A liquid clustered table mapping 1:1 to your TAXII file fields is the recommended approach.

Tips for best results

• Be specific in your prompts. Naming the table directly improves relevance.
• Use instructions to reduce friction. Once the threat intel table is documented in Genie's instructions, analysts can ask natural language questions without specifying table names.
• Iterate on instructions. If Genie consistently pulls from the wrong source, refine your instructions to be more explicit about which table to use for which type of question.
• For more on creating and tuning agents, see Create and manage a Genie Agent.

Genie Code example prompts

Genie Code can help you write SQL queries, build detection rules, explore datasources, and more. The example prompts below are organized by use case.

tip

To triage a security alert with Genie step by step, see Triage alerts with Genie.

General

Show general prompts

SQL queries

• "Execute this SQL query: SELECT * FROM my_table LIMIT 10"
• "Run a query to count rows in the authentication table"
• "Show me the schema of this table"
• "Query the last hour of network traffic data"

Table discovery and schema

• "Search for all tables with 'user' in the name"
• "Find tables in the security catalog"
• "What columns does the http_activity table have?"
• "Show me data types for all fields in this table"
• "List all available catalogs and schemas"

Asset search

• "Find Unity Catalog volumes with sensitive data"
• "Search for dashboards related to security monitoring"
• "Locate notebooks that query authentication logs"

Documentation search

• "Search documentation for Spark SQL date functions"
• "How do I use window functions in Databricks?"
• "Find documentation about Delta Lake time travel"

Detection

Use these prompts when building and managing detection rules.

Show detection prompts

Describe what you want to detect

• "I want to detect data exfiltration from our network"
• "Create a rule to find users accessing sensitive files they shouldn't"

Discover data sources

• "What tables contain authentication events?"
• "Show me sample data from the network_activity table"
• "What fields are available in the authentication table?"

Write detection SQL

Different detection logic types:

• "Write SQL to detect IPs with more than 100 failed logins in an hour" (threshold)
• "Detect users with unusual login counts compared to their baseline" (anomaly)
• "Detect when a user logs into a host for the first time" (first-time-seen)
• "Detect users logging in from multiple countries within 2 hours" (geo-anomaly)
• "Detect database queries running after midnight" (time-based)

Modify existing queries:

• "Add a filter to exclude service accounts"
• "Increase the threshold to reduce noise"
• "Combine this with after-hours filtering"

Test and debug queries

• "Run this detection query and show me the results"
• "The query returns no results, help me debug"
• "This query is too slow, help me optimize it"
• "This generates too many false positives, how can I tune it?"

Map to MITRE ATT&CK

• "What MITRE techniques apply to brute force attacks?"
• "Map this detection to the appropriate tactics and techniques"
• "Show me details about technique T1078"

Define signal output

• "Write a summary template for this alert"
• "What context fields should I include?"
• "Show me what the alert looks like with sample data"

Set up scheduling and compute

• "Configure this to run every 15 minutes"
• "Set up a daily compliance check at 2 AM"
• "Run this only during business hours to save costs"
• "Should I use automatic or dedicated compute?"

Work with existing rules

• "Show me the configuration for the 'Brute Force Login' rule"
• "List all detection rules in this workspace"
• "Modify the threshold in the existing rule"

Learn about detection concepts

• "Explain how threshold detection works"
• "What's the difference between anomaly and threshold detection?"
• "What are the cost implications of different schedules?"

Build multi-step workflows

• "Build a complete brute force detection from scratch"
• "Create a correlation between failed logins and privilege escalation"

Troubleshoot detections

• "Why isn't this detection triggering?"
• "Validate all field names exist in the schema"
• "The baseline is wrong, how do I rebuild it?"

Query

Use these prompts on the query page.

Show query prompts

Explore data

• "How many rows are in the raw table of datasource Okta?"
• "Show me the parsed table schema for AWS CloudTrail"
• "Compare row counts between the raw and parsed tables for datasource crowdstrike_events"
• "Show me the last 10 events from the Crowdstrike parsed table"

Refine results

• "Filter these results to only show events from the U.S."
• "Add a column that calculates risk score"
• "Remove rows where status is 'test'"
• "Group these results by user and count events"
• "Sort by the timestamp column descending"

Optimize queries

• "This query is taking forever. How can I speed it up?"
• "It returned 5 million rows. How do I reduce that?"
• "Is this query efficient?"
• "How can I optimize this JOIN?"
• "Help me fix the query error"

Interpret results

• "What do these results tell me?"
• "Are there any patterns in this data?"
• "Summarize what I'm looking at"
• "What should I investigate based on this?"

Datasource

Use these prompts when configuring and managing datasources.

Show datasource prompts

Assess data quality

• "Show me a summary of my data quality"
• "Are there any columns with mostly null values?"
• "What's the distribution of values in the 'event_type' column?"
• "Analyze the raw schema and show me what columns exist"
• "Which columns have JSON data that needs extraction?"

Diagnose errors

• "Why did my datasource preview fail?"
• "The preview returned an error, help me debug"
• "I'm getting 'column does not exist' error, what's wrong?"

Extract JSON fields

• "Flatten the 'user' JSON object into separate columns"
• "Extract fields from the 'metadata' JSON column"
• "Preserve the original JSON column after extraction"
• "Add a prefix to extracted JSON fields"

Transform fields

• "Add a new field called 'is_admin' that's true when role contains 'admin'"
• "Rename the 'userId' column to 'user_id'"
• "Cast the 'timestamp' field to proper TIMESTAMP type"
• "Create a computed field for user full name from first and last name"

Filter data

• "Filter out internal IP addresses"
• "Only include records where status is 'success'"
• "Exclude records with null values in the 'user_id' field"

Join and enrich data

• "Join with our user directory to enrich with department info"
• "Add IP reputation data from an external reference table"
• "How do I configure a join with another Unity Catalog table?"

Create and modify tables

• "Create a new parsed table for only authentication events"
• "Split the raw data into separate tables by event_type"
• "Add a custom table with specific fields only"
• "Modify the existing 'security_events' table to add a new field"

Configure column preservation

• "Preserve all unreferenced columns from the raw table"
• "Embed unreferenced columns into a single column"
• "Omit columns that aren't explicitly mapped"

Validate and preview

• "Validate this field expression before I apply it"
• "Preview the transformation results"
• "Show me sample data after applying these changes"

Get recommendations

• "My data looks messy, what transformations do you recommend?"
• "Are there any data quality issues I should address?"
• "What's the best way to handle this high null rate column?"

Understand configuration options

• "Explain the difference between addTables and modifyTables"
• "What does 'unreferencedColumns.preserve' do?"
• "How do static joins work in enrichments?"

Additional resources

• Alerts
• Signals
• Detection rules
• Genie Agents
• Genie Code