AWS GuardDuty

Preview

Lakewatch is in Private Preview. To request access, reach out to your Databricks account team.

Overview

AWS GuardDuty is a threat detection service that continuously monitors your AWS accounts and workloads for malicious activity and unauthorized behavior. Lakewatch ingests AWS GuardDuty from cloud storage using Auto Loader and schematizes the data with the built-in aws_guardduty parser. This is a cloud storage source, not an API connector.

Onboard AWS GuardDuty

To ingest AWS GuardDuty, create a cloud-storage datasource for it and attach the built-in aws_guardduty parser so events are written to a schematized parsed table. For prerequisites, Unity Catalog access, and the full datasource setup steps, see Cloud storage.

Parsed schema and observables

The built-in aws_guardduty parser schematizes AWS GuardDuty into a parsed table. Every parsed table also carries the Lakewatch-managed columns:

• lw_id: A non-null string Lakewatch uses as the primary key, supporting deduplication across incremental reads.
• time: A non-null event timestamp assigned by the parser.
• lw_meta: Standard Lakewatch processing metadata.
• lw_observables: Values extracted from fields marked as security observables in the parser.

For more information, see the Parser reference.

Parsed table schema

The aws_guardduty parser writes the following columns (generated from the parser definition):

Column	Data type
schemaVersion	STRING
accountId	STRING
region	STRING
partition	STRING
id	STRING
arn	STRING
type	STRING
resource	VARIANT
severity	DOUBLE
createdAt	TIMESTAMP
time	TIMESTAMP
title	STRING
description	STRING
service	STRUCT<additionalInfo: VARIANT, action: VARIANT, serviceName: STRING, detectorId: STRING, resourceRole: STRING, eventFirstSeen: TIMESTAMP, eventLastSeen: TIMESTAMP, archived: BOOLEAN, count: BIGINT, featureName: STRING, runtimeDetails: VARIANT, detection: VARIANT>
associatedAttackSequenceArn	STRING

Column	Data type
schemaVersion	STRING
accountId	STRING
region	STRING
partition	STRING
id	STRING
arn	STRING
type	STRING
resource	VARIANT
severity	DOUBLE
createdAt	TIMESTAMP
time	TIMESTAMP
title	STRING
description	STRING
service	STRUCT<additionalInfo: VARIANT, action: VARIANT, serviceName: STRING, detectorId: STRING, resourceRole: STRING, eventFirstSeen: TIMESTAMP, eventLastSeen: TIMESTAMP, archived: BOOLEAN, count: BIGINT, featureName: STRING, runtimeDetails: VARIANT, detection: VARIANT>
associatedAttackSequenceArn	STRING

Parser observables

The parser extracts the following fields into lw_observables (generated from the parser definition):

Field	Observable
accountId	aws_account_id
arn	aws_arn
associatedAttackSequenceArn	aws_arn

Field	Observable
accountId	aws_account_id
arn	aws_arn
associatedAttackSequenceArn	aws_arn

The following VARIANT fields are scanned for observables by matching nested key names; the observable types extracted from each:

Field	Observable types scanned
resource	actor_id, aws_account_id, aws_arn, aws_instance_id, domain, ip, resource_tag, trace_id, username
service.additionalInfo	actor_id, aws_account_id, aws_arn, aws_instance_id, domain, ip, resource_tag, trace_id, username
service.action	actor_id, aws_account_id, aws_arn, aws_instance_id, domain, ip, resource_tag, trace_id, username
service.detection	actor_id, aws_account_id, aws_arn, aws_instance_id, domain, ip, resource_tag, trace_id, username

Field	Observable types scanned
resource	actor_id, aws_account_id, aws_arn, aws_instance_id, domain, ip, resource_tag, trace_id, username
service.additionalInfo	actor_id, aws_account_id, aws_arn, aws_instance_id, domain, ip, resource_tag, trace_id, username
service.action	actor_id, aws_account_id, aws_arn, aws_instance_id, domain, ip, resource_tag, trace_id, username
service.detection	actor_id, aws_account_id, aws_arn, aws_instance_id, domain, ip, resource_tag, trace_id, username