AWS CloudTrail

Preview

Lakewatch is in Private Preview. To request access, reach out to your Databricks account team.

Overview

AWS CloudTrail records account activity and API usage across your AWS infrastructure. Lakewatch ingests CloudTrail logs from cloud storage (Amazon S3) using Auto Loader and schematizes them with built-in parsers. CloudTrail is a cloud storage source, not an API connector.

Onboard CloudTrail logs

To ingest CloudTrail, create a cloud-storage datasource that reads your CloudTrail logs from Amazon S3, and attach the built-in parser that matches the CloudTrail objects you're ingesting so events are written to a schematized parsed table. For prerequisites, Unity Catalog access, and the full datasource setup steps, see Cloud storage.

Parsed schema and observables

CloudTrail delivers three types of objects to Amazon S3, and Lakewatch provides a built-in parser for each. Attach the parser that matches the objects your datasource ingests:

aws_cloudtrail: Standard CloudTrail management, data, and network activity events.
aws_cloudtrail_digest: Digest files, which list the log files delivered in the previous hour with hash values for log-file integrity validation.
aws_cloudtrail_insight: CloudTrail Insights events, which record unusual API call-rate activity.

Every parsed table also carries the Lakewatch-managed columns:

lw_id: A non-null string Lakewatch uses as the primary key, supporting deduplication across incremental reads.
time: A non-null event timestamp.
lw_meta: Standard Lakewatch processing metadata.
lw_observables: Values extracted from fields marked as security observables in the parser.

For more information, see the Parser reference.

`aws_cloudtrail`

Schematizes standard CloudTrail events. Parses time from the eventTime field.

Parsed table schema

The aws_cloudtrail parser writes the following columns (generated from the parser definition):

Column	Data type
`additionalEventData`	`VARIANT`
`apiVersion`	`STRING`
`awsRegion`	`STRING`
`errorCode`	`STRING`
`errorMessage`	`STRING`
`eventID`	`STRING`
`eventName`	`STRING`
`eventSource`	`STRING`
`time`	`TIMESTAMP`
`eventType`	`STRING`
`eventVersion`	`STRING`
`managementEvent`	`BOOLEAN`
`readOnly`	`BOOLEAN`
`recipientAccountId`	`STRING`
`requestID`	`STRING`
`requestParameters`	`VARIANT`
`resources`	`ARRAY<STRUCT<arn: STRING, accountId: STRING, type: STRING>>`
`responseElements`	`VARIANT`
`serviceEventDetails`	`VARIANT`
`sharedEventID`	`STRING`
`sourceIPAddress`	`STRING`
`userAgent`	`STRING`
`userIdentity`	STRUCT<type: STRING, principalId: STRING, arn: STRING, accountId: STRING, accessKeyId: STRING, userName: STRING, sessionContext: STRUCT<attributes: STRUCT<mfaAuthenticated: STRING, creationDate: STRING>, sessionIssuer: STRUCT<type: STRING, principalId: STRING, arn: STRING, accountId: STRING, userName: STRING>, webIdFederationData: STRUCT<federatedProvider: STRING, attributes: VARIANT>, ec2RoleDelivery: STRING, sourceIdentity: STRING>, invokedBy: STRING, identityProvider: STRING, onBehalfOf: STRUCT<userId: STRING, identityStoreArn: STRING>, inScopeOf: STRUCT<sourceArn: STRING, sourceAccount: STRING, issuerType: STRING, credentialsIssuedTo: STRING>, credentialId: STRING>
`vpcEndpointId`	`STRING`
`eventCategory`	`STRING`
`sessionCredentialFromConsole`	`BOOLEAN`
`edgeDeviceDetails`	`VARIANT`
`tlsDetails`	`STRUCT<tlsVersion: STRING, cipherSuite: STRING, clientProvidedHostHeader: STRING>`
`addendum`	`STRUCT<reason: STRING, updatedFields: STRING, originalRequestID: STRING, originalEventID: STRING>`

Column	Data type
`additionalEventData`	`VARIANT`
`apiVersion`	`STRING`
`awsRegion`	`STRING`
`errorCode`	`STRING`
`errorMessage`	`STRING`
`eventID`	`STRING`
`eventName`	`STRING`
`eventSource`	`STRING`
`time`	`TIMESTAMP`
`eventType`	`STRING`
`eventVersion`	`STRING`
`managementEvent`	`BOOLEAN`
`readOnly`	`BOOLEAN`
`recipientAccountId`	`STRING`
`requestID`	`STRING`
`requestParameters`	`VARIANT`
`resources`	`ARRAY<STRUCT<arn: STRING, accountId: STRING, type: STRING>>`
`responseElements`	`VARIANT`
`serviceEventDetails`	`VARIANT`
`sharedEventID`	`STRING`
`sourceIPAddress`	`STRING`
`userAgent`	`STRING`
`userIdentity`	STRUCT<type: STRING, principalId: STRING, arn: STRING, accountId: STRING, accessKeyId: STRING, userName: STRING, sessionContext: STRUCT<attributes: STRUCT<mfaAuthenticated: STRING, creationDate: STRING>, sessionIssuer: STRUCT<type: STRING, principalId: STRING, arn: STRING, accountId: STRING, userName: STRING>, webIdFederationData: STRUCT<federatedProvider: STRING, attributes: VARIANT>, ec2RoleDelivery: STRING, sourceIdentity: STRING>, invokedBy: STRING, identityProvider: STRING, onBehalfOf: STRUCT<userId: STRING, identityStoreArn: STRING>, inScopeOf: STRUCT<sourceArn: STRING, sourceAccount: STRING, issuerType: STRING, credentialsIssuedTo: STRING>, credentialId: STRING>
`vpcEndpointId`	`STRING`
`eventCategory`	`STRING`
`sessionCredentialFromConsole`	`BOOLEAN`
`edgeDeviceDetails`	`VARIANT`
`tlsDetails`	`STRUCT<tlsVersion: STRING, cipherSuite: STRING, clientProvidedHostHeader: STRING>`
`addendum`	`STRUCT<reason: STRING, updatedFields: STRING, originalRequestID: STRING, originalEventID: STRING>`

Parser observables

The parser extracts the following fields into lw_observables (generated from the parser definition):

Field	Observable
`recipientAccountId`	`aws_account_id`
`resources[].arn`	`aws_arn`
`resources[].accountId`	`aws_account_id`
`sourceIPAddress`	`hostname`
`userIdentity.principalId`	`actor_id`
`userIdentity.arn`	`aws_arn`
`userIdentity.accountId`	`aws_account_id`
`userIdentity.accessKeyId`	`trace_id`
`userIdentity.userName`	`username`
`userIdentity.sessionContext.sessionIssuer.principalId`	`actor_id`
`userIdentity.sessionContext.sessionIssuer.arn`	`aws_arn`
`userIdentity.sessionContext.sessionIssuer.accountId`	`aws_account_id`
`userIdentity.sessionContext.sessionIssuer.userName`	`username`
`userIdentity.sessionContext.webIdFederationData.federatedProvider`	`aws_arn`
`userIdentity.onBehalfOf.identityStoreArn`	`aws_arn`
`userIdentity.inScopeOf.sourceArn`	`aws_arn`
`userIdentity.inScopeOf.sourceAccount`	`aws_account_id`

Field	Observable
`recipientAccountId`	`aws_account_id`
`resources[].arn`	`aws_arn`
`resources[].accountId`	`aws_account_id`
`sourceIPAddress`	`hostname`
`userIdentity.principalId`	`actor_id`
`userIdentity.arn`	`aws_arn`
`userIdentity.accountId`	`aws_account_id`
`userIdentity.accessKeyId`	`trace_id`
`userIdentity.userName`	`username`
`userIdentity.sessionContext.sessionIssuer.principalId`	`actor_id`
`userIdentity.sessionContext.sessionIssuer.arn`	`aws_arn`
`userIdentity.sessionContext.sessionIssuer.accountId`	`aws_account_id`
`userIdentity.sessionContext.sessionIssuer.userName`	`username`
`userIdentity.sessionContext.webIdFederationData.federatedProvider`	`aws_arn`
`userIdentity.onBehalfOf.identityStoreArn`	`aws_arn`
`userIdentity.inScopeOf.sourceArn`	`aws_arn`
`userIdentity.inScopeOf.sourceAccount`	`aws_account_id`

The following VARIANT fields are scanned for observables by matching nested key names. The observable types extracted from each:

Field	Observable types scanned
`additionalEventData`	`actor_id`, `aws_account_id`, `aws_arn`, `aws_instance_id`, `domain`, `ip`, `resource_tag`, `trace_id`, `username`
`requestParameters`	`actor_id`, `aws_account_id`, `aws_arn`, `aws_instance_id`, `domain`, `ip`, `resource_tag`, `trace_id`, `username`
`responseElements`	`actor_id`, `aws_account_id`, `aws_arn`, `aws_instance_id`, `domain`, `ip`, `resource_tag`, `trace_id`, `username`
`serviceEventDetails`	`actor_id`, `aws_account_id`, `aws_arn`, `aws_instance_id`, `domain`, `ip`, `resource_tag`, `trace_id`, `username`
`userIdentity.sessionContext.webIdFederationData.attributes`	`actor_id`, `aws_account_id`, `aws_arn`, `aws_instance_id`, `domain`, `ip`, `resource_tag`, `trace_id`, `username`

Field	Observable types scanned
`additionalEventData`	`actor_id`, `aws_account_id`, `aws_arn`, `aws_instance_id`, `domain`, `ip`, `resource_tag`, `trace_id`, `username`
`requestParameters`	`actor_id`, `aws_account_id`, `aws_arn`, `aws_instance_id`, `domain`, `ip`, `resource_tag`, `trace_id`, `username`
`responseElements`	`actor_id`, `aws_account_id`, `aws_arn`, `aws_instance_id`, `domain`, `ip`, `resource_tag`, `trace_id`, `username`
`serviceEventDetails`	`actor_id`, `aws_account_id`, `aws_arn`, `aws_instance_id`, `domain`, `ip`, `resource_tag`, `trace_id`, `username`
`userIdentity.sessionContext.webIdFederationData.attributes`	`actor_id`, `aws_account_id`, `aws_arn`, `aws_instance_id`, `domain`, `ip`, `resource_tag`, `trace_id`, `username`

`aws_cloudtrail_digest`

Schematizes CloudTrail digest files, which support log-file integrity validation. Parses time from the digestEndTime field.

Parsed table schema

The aws_cloudtrail_digest parser writes the following columns (generated from the parser definition):

Column	Data type
`awsAccountId`	`STRING`
`digestStartTime`	`TIMESTAMP`
`digestS3Bucket`	`STRING`
`digestS3Object`	`STRING`
`newestEventTime`	`TIMESTAMP`
`oldestEventTime`	`TIMESTAMP`
`previousDigestS3Bucket`	`STRING`
`previousDigestS3Object`	`STRING`
`previousDigestHashValue`	`STRING`
`previousDigestHashAlgorithm`	`STRING`
`previousDigestSignature`	`STRING`
`digestPublicKeyFingerprint`	`STRING`
`digestSignatureAlgorithm`	`STRING`
`logFiles`	`ARRAY<STRUCT<s3Bucket: STRING, s3Object: STRING, hashValue: STRING, hashAlgorithm: STRING, newestEventTime: TIMESTAMP, oldestEventTime: TIMESTAMP>>`
`time`	`TIMESTAMP`

Column	Data type
`awsAccountId`	`STRING`
`digestStartTime`	`TIMESTAMP`
`digestS3Bucket`	`STRING`
`digestS3Object`	`STRING`
`newestEventTime`	`TIMESTAMP`
`oldestEventTime`	`TIMESTAMP`
`previousDigestS3Bucket`	`STRING`
`previousDigestS3Object`	`STRING`
`previousDigestHashValue`	`STRING`
`previousDigestHashAlgorithm`	`STRING`
`previousDigestSignature`	`STRING`
`digestPublicKeyFingerprint`	`STRING`
`digestSignatureAlgorithm`	`STRING`
`logFiles`	`ARRAY<STRUCT<s3Bucket: STRING, s3Object: STRING, hashValue: STRING, hashAlgorithm: STRING, newestEventTime: TIMESTAMP, oldestEventTime: TIMESTAMP>>`
`time`	`TIMESTAMP`

Parser observables

The parser extracts the following fields into lw_observables (generated from the parser definition):

Field	Observable
`awsAccountId`	`aws_account_id`
`previousDigestHashValue`	`hash`
`logFiles[].hashValue`	`hash`

`aws_cloudtrail_insight`

Schematizes CloudTrail Insights events, which record unusual API call-rate activity. Parses time from the eventTime field.

Parsed table schema

The aws_cloudtrail_insight parser writes the following columns (generated from the parser definition):

Column	Data type
`eventVersion`	`STRING`
`awsRegion`	`STRING`
`eventID`	`STRING`
`eventType`	`STRING`
`recipientAccountId`	`STRING`
`sharedEventID`	`STRING`
`insightDetails`	`STRUCT<state: STRING, eventSource: STRING, eventName: STRING, insightType: STRING, insightContext: STRUCT<statistics: STRUCT<baseline: STRUCT<average: DOUBLE>, insight: STRUCT<average: DOUBLE>, insightDuration: DOUBLE>>>`
`eventCategory`	`STRING`
`time`	`TIMESTAMP`

Column	Data type
`eventVersion`	`STRING`
`awsRegion`	`STRING`
`eventID`	`STRING`
`eventType`	`STRING`
`recipientAccountId`	`STRING`
`sharedEventID`	`STRING`
`insightDetails`	`STRUCT<state: STRING, eventSource: STRING, eventName: STRING, insightType: STRING, insightContext: STRUCT<statistics: STRUCT<baseline: STRUCT<average: DOUBLE>, insight: STRUCT<average: DOUBLE>, insightDuration: DOUBLE>>>`
`eventCategory`	`STRING`
`time`	`TIMESTAMP`

Parser observables

The parser extracts the following fields into lw_observables (generated from the parser definition):

Field	Observable
`recipientAccountId`	`aws_account_id`
`sharedEventID`	`trace_id`

Overview​

Onboard CloudTrail logs​

Parsed schema and observables​

aws_cloudtrail​

aws_cloudtrail_digest​

aws_cloudtrail_insight​

Overview

Onboard CloudTrail logs

Parsed schema and observables

`aws_cloudtrail`

`aws_cloudtrail_digest`

`aws_cloudtrail_insight`