Google Cloud audit logs

Preview

Lakewatch is in Private Preview. To request access, reach out to your Databricks account team.

Overview

Google Cloud Audit Logs record administrative activity, data access, and system events across your Google Cloud projects, folders, and organizations. Lakewatch ingests Google Cloud audit logs from cloud storage using Auto Loader and schematizes the data with the built-in gcp_audit_log parser. This is a cloud storage source, not an API connector.

Onboard Google Cloud audit logs

To ingest Google Cloud audit logs, create a cloud-storage datasource for it and attach the built-in gcp_audit_log parser so events are written to a schematized parsed table. For prerequisites, Unity Catalog access, and the full datasource setup steps, see Cloud storage.

Parsed schema and observables

The built-in gcp_audit_log parser schematizes Google Cloud audit logs into a parsed table. Every parsed table also carries the Lakewatch-managed columns:

• lw_id: A non-null string Lakewatch uses as the primary key, supporting deduplication across incremental reads.
• time: A non-null event timestamp assigned by the parser.
• lw_meta: Standard Lakewatch processing metadata.
• lw_observables: Values extracted from fields marked as security observables in the parser.

For more information, see the Parser reference.

Parsed table schema

The gcp_audit_log parser writes the following columns (generated from the parser definition):

Column	Data type
logName	STRING
severity	STRING
insertId	STRING
resource	STRUCT<type: STRING, labels: VARIANT>
time	TIMESTAMP
receiveTimestamp	TIMESTAMP
labels	VARIANT
operation	STRUCT<id: STRING, producer: STRING, first: BOOLEAN, last: BOOLEAN>
trace	STRING
httpRequest	STRUCT<requestMethod: STRING, requestURL: STRING, requestSize: BIGINT, status: SMALLINT, responseSize: BIGINT, userAgent: STRING, remoteIP: STRING, serverIP: STRING, referer: STRING, latency: STRING, cacheLookup: BOOLEAN, cacheHit: BOOLEAN, cacheValidatedWithOriginServer: BOOLEAN, cacheFillBytes: BIGINT, protocol: STRING>
spanId	STRING
traceSampled	BOOLEAN
sourceLocation	STRUCT<file: STRING, line: BIGINT, function: STRING>
protoPayload	STRUCT<@type: STRING, serviceName: STRING, methodName: STRING, resourceName: STRING, numResponseItems: BIGINT, status: STRUCT<code: INT, message: STRING, details: VARIANT>, authenticationInfo: STRUCT<principalSubject: STRING, serviceAccountKeyName: STRING, principalEmail: STRING, authoritySelector: STRING, thirdPartyPrincipal: VARIANT, serviceAccountDelegationInfo: ARRAY<STRUCT<firstPartyPrincipal: STRUCT<principalEmail: STRING, serviceMetadata: VARIANT>, thirdPartyPrincipal: STRUCT<thirdPartyClaims: VARIANT>, principalSubject: STRING>>>, authorizationInfo: ARRAY<STRUCT<resource: STRING, permission: STRING, granted: BOOLEAN, resourceAttributes: STRUCT<service: STRING, name: STRING, type: STRING, labels: STRING, uid: STRING>>>, requestMetadata: STRUCT<callerIP: STRING, callerSuppliedUserAgent: STRING, callerNetwork: STRING, requestAttributes: VARIANT, destinationAttributes: VARIANT>, resourceLocation: STRUCT<currentLocations: ARRAY<STRING>, originalLocations: ARRAY<STRING>>, request: VARIANT, response: VARIANT, metadata: VARIANT, serviceData: VARIANT>

Column	Data type
logName	STRING
severity	STRING
insertId	STRING
resource	STRUCT<type: STRING, labels: VARIANT>
time	TIMESTAMP
receiveTimestamp	TIMESTAMP
labels	VARIANT
operation	STRUCT<id: STRING, producer: STRING, first: BOOLEAN, last: BOOLEAN>
trace	STRING
httpRequest	STRUCT<requestMethod: STRING, requestURL: STRING, requestSize: BIGINT, status: SMALLINT, responseSize: BIGINT, userAgent: STRING, remoteIP: STRING, serverIP: STRING, referer: STRING, latency: STRING, cacheLookup: BOOLEAN, cacheHit: BOOLEAN, cacheValidatedWithOriginServer: BOOLEAN, cacheFillBytes: BIGINT, protocol: STRING>
spanId	STRING
traceSampled	BOOLEAN
sourceLocation	STRUCT<file: STRING, line: BIGINT, function: STRING>
protoPayload	STRUCT<@type: STRING, serviceName: STRING, methodName: STRING, resourceName: STRING, numResponseItems: BIGINT, status: STRUCT<code: INT, message: STRING, details: VARIANT>, authenticationInfo: STRUCT<principalSubject: STRING, serviceAccountKeyName: STRING, principalEmail: STRING, authoritySelector: STRING, thirdPartyPrincipal: VARIANT, serviceAccountDelegationInfo: ARRAY<STRUCT<firstPartyPrincipal: STRUCT<principalEmail: STRING, serviceMetadata: VARIANT>, thirdPartyPrincipal: STRUCT<thirdPartyClaims: VARIANT>, principalSubject: STRING>>>, authorizationInfo: ARRAY<STRUCT<resource: STRING, permission: STRING, granted: BOOLEAN, resourceAttributes: STRUCT<service: STRING, name: STRING, type: STRING, labels: STRING, uid: STRING>>>, requestMetadata: STRUCT<callerIP: STRING, callerSuppliedUserAgent: STRING, callerNetwork: STRING, requestAttributes: VARIANT, destinationAttributes: VARIANT>, resourceLocation: STRUCT<currentLocations: ARRAY<STRING>, originalLocations: ARRAY<STRING>>, request: VARIANT, response: VARIANT, metadata: VARIANT, serviceData: VARIANT>

Parser observables

The parser extracts the following fields into lw_observables (generated from the parser definition):

Field	Observable
httpRequest.requestURL	url
httpRequest.remoteIP	ip
httpRequest.serverIP	ip
httpRequest.referer	url
protoPayload.authenticationInfo.principalEmail	email
protoPayload.authenticationInfo.serviceAccountDelegationInfo[].firstPartyPrincipal.principalEmail	email
protoPayload.requestMetadata.callerIP	ip

Field	Observable
httpRequest.requestURL	url
httpRequest.remoteIP	ip
httpRequest.serverIP	ip
httpRequest.referer	url
protoPayload.authenticationInfo.principalEmail	email
protoPayload.authenticationInfo.serviceAccountDelegationInfo[].firstPartyPrincipal.principalEmail	email
protoPayload.requestMetadata.callerIP	ip

The following VARIANT fields are scanned for observables by matching nested key names. The table lists the observable types extracted from each:

Field	Observable types scanned
protoPayload.request	domain, email, ip, username
protoPayload.response	domain, email, ip, username
protoPayload.metadata	domain, email, ip, username

Field	Observable types scanned
protoPayload.request	domain, email, ip, username
protoPayload.response	domain, email, ip, username
protoPayload.metadata	domain, email, ip, username