Azure audit logs

Preview

Lakewatch is in Private Preview. To request access, reach out to your Databricks account team.

Overview

Azure audit logs capture directory activity from Microsoft Entra ID (formerly Azure Active Directory), such as changes to users, groups, and applications. Lakewatch ingests Azure audit logs from cloud storage using Auto Loader and schematizes the data with the built-in azure_audit parser. This is a cloud storage source, not an API connector.

Onboard Azure audit logs

To ingest Azure audit logs, create a cloud-storage datasource for it and attach the built-in azure_audit parser so events are written to a schematized parsed table. For prerequisites, Unity Catalog access, and the full datasource setup steps, see Cloud storage.

Parsed schema and observables

The built-in azure_audit parser schematizes Azure audit logs into a parsed table. Every parsed table also carries the Lakewatch-managed columns:

• lw_id: A non-null string Lakewatch uses as the primary key, supporting deduplication across incremental reads.
• time: A non-null event timestamp assigned by the parser.
• lw_meta: Standard Lakewatch processing metadata.
• lw_observables: Values extracted from fields marked as security observables in the parser.

For more information, see the Parser reference.

Parsed table schema

The azure_audit parser writes the following columns (generated from the parser definition):

Column	Data type
Level	STRING
callerIpAddress	STRING
category	STRING
correlationId	STRING
durationMs	DOUBLE
identity	STRING
location	STRING
locationDetails	VARIANT
networkLocationDetails	VARIANT
operationName	STRING
operationVersion	STRING
time	TIMESTAMP
properties	STRUCT<aadTenantId: STRING, activityDateTime: TIMESTAMP, activityDisplayName: STRING, additionalDetails: ARRAY<STRUCT<key: STRING, value: STRING>>, alternateSignInName: STRING, appDisplayName: STRING, appliedConditionalAccessPolicies: VARIANT, appliedEventListeners: VARIANT, appId: STRING, appServicePrincipalId: STRING, authenticationAppDeviceDetails: VARIANT, authenticationStrengths: VARIANT, authenticationAppPolicyEvaluationDetails: VARIANT, authenticationContextClassReferences: VARIANT, authenticationDetails: VARIANT, authenticationMethodsUsed: VARIANT, authenticationProcessingDetails: VARIANT, authenticationProtocol: STRING, authenticationRequirement: STRING, authenticationRequirementPolicies: VARIANT, autonomousSystemNumber: STRING, _billedSize: DOUBLE, category: STRING, clientAppUsed: STRING, clientCredentialType: STRING, conditionalAccessAudiences: VARIANT, conditionalAccessPolicies: VARIANT, conditionalAccessStatus: STRING, correlationId: STRING, createdDateTime: TIMESTAMP, crossTenantAccessType: STRING, deviceDetail: VARIANT, federatedCredentialId: STRING, flaggedForReview: BOOLEAN, globalSecureAccessIpAddress: STRING, homeTenantId: STRING, homeTenantName: STRING, id: STRING, incomingTokenType: STRING, ipAddress: STRING, ipAddressFromResourceProvider: STRING, _isBillable: STRING, isDeleted: BOOLEAN, initiatedBy: STRUCT<app: STRUCT<displayName: STRING, servicePrincipalId: STRING, appId: STRING>, user: STRUCT<id: STRING, displayName: STRING, userPrincipalName: STRING, ipAddress: STRING, roles: VARIANT>>, isProcessing: BOOLEAN, loggedByService: STRING, location: VARIANT, networkLocationDetails: VARIANT, operationType: STRING, result: STRING, resultReason: STRING, isInteractive: BOOLEAN, isRisky: BOOLEAN, isTenantRestricted: BOOLEAN, isThroughGlobalSecureAccess: BOOLEAN, originalRequestId: STRING, originalTransferMethod: STRING, privateLinkDetails: VARIANT, processingTimeInMilliseconds: BIGINT, resource: STRING, resourceDisplayName: STRING, resourceGroup: STRING, resourceId: STRING, resourceIdentity: STRING, resourceProvider: STRING, resourceOwnerTenantId: STRING, resourceServicePrincipalId: STRING, resourceTenantId: STRING, riskEventTypes: VARIANT, riskEventTypes_v2: VARIANT, riskLastUpdatedDateTime: TIMESTAMP, riskDetail: STRING, riskLevel: STRING, riskLevelAggregated: STRING, riskLevelDuringSignIn: STRING, riskState: STRING, rngcStatus: STRING, servicePrincipalId: STRING, servicePrincipalCredentialKeyId: STRING, servicePrincipalName: STRING, sessionId: STRING, sessionLifetimePolicies: VARIANT, signInIdentifier: STRING, signInIdentifierType: STRING, signInTokenProtectionStatus: STRING, sourceSystem: VARIANT, ssoExtensionVersion: STRING, status: VARIANT, targetResources: ARRAY<STRUCT<displayName: STRING, id: STRING, modifiedProperties: ARRAY<STRUCT<oldValue: STRING, displayName: STRING, newValue: STRING>>, type: STRING, administrativeUnits: VARIANT, groupType: STRING, userPrincipalName: STRING>>, tenantId: STRING, timeGenerated: TIMESTAMP, tokenIssuerName: STRING, tokenIssuerType: STRING, tokenProtectionStatusDetails: VARIANT, type: STRING, uniqueTokenIdentifier: STRING, userAgent: STRING, userDisplayName: STRING, userId: STRING, userPrincipalName: STRING, userType: STRING, activity: STRING, additionalInfo: STRING, detectedDateTime: TIMESTAMP, detectionTimingType: STRING, lastUpdatedDateTime: TIMESTAMP, mitreTechniqueId: STRING, riskEventType: STRING, riskType: STRING, source: STRING, identity: STRING, operationName: STRING, resultDescription: STRING, resultType: STRING, C_DeviceId: STRING, C_Sid: STRING, C_Iat: STRING, C_Idtyp: STRING, UserPrincipalObjectID: STRING, __UDI_RequiredFields_EventTime: TIMESTAMP, __UDI_RequiredFields_RegionScope: STRING, __UDI_RequiredFields_TenantId: STRING, __UDI_RequiredFields_UniqueId: STRING, apiVersion: STRING, atContentH: STRING, atContentP: STRING, clientAuthMethod: STRING, clientRequestId: STRING, durationMs: DOUBLE, identityProvider: STRING, operationId: STRING, requestMethod: STRING, requestUri: STRING, responseSizeBytes: BIGINT, responseStatusCode: BIGINT, roles: STRING, scopes: STRING, signInActivityId: STRING, tokenIssuedAt: TIMESTAMP, wids: STRING, requestId: STRING, appOwnerTenantId: STRING, servicePrincipalCredentialThumbprint: STRING, mfaDetail: STRUCT<authDetail: STRING, authMethod: STRING>>
resourceId	STRING
resultDescription	STRING
resultSignature	STRING
resultType	STRING
tenantId	STRING

Column	Data type
Level	STRING
callerIpAddress	STRING
category	STRING
correlationId	STRING
durationMs	DOUBLE
identity	STRING
location	STRING
locationDetails	VARIANT
networkLocationDetails	VARIANT
operationName	STRING
operationVersion	STRING
time	TIMESTAMP
properties	STRUCT<aadTenantId: STRING, activityDateTime: TIMESTAMP, activityDisplayName: STRING, additionalDetails: ARRAY<STRUCT<key: STRING, value: STRING>>, alternateSignInName: STRING, appDisplayName: STRING, appliedConditionalAccessPolicies: VARIANT, appliedEventListeners: VARIANT, appId: STRING, appServicePrincipalId: STRING, authenticationAppDeviceDetails: VARIANT, authenticationStrengths: VARIANT, authenticationAppPolicyEvaluationDetails: VARIANT, authenticationContextClassReferences: VARIANT, authenticationDetails: VARIANT, authenticationMethodsUsed: VARIANT, authenticationProcessingDetails: VARIANT, authenticationProtocol: STRING, authenticationRequirement: STRING, authenticationRequirementPolicies: VARIANT, autonomousSystemNumber: STRING, _billedSize: DOUBLE, category: STRING, clientAppUsed: STRING, clientCredentialType: STRING, conditionalAccessAudiences: VARIANT, conditionalAccessPolicies: VARIANT, conditionalAccessStatus: STRING, correlationId: STRING, createdDateTime: TIMESTAMP, crossTenantAccessType: STRING, deviceDetail: VARIANT, federatedCredentialId: STRING, flaggedForReview: BOOLEAN, globalSecureAccessIpAddress: STRING, homeTenantId: STRING, homeTenantName: STRING, id: STRING, incomingTokenType: STRING, ipAddress: STRING, ipAddressFromResourceProvider: STRING, _isBillable: STRING, isDeleted: BOOLEAN, initiatedBy: STRUCT<app: STRUCT<displayName: STRING, servicePrincipalId: STRING, appId: STRING>, user: STRUCT<id: STRING, displayName: STRING, userPrincipalName: STRING, ipAddress: STRING, roles: VARIANT>>, isProcessing: BOOLEAN, loggedByService: STRING, location: VARIANT, networkLocationDetails: VARIANT, operationType: STRING, result: STRING, resultReason: STRING, isInteractive: BOOLEAN, isRisky: BOOLEAN, isTenantRestricted: BOOLEAN, isThroughGlobalSecureAccess: BOOLEAN, originalRequestId: STRING, originalTransferMethod: STRING, privateLinkDetails: VARIANT, processingTimeInMilliseconds: BIGINT, resource: STRING, resourceDisplayName: STRING, resourceGroup: STRING, resourceId: STRING, resourceIdentity: STRING, resourceProvider: STRING, resourceOwnerTenantId: STRING, resourceServicePrincipalId: STRING, resourceTenantId: STRING, riskEventTypes: VARIANT, riskEventTypes_v2: VARIANT, riskLastUpdatedDateTime: TIMESTAMP, riskDetail: STRING, riskLevel: STRING, riskLevelAggregated: STRING, riskLevelDuringSignIn: STRING, riskState: STRING, rngcStatus: STRING, servicePrincipalId: STRING, servicePrincipalCredentialKeyId: STRING, servicePrincipalName: STRING, sessionId: STRING, sessionLifetimePolicies: VARIANT, signInIdentifier: STRING, signInIdentifierType: STRING, signInTokenProtectionStatus: STRING, sourceSystem: VARIANT, ssoExtensionVersion: STRING, status: VARIANT, targetResources: ARRAY<STRUCT<displayName: STRING, id: STRING, modifiedProperties: ARRAY<STRUCT<oldValue: STRING, displayName: STRING, newValue: STRING>>, type: STRING, administrativeUnits: VARIANT, groupType: STRING, userPrincipalName: STRING>>, tenantId: STRING, timeGenerated: TIMESTAMP, tokenIssuerName: STRING, tokenIssuerType: STRING, tokenProtectionStatusDetails: VARIANT, type: STRING, uniqueTokenIdentifier: STRING, userAgent: STRING, userDisplayName: STRING, userId: STRING, userPrincipalName: STRING, userType: STRING, activity: STRING, additionalInfo: STRING, detectedDateTime: TIMESTAMP, detectionTimingType: STRING, lastUpdatedDateTime: TIMESTAMP, mitreTechniqueId: STRING, riskEventType: STRING, riskType: STRING, source: STRING, identity: STRING, operationName: STRING, resultDescription: STRING, resultType: STRING, C_DeviceId: STRING, C_Sid: STRING, C_Iat: STRING, C_Idtyp: STRING, UserPrincipalObjectID: STRING, __UDI_RequiredFields_EventTime: TIMESTAMP, __UDI_RequiredFields_RegionScope: STRING, __UDI_RequiredFields_TenantId: STRING, __UDI_RequiredFields_UniqueId: STRING, apiVersion: STRING, atContentH: STRING, atContentP: STRING, clientAuthMethod: STRING, clientRequestId: STRING, durationMs: DOUBLE, identityProvider: STRING, operationId: STRING, requestMethod: STRING, requestUri: STRING, responseSizeBytes: BIGINT, responseStatusCode: BIGINT, roles: STRING, scopes: STRING, signInActivityId: STRING, tokenIssuedAt: TIMESTAMP, wids: STRING, requestId: STRING, appOwnerTenantId: STRING, servicePrincipalCredentialThumbprint: STRING, mfaDetail: STRUCT<authDetail: STRING, authMethod: STRING>>
resourceId	STRING
resultDescription	STRING
resultSignature	STRING
resultType	STRING
tenantId	STRING

Parser observables

The parser extracts the following fields into lw_observables (generated from the parser definition):

Field	Observable
callerIpAddress	ip
correlationId	trace_id
properties.alternateSignInName	username
properties.ipAddress	ip
properties.ipAddressFromResourceProvider	ip
properties.initiatedBy.user.displayName	username
properties.initiatedBy.user.ipAddress	ip
properties.userDisplayName	username
properties.userPrincipalName	username
properties.userPrincipalName	email
properties.mitreTechniqueId	mitre_attack_technique

Field	Observable
callerIpAddress	ip
correlationId	trace_id
properties.alternateSignInName	username
properties.ipAddress	ip
properties.ipAddressFromResourceProvider	ip
properties.initiatedBy.user.displayName	username
properties.initiatedBy.user.ipAddress	ip
properties.userDisplayName	username
properties.userPrincipalName	username
properties.userPrincipalName	email
properties.mitreTechniqueId	mitre_attack_technique