Okta system logs

Preview

Lakewatch is in Private Preview. To request access, reach out to your Databricks account team.

Lakewatch provides an API connector to ingest system logs from Okta. This article covers prerequisites, configuration instructions, connector schemas, and Lakewatch parser observables.

Prerequisites

The connector requires:

1. Serverless Lakeflow Spark Declarative Pipelines turned on in your workspace.
2. Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
3. An active Okta super administrator, org administrator, or read-only administrator account. Databricks recommends using a read-only administrator account for least-privilege System Log access.
4. Access to the Okta Admin Console.

note

An API token has the same permissions as the administrator who creates it. If that administrator's permissions change or the account is deactivated, the token is affected. For details, see Manage Okta API tokens.

Known limitations

• SCD type 2 is not supported, since Okta system logs are append-only.
• The connector supports Authentication via API Token only.
• Historical lookback can't exceed the 90 days Okta retains System Logs. See Choose how far back to ingest.

Configure Okta for ingestion

1. Sign in to the Okta Admin Console with the administrator account that owns the token.
2. Go to Security > API.
3. Click the Tokens tab.
4. Click Create token.
5. Enter a name for the token.
6. For API calls made with this token must originate from, select Any IP.
7. Click Create token.
8. Copy the displayed SSWS API token. Okta doesn't display the token again.
9. Note your Okta organization domain, such as company-name.okta.com.
10. On the Tokens tab, confirm that the token's Role is Super Admin, Organization Admin, or Read-only Admin.

For more information, see Manage Okta API tokens.

tip

When creating a connection through the Lakewatch UI, you need the Domain and SSWS API Token gathered above.

Next steps

After completing the configuration above, create a datasource with the Okta connector in the Lakewatch UI. See API connectors for instructions.

---

Connector and Lakewatch schemas

The connector ingests the following source table from the default source schema:

Source table	Okta data	Built-in Lakewatch parser
system_logs	Okta System Log security events	okta_system_log

Source table	Okta data	Built-in Lakewatch parser
system_logs	Okta System Log security events	okta_system_log

The connector writes the source table to the datasource's raw destination. The built-in parser writes a separate parsed table named after the parser, unless you set a parsed destination when you configure the datasource.

Lakewatch-managed columns

The connector adds the following columns to each raw source row:

• lw_id: A non-null string that Lakewatch uses as the primary key. The connector derives a stable, time-sortable ID from the Okta published and uuid fields. The ID supports deduplication across incremental syncs.
• time: A non-null timestamp parsed from published as an RFC 3339 timestamp. The connector uses the source published value for incremental ordering.

The parsed table carries lw_id forward. The parser also assigns the parsed time column and adds:

• lw_meta: Standard Lakewatch processing metadata.
• lw_observables: Values extracted from fields marked as security observables in the parser.

For more information, see Parser reference.

Raw connector table schema

The following schema combines the standard Lakeflow Connect fields with the Lakewatch-only lw_id and time columns. For nested field definitions, see the Okta System Log API reference.

Schema

Column	Data type
lw_id	STRING
time	TIMESTAMP
uuid	STRING
published	TIMESTAMP
eventType	STRING
version	STRING
severity	STRING
legacyEventType	STRING
displayMessage	STRING
actor	STRUCT<id: STRING, displayName: STRING, type: STRING, alternateId: STRING, detailEntry: MAP<STRING, STRING>>
authenticationContext	STRUCT<authenticationProvider: STRING, authenticationStep: INT, credentialProvider: STRING, credentialType: STRING, interface: STRING, externalSessionId: STRING, issuer: STRUCT<id: STRING, type: STRING>>
client	STRUCT<id: STRING, ipAddress: STRING, userAgent: STRUCT<rawUserAgent: STRING, os: STRING, browser: STRING>, zone: STRING, device: STRING, geographicalContext: STRUCT<city: STRING, state: STRING, country: STRING, postalCode: STRING, geolocation: STRUCT<lat: DOUBLE, lon: DOUBLE>>>
debugContext	STRUCT<debugData: MAP<STRING, STRING>>
outcome	STRUCT<result: STRING, reason: STRING>
device	STRUCT<id: STRING, name: STRING, os_platform: STRING, os_version: STRING, managed: BOOLEAN, registered: BOOLEAN, device_integrator: STRING, disk_encryption_type: STRING, screen_lock_type: STRING, jailbreak: BOOLEAN, secure_hardware_present: BOOLEAN>
transaction	STRUCT<type: STRING, id: STRING, detail: MAP<STRING, STRING>>
request	STRUCT<ipChain: ARRAY<STRUCT<ip: STRING, geographicalContext: STRUCT<city: STRING, state: STRING, country: STRING, postalCode: STRING, geolocation: STRUCT<lat: DOUBLE, lon: DOUBLE>>, version: STRING, source: STRING, ipDetails: STRUCT<asNumber: BIGINT, asOrg: STRING, isp: STRING, domain: STRING, ipServiceCategories: ARRAY<STRUCT<isAnonymous: BOOLEAN, operator: STRING, type: STRING>>>>>>
target	ARRAY<STRUCT<id: STRING, type: STRING, alternateId: STRING, displayName: STRING, detailEntry: MAP<STRING, STRING>, changeDetails: STRUCT<from: STRUCT<id: STRING, type: STRING, alternateId: STRING, displayName: STRING, detailEntry: MAP<STRING, STRING>>, to: STRUCT<id: STRING, type: STRING, alternateId: STRING, displayName: STRING, detailEntry: MAP<STRING, STRING>>>>>
securityContext	STRUCT<asNumber: BIGINT, asOrg: STRING, isp: STRING, domain: STRING, isProxy: BOOLEAN, ipDetails: STRUCT<asNumber: BIGINT, asOrg: STRING, isp: STRING, domain: STRING, ipServiceCategories: ARRAY<STRUCT<isAnonymous: BOOLEAN, operator: STRING, type: STRING>>>, risk: STRUCT<detectionName: STRING, issuer: STRING, level: STRING, previousLevel: STRING, reasons: ARRAY<STRING>>, UserBehaviours: ARRAY<STRUCT<id: STRING, name: STRING, result: STRING>>>

Column	Data type
lw_id	STRING
time	TIMESTAMP
uuid	STRING
published	TIMESTAMP
eventType	STRING
version	STRING
severity	STRING
legacyEventType	STRING
displayMessage	STRING
actor	STRUCT<id: STRING, displayName: STRING, type: STRING, alternateId: STRING, detailEntry: MAP<STRING, STRING>>
authenticationContext	STRUCT<authenticationProvider: STRING, authenticationStep: INT, credentialProvider: STRING, credentialType: STRING, interface: STRING, externalSessionId: STRING, issuer: STRUCT<id: STRING, type: STRING>>
client	STRUCT<id: STRING, ipAddress: STRING, userAgent: STRUCT<rawUserAgent: STRING, os: STRING, browser: STRING>, zone: STRING, device: STRING, geographicalContext: STRUCT<city: STRING, state: STRING, country: STRING, postalCode: STRING, geolocation: STRUCT<lat: DOUBLE, lon: DOUBLE>>>
debugContext	STRUCT<debugData: MAP<STRING, STRING>>
outcome	STRUCT<result: STRING, reason: STRING>
device	STRUCT<id: STRING, name: STRING, os_platform: STRING, os_version: STRING, managed: BOOLEAN, registered: BOOLEAN, device_integrator: STRING, disk_encryption_type: STRING, screen_lock_type: STRING, jailbreak: BOOLEAN, secure_hardware_present: BOOLEAN>
transaction	STRUCT<type: STRING, id: STRING, detail: MAP<STRING, STRING>>
request	STRUCT<ipChain: ARRAY<STRUCT<ip: STRING, geographicalContext: STRUCT<city: STRING, state: STRING, country: STRING, postalCode: STRING, geolocation: STRUCT<lat: DOUBLE, lon: DOUBLE>>, version: STRING, source: STRING, ipDetails: STRUCT<asNumber: BIGINT, asOrg: STRING, isp: STRING, domain: STRING, ipServiceCategories: ARRAY<STRUCT<isAnonymous: BOOLEAN, operator: STRING, type: STRING>>>>>>
target	ARRAY<STRUCT<id: STRING, type: STRING, alternateId: STRING, displayName: STRING, detailEntry: MAP<STRING, STRING>, changeDetails: STRUCT<from: STRUCT<id: STRING, type: STRING, alternateId: STRING, displayName: STRING, detailEntry: MAP<STRING, STRING>>, to: STRUCT<id: STRING, type: STRING, alternateId: STRING, displayName: STRING, detailEntry: MAP<STRING, STRING>>>>>
securityContext	STRUCT<asNumber: BIGINT, asOrg: STRING, isp: STRING, domain: STRING, isProxy: BOOLEAN, ipDetails: STRUCT<asNumber: BIGINT, asOrg: STRING, isp: STRING, domain: STRING, ipServiceCategories: ARRAY<STRUCT<isAnonymous: BOOLEAN, operator: STRING, type: STRING>>>, risk: STRUCT<detectionName: STRING, issuer: STRING, level: STRING, previousLevel: STRING, reasons: ARRAY<STRING>>, UserBehaviours: ARRAY<STRUCT<id: STRING, name: STRING, result: STRING>>>

Parser observables

The okta_system_log parser writes nested objects as VARIANT and extracts the following fields into lw_observables.

Observables

Field	Observable
actor.id	actor_id
actor.alternateId	email
actor.alternateId	actor_id
actor.alternateId	username
client.ipAddress	ip
request.ipChain[].ip	ip
request.ipChain[].ipDetails.domain	domain
target[].id	actor_id
target[].alternateId	email
target[].alternateId	actor_id
target[].alternateId	username
securityContext.domain	domain
securityContext.ipDetails.domain	domain

Field	Observable
actor.id	actor_id
actor.alternateId	email
actor.alternateId	actor_id
actor.alternateId	username
client.ipAddress	ip
request.ipChain[].ip	ip
request.ipChain[].ipDetails.domain	domain
target[].id	actor_id
target[].alternateId	email
target[].alternateId	actor_id
target[].alternateId	username
securityContext.domain	domain
securityContext.ipDetails.domain	domain

---

Troubleshooting

I'm getting rate-limit errors while ingesting data.

1. In the Okta Admin Panel, go to Security >> API.
2. In the Tokens tab, find the API token that you used to Authenticate to Databricks.
3. Click the name of the Token and in the Token rate limits section and then click Edit and increase the slider to more than 50% of Each API rate limit.