GitHub audit logs

Preview

Lakewatch is in Private Preview. To request access, reach out to your Databricks account team.

Lakewatch provides an API connector to ingest organization audit logs from GitHub. This article covers prerequisites, configuration instructions, the connector schema, and Lakewatch parser observables.

Prerequisites

The connector requires:

1. Serverless Lakeflow Spark Declarative Pipelines turned on in your workspace.
2. Unity Catalog, with privileges to create or use a Unity Catalog connection. See Connection permissions.
3. A GitHub account with access to the organization that you want to ingest audit logs from.
4. Permission to create OAuth Apps in your GitHub account's developer settings.

Known limitations

• The audit_logs table is available only for organization accounts.
• On the GitHub Free plan, audit log history on github.com is limited to 90 days.
• Historical lookback can't exceed 180 days. See Choose how far back to ingest.
• The connector supports only long-lived OAuth user-to-machine (U2M) access tokens. If the OAuth App issues expiring user access tokens, ingestion fails after the token expires.
• GitHub includes actor IP addresses in audit events only when IP address display is enabled in the GitHub organization or enterprise settings.

Configure GitHub for ingestion

1. Sign in to your GitHub account.

2. In the upper-right corner of any page, click your profile photo to open the user navigation menu.

3. Click Settings.

4. Click Developer settings.

5. In the OAuth Apps section, click New OAuth App.

6. Enter a memorable application name in the Application name field.

7. In the Homepage URL field, enter your Databricks instance's primary URL. For example, https://my-databricks-account.cloud.databricks.com. Optionally, enter an application description.

8. Set the authorization callback URL to the following:

   https://<your_workspace_domain>/login/oauth/lakehousefederation.html

   For example:

   https://my-databricks-account.cloud.databricks.com/login/oauth/lakehousefederation.html

9. Click Register Application. This takes you to the application's details page.

10. Note the Client ID. You need it when you create your Unity Catalog connection.

11. Click Generate a new client secret to generate a client secret for your application.

12. Note the generated Client Secret. This value is displayed only one time.

tip

When you create a connection through the Lakewatch UI, enter the Client ID and Client Secret gathered above. For GitHub Enterprise Server, also enter the optional Enterprise domain. You are redirected to GitHub to authorize the requested scopes.

For more information, see Configure OAuth U2M for GitHub ingestion.

Next steps

After completing the configuration above, create a datasource with the GitHub connector in the Lakewatch UI. See API connectors for instructions.

---

Connector and Lakewatch schemas

Lakewatch ingests the following source table from the GitHub connector's default source schema:

Source table	GitHub data	Built-in Lakewatch parser
audit_logs	Organization actions performed by users, integrations, and systems	github_audit_lfc

Source table	GitHub data	Built-in Lakewatch parser
audit_logs	Organization actions performed by users, integrations, and systems	github_audit_lfc

The connector writes the source table to the datasource's raw destination. The built-in parser writes a separate parsed table named after the parser, unless you set a parsed destination when you configure the datasource.

Lakewatch-managed columns

The connector adds the following columns to each raw source row:

• lw_id: A non-null string that Lakewatch uses as the primary key. The connector derives a stable, time-sortable ID from the source @timestamp and _document_id fields.
• time: A non-null UTC timestamp converted from the source @timestamp value, which GitHub reports as Unix epoch milliseconds. The connector uses time as the cursor field for incremental ingestion.

The parsed table carries lw_id forward. The parser also assigns the parsed time column and adds:

• lw_meta: Standard Lakewatch processing metadata.
• lw_observables: Values extracted from fields marked as security observables in the parser.

For more information, see Parser reference.

Raw connector table schema

The connector extracts common GitHub audit event fields into typed columns and preserves the complete source event in payload.

Schema

Column	Data type
lw_id	STRING
_document_id	STRING
time	TIMESTAMP
action	STRING
actor	STRING
org	STRING
repo	STRING
payload	VARIANT

Column	Data type
lw_id	STRING
_document_id	STRING
time	TIMESTAMP
action	STRING
actor	STRING
org	STRING
repo	STRING
payload	VARIANT

For field definitions, see the GitHub audit log API.

Parser observables

The github_audit_lfc parser keeps the complete source event as VARIANT in payload and extracts the following fields into lw_observables.

Observables

Field	Observable
actor	username
payload.actor_ip	ip
payload.email	email
payload.invitee_email	email
payload.user	username
payload.blocked_user	username
payload.external_identity_nameid	username
payload.external_identity_username	username
payload.user_programmatic_access_name	username
payload.invitee	username
payload.inviter	username
payload.authorized_actors[]	username
payload.authorized_actor_names[]	username
payload.actor_id	actor_id
payload.user_id	actor_id
payload.referrer	url
payload.url	url
payload.pull_request_url	url
payload.config.url	url
payload.config_was.url	url
payload.head_sha	hash
payload.after	hash
payload.before	hash
payload.calling_workflow_shas[]	hash

Field	Observable
actor	username
payload.actor_ip	ip
payload.email	email
payload.invitee_email	email
payload.user	username
payload.blocked_user	username
payload.external_identity_nameid	username
payload.external_identity_username	username
payload.user_programmatic_access_name	username
payload.invitee	username
payload.inviter	username
payload.authorized_actors[]	username
payload.authorized_actor_names[]	username
payload.actor_id	actor_id
payload.user_id	actor_id
payload.referrer	url
payload.url	url
payload.pull_request_url	url
payload.config.url	url
payload.config_was.url	url
payload.head_sha	hash
payload.after	hash
payload.before	hash
payload.calling_workflow_shas[]	hash