Extended network traffic logs
This feature is in Private Preview. To try it, reach out to your Databricks contact.
Extended network traffic logs record outbound network traffic from Databricks serverless compute. By default, the outbound network access events system table records only the requests that serverless egress controls deny. With extended network traffic logs, it also records the requests that were allowed, so you can see where your serverless workloads connect and troubleshoot connectivity.
To query the logs, see Query extended network traffic logs.
Use cases
Use extended network traffic logs to:
- Investigate network activity: See which destinations your serverless workloads accessed, when, and from which workspace.
- Refine network policies: Find destinations that are rarely accessed and decide whether to remove them from your serverless egress allowlists.
- Troubleshoot connectivity: Determine whether serverless egress controls allowed or denied a request.
- Investigate networking costs: Identify cross-region, internet, and private destinations accessed by serverless workloads.
- Optimize storage connectivity: Find frequently accessed storage destinations that might benefit from private connectivity or from being in the same region as your workloads.
How it works
Allowed network access events
Extended network traffic logs add ALLOWED events to the existing system.access.outbound_network table. This table records outbound network access events from serverless workloads, including SQL warehouses, jobs, Lakeflow pipelines, Databricks Apps, and Model Serving endpoints. Allowed events cover requests to cloud storage, internet destinations, and resources reached through private connectivity. For the complete table schema, see Network access events system table reference.
The access_type column indicates the outcome of each event:
Value | Description |
|---|---|
| Outbound network access was allowed. Available only with extended network traffic logs. |
| Outbound network access was denied by serverless egress controls in enforced mode. |
| Outbound network access would have been denied by serverless egress controls, but was allowed because the policy was in dry-run mode. |
The access_type and destination_type values recorded for a request depend on the destination and your serverless egress policy mode:
Network request | Unrestricted egress | Dry-run mode | Enforced mode |
|---|---|---|---|
Allowed storage access |
| Not logged |
|
Storage access that violates the policy | Not applicable |
|
|
Allowed access to VPC resources |
|
|
|
Access to VPC resources that violates the policy | Not applicable |
|
|
Allowed internet access |
|
|
|
Internet access that violates the policy | Not applicable |
|
|
For allowed DNS events, destination contains the domain name or storage hostname. For allowed STORAGE events, destination contains the storage hostname and path. The resolved destination IP addresses are in dns_event.rdata or storage_event.rdata. These fields are empty for denied events.
Limitations
Extended network traffic logs have the following limitations:
- Allowed events aren't logged for external model access from Model Serving endpoints.
- Allowed events aren't logged for storage access from workspaces whose serverless egress controls are in dry-run mode.
- Allowed events aren't logged for access to workspace-internal storage, such as MLflow and dbt artifacts.
- Allowed events aren't logged for access to OpenSharing provider storage.
- Allowed traffic from workloads that run for less than about one minute is less likely to be logged, especially for scheduled, non-interactive workloads and for workspaces with unrestricted egress policies.
These limitations don't affect how serverless egress controls are enforced or how denied requests are logged.
Frequently asked questions
The following questions cover common behavior you might see in extended network traffic logs.
Do I need serverless egress controls to use extended network traffic logs?
No. Allowed outbound network access events are recorded for workspaces with unrestricted serverless egress policies too.
Why do network access events appear after a workload finishes?
Databricks might continue to access storage after a workload finishes, for example to maintain Delta logs or metadata. These requests can generate network access events when no user workloads are running. The events stop after the serverless compute instance terminates.
Why do I see unfamiliar hostnames in my network access logs?
Some logged destinations are Databricks-managed resources used for platform operations:
Hostname pattern | Purpose |
|---|---|
| Improves serverless compute startup times. |
| Databricks default storage. |
| Access to default sample datasets. |
If you see other hostnames you don't recognize, contact your Databricks account team.
How long does it take for network traffic logs to appear?
Most outbound network access events appear within 15 minutes of the request. Latency can vary, and Databricks doesn't provide a latency service level agreement (SLA) for these logs.