Download billable usage logs using the Account API

Preview

This feature is in Public Preview.

As a Databricks account owner or account admin, you can use the account console to download billable usage logs. To access this data programmatically, you can also use the Account API to download the logs. This article explains how to call that API.

Alternatively, you can configure daily delivery of billable usage logs in CSV file format to an AWS S3 storage bucket. See Deliver and access billable usage logs.

Requirements

  • Email address and password for an account owner (or account admin, if you are on an E2 account) to authenticate with the APIs. The email address and password are both case sensitive.
  • Account ID. For accounts on the E2 version of the platform, get your account ID from the user profile drop-down in the account console. For non-E2 accounts, get your account ID from the account console’s Usage Overview tab. Contact your Databricks representative if you cannot find your account ID.

How to authenticate to the Account API

The APIs described in this article are published on the accounts.cloud.databricks.com base endpoint for all AWS regional deployments.

Use the following base URL for API requests: https://accounts.cloud.databricks.com/api/2.0/

This REST API requires HTTP basic authentication, which involves setting the HTTP header Authorization. In this article, username refers to your account owner (or account admin, if you are on an E2 account) email address. The email address is case sensitive. There are several ways to provide your credentials to tools such as curl.

  • Pass your username and account password separately in the headers of each request in <username>:<password> syntax.

    For example:

    curl -X GET -u `<username>:<password>` -H "Content-Type: application/json" \
     'https://accounts.cloud.databricks.com/api/2.0/accounts/<account-id>/<endpoint>'
    
  • Apply base64 encoding to your <username>:<password> string and provide it directly in the HTTP header:

    curl -X GET -H "Content-Type: application/json" \
      -H 'Authorization: Basic <base64-username-pw>'
      'https://accounts.cloud.databricks.com/api/2.0/accounts/<account-id>/<endpoint>'
    
  • Create a .netrc file with machine, login, and password properties:

    machine accounts.cloud.databricks.com
    login <username>
    password <password>
    

    To invoke the .netrc file, use -n in your curl command:

    curl -n -X GET 'https://accounts.cloud.databricks.com/api/2.0/accounts/<account-id>/workspaces'
    

    This article’s examples use this authentication style.

For the complete API reference, see Account API.

Call the billable usage log download API

To download billable usage data, call the billable usage download API (GET '/accounts/<account_id>/usage/download).

Add the following query fields:

  • start_month: (Required) The month and year at which log delivery starts. Format is text in YYYY-MM format. You can enter any month and year on or after 2019-03.
  • end_month: (Required) The month and year at which log delivery ends. Format is text in YYYY-MM format. You can enter any month and year on or after 2019-03.
  • personal_data: (Optional) Specify whether to include personally identifiable information in the billable usage logs, for example the email addresses of cluster creators. Handle this information with care. The default is false, which means do not include this information.

For example:

curl -X GET -n \
  'https://accounts.cloud.databricks.com/api/2.0/accounts/<databricks-account-id>/usage/download?start_month=2020-01&end_month=2020-12&personal_data=false'

Using the log files for analysis

For the CSV schema, see CSV file schema

For information about how to analyze these files using Databricks, see Analyze usage data in Databricks