Monitor the cost of serverless compute

Preview

Serverless compute for notebooks and workflows in Public Preview. For information on eligibility and enablement, see Enable serverless compute public preview.

This article explains how to use the billable usage system table (Public Preview) to monitor the cost of your serverless compute usage.

You can monitor the usage of serverless compute for notebooks and workflows by querying the billable usage system table (system.billing.usage), which includes user and workload attributes related to serverless compute costs. The applicable fields include:

  • The identity_metadata column includes the run_as field, which shows the user or service principal whose credentials were used to run the workload.

  • The usage_metadata column has fields that describe the workload: job_run_id and notebook_id.

Considerations for serverless usage records

When analyzing your serverless usage, consider the following:

  • You may see multiple records associated with a given serverless compute workload in a given hour. For example, you may see multiple records with the same job_id and job_run_id but with different DBU consumption values for each. The sum of these DBUs collectively represents the hourly DBU consumption for a given job run.

  • You may also see records with DBU consumption billed using a serverless SKU, but with null values for run_as, job_id, job_run_id, and notebook_id. These represent costs associated with shared resources that are not directly attributable to any particular workload. As you increase your usage of serverless compute and add more workloads, the proportion of these shared costs on your bill will decrease as they are shared across more workloads.

Cost observability dashboard

To help you get started monitoring your serverless costs, download the following cost observability dashboard from Github. See Serverless cost observability dashboard.

Serverless billing observability dashboard

After you download the JSON file, import the dashboard into your workspace. For instructions on importing dashboards, see Import a dashboard file.

Use alerts to track serverless spend

Alerts are a powerful way to stay informed about your serverless spend. With alerts, you can receive notifications when certain conditions are met in your query results. To learn how to create alerts, see Create an alert.

You can add alerts to the following queries to monitor budgets. In each query, replace {budget} with your chosen budget.

Alert when any workspace spend exceeds a threshold in the last 30 days

You can set an alert to trigger whenever this query returns a row. Replace {budget} with your chosen budget.

SELECT
   t1.workspace_id,
   SUM(t1.usage_quantity * list_prices.pricing.default) as list_cost
FROM system.billing.usage t1
INNER JOIN system.billing.list_prices on
   t1.cloud = list_prices.cloud and
   t1.sku_name = list_prices.sku_name and
   t1.usage_start_time >= list_prices.price_start_time and
   (t1.usage_end_time <= list_prices.price_end_time or list_prices.price_end_time is null)
WHERE
   t1.sku_name LIKE '%SERVERLESS%'
   AND billing_origin_product IN ("JOBS", "NOTEBOOKS")
   AND t1.usage_date >= CURRENT_DATE() - INTERVAL 30 DAYS
GROUP BY
   t1.workspace_id
HAVING
   list_cost > {budget}

Alert when a user exceeds the threshold in the last 30 days

You can set an alert to trigger whenever this query returns a row. Replace {budget} with your chosen budget.

SELECT
   t1.identity_metadata.run_as,
   SUM(t1.usage_quantity * list_prices.pricing.default) as list_cost
FROM system.billing.usage t1
INNER JOIN system.billing.list_prices on
   t1.cloud = list_prices.cloud and
   t1.sku_name = list_prices.sku_name and
   t1.usage_start_time >= list_prices.price_start_time and
   (t1.usage_end_time <= list_prices.price_end_time or list_prices.price_end_time is null)
WHERE
   t1.sku_name LIKE '%SERVERLESS%'
   AND billing_origin_product IN ("JOBS", "NOTEBOOKS")
   AND t1.usage_date >= CURRENT_DATE() - INTERVAL 30 DAYS
GROUP BY
   t1.identity_metadata.run_as
HAVING
   list_cost > {budget}

Alert when a job exceeds the threshold in the last 30 days

You can set an alert to trigger whenever this query returns a row. Replace {budget} with your chosen budget.

SELECT
   t1.workspace_id,
   t1.usage_metadata.job_id,
   SUM(t1.usage_quantity * list_prices.pricing.default) as list_cost
FROM system.billing.usage t1
INNER JOIN system.billing.list_prices on
   t1.cloud = list_prices.cloud and
   t1.sku_name = list_prices.sku_name and
   t1.usage_start_time >= list_prices.price_start_time and
   (t1.usage_end_time <= list_prices.price_end_time or list_prices.price_end_time is null)
WHERE
   t1.sku_name LIKE '%SERVERLESS%'
   AND billing_origin_product IN ("JOBS")
   AND t1.usage_date >= CURRENT_DATE() - INTERVAL 30 DAYS
GROUP BY
   t1.workspace_id, t1.usage_metadata.job_id,
HAVING
   list_cost > {budget}

Sample queries

Use the following queries to gain insights into serverless usage in your account:

Identify expensive serverless compute notebooks

This query returns a list of notebooks and how many DBUs each notebook consumed, in descending order by DBU consumption:

SELECT
  usage_metadata.notebook_id,
  SUM(usage_quantity) as total_dbu
FROM
  system.billing.usage
WHERE
  usage_metadata.notebook_id is not null
  and billing_origin_product = 'INTERACTIVE'
  and product_features.is_serverless
  and usage_unit = 'DBU'
  and usage_date >= DATEADD(day, -30, current_date)
GROUP BY
  1
ORDER BY
  total_dbu DESC

Identify expensive serverless compute jobs

This query returns a list of jobs and how many DBUs each job consumed, in descending order by DBU consumption:

SELECT
  usage_metadata.job_id,
  SUM(usage_quantity) as total_dbu
FROM
  system.billing.usage
WHERE
  usage_metadata.job_id is not null
  and billing_origin_product = 'JOBS'
  and product_features.is_serverless
  and usage_unit = 'DBU'
  and usage_date >= DATEADD(day, -30, current_date)
GROUP BY
  1
ORDER BY
  total_dbu DESC

Report on DBUs consumed by a particular user

This query returns a list of notebooks and jobs that use serverless compute run by a particular user or service principal, and the number of DBUs consumed by each workload:

SELECT
  usage_metadata.job_id,
  usage_metadata.notebook_id,
  SUM(usage_quantity) as total_dbu
FROM
  system.billing.usage
WHERE
  identity_metadata.run_as = '<emailaddress@domain.com>'
  and billing_origin_product in ('JOBS','INTERACTIVE')
  and product_features.is_serverless
  and usage_unit = 'DBU'
  and usage_date >= DATEADD(day, -30, current_date)
GROUP BY
  1,2
ORDER BY
  total_dbu DESC

Report on serverless compute DBUs consumed by workloads that share a custom tag

This query returns a list of jobs that use serverless compute that share the same custom tag, and the number of DBUs consumed by each workload:

SELECT
  usage_metadata.job_id,
  usage_metadata.notebook_id,
  SUM(usage_quantity) as total_dbu
FROM
  system.billing.usage
WHERE
  custom_tags.<key> = '<value>'
  and billing_origin_product in ('JOBS','INTERACTIVE')
  and product_features.is_serverless
  and usage_unit = 'DBU'
  and usage_date >= DATEADD(day, -30, current_date)
GROUP BY
  1,2
ORDER BY
  total_dbu DESC