Skip to main content
Last updated on

Tutorial: Track foundation model spend by user, team, or project

Beta

This feature is in Beta. Account admins can manage access to this feature from the account console Previews page. See Manage Databricks previews.

In this tutorial, you analyze what's driving your foundation model spend with Unity AI Gateway. You attribute cost by user across the model services you govern, then add tags to track spend by team, project, or use case using system tables:

Prerequisites​

note

Dollar estimates in this tutorial are based on list price. They exclude negotiated discounts and billing credits. Use them for attribution and trend analysis, and your billing statement for reconciliation.

Step 1: See which users are driving spend​

prompt

Genie Code (Agent mode) can do this for you. Try this example prompt:

Query system.billing.usage for MODEL_SERVING records from the past 30 days where usage_metadata.ai_gateway.endpoint_name is set. Convert DBUs to list-price USD using system.billing.list_prices at the price effective for each record, break down by model service, destination model, and requesting user, and sort by cost.

Go to AI Gateway > Govern > Usage Dashboard > Cost Analysis, which breaks down spend by Databricks-provided or external provider model, endpoint, and user without writing SQL. The Top Users by Cost (USD) chart under Cost Breakdown ranks the top spenders.

Alternatively, query the system tables directly to see who is consuming the most and where. The table you query depends on the model type:

Query system.billing.usage and join system.billing.list_prices to convert DBU usage to dollars. The identity_metadata.run_by field identifies the principal that issued each request:

SQL
SELECT
 u.usage_metadata.ai_gateway.endpoint_name AS model_service_name,
 u.usage_metadata.ai_gateway.destination_model AS destination_model,
 u.identity_metadata.run_by AS run_by,
 SUM(u.usage_quantity * lp.pricing.effective_list.default) AS list_cost_usd
FROM system.billing.usage u
JOIN system.billing.list_prices lp
 ON  u.sku_name  = lp.sku_name
 AND u.cloud     = lp.cloud
 AND u.usage_unit = lp.usage_unit
 AND u.usage_end_time >= lp.price_start_time
 AND (lp.price_end_time IS NULL OR u.usage_end_time < lp.price_end_time)
WHERE u.billing_origin_product = 'MODEL_SERVING'
 AND u.usage_metadata.ai_gateway.endpoint_name IS NOT NULL
 AND u.identity_metadata.run_by IS NOT NULL
 AND u.usage_unit = 'DBU'
 AND lp.currency_code = 'USD'
 AND u.usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY model_service_name, destination_model, run_by
ORDER BY list_cost_usd DESC;

Step 2: Set up usage and spend tracking by team or project​

Spend by user tells you who, but not which team or project. To attribute token usage to a team or project, tag your traffic based on how your model services map to teams or projects:

  • Service (endpoint) tags: when each team or project has its own model service. The tag applies to every request routed through that model service, so you set it once.
  • Request tags: when many teams or projects share the same model service. The caller sets the tag per request, so you attribute usage within a shared service.

Add a tag such as team or project to the model service. Every request routed through it inherits the tag.

  1. In the workspace UI, go to AI Gateway and open the model service.
  2. On the model service Overview page, under the Tags section, add a tag such as team = ml-platform.

Service tags appear in the endpoint_tags field in system.ai_gateway.usage and propagate to the custom_tags field in system.billing.usage. For details on creating and configuring model services, see Create and manage model services.

note

Which tag to use for cost attribution depends on the model:

  • Databricks-provided models: use service tags. Only service tags propagate to system.billing.usage.
  • External provider models: use service or request tags. Both propagate to system.ai_gateway.external_model_spend, which captures external model spend estimates.

Step 3: Analyze usage by team or project​

prompt

Genie Code (Agent mode) can do this for you. Try this example prompt:

Query system.ai_gateway.usage for the past 30 days to show request count and total tokens by the 'team' service tag in endpoint_tags, model service, and destination model. Sort by total tokens, highest first.

With tags in place, aggregate usage in system.ai_gateway.usage, which captures request and token metrics for every request routed to Databricks-provided and external provider models. Group by endpoint_tags['team'] for service tags, or by request_tags['project'] for request tags:

SQL
SELECT
  endpoint_tags['team'] AS team,
  endpoint_name,
  destination_model,
  COUNT(*) AS request_count,
  SUM(total_tokens) AS total_tokens
FROM system.ai_gateway.usage
WHERE endpoint_tags['team'] IS NOT NULL
GROUP BY endpoint_tags['team'], endpoint_name, destination_model
ORDER BY total_tokens DESC;

To explore either breakdown without SQL, use the tag filters on the Cost analysis page.

Step 4: Analyze dollar spend by team or project​

prompt

Genie Code (Agent mode) can do this for you. Try this example prompt:

Query system.billing.usage for MODEL_SERVING records from the past 30 days where usage_metadata.ai_gateway.endpoint_name is set. Convert DBUs to list-price USD using system.billing.list_prices at the price effective for each record, group by the 'team' service tag in custom_tags, and sort by cost.

With tags in place, attribute spend in USD to a team or project. To group spend by tag without writing SQL, go to AI Gateway > Govern > Usage Dashboard > Cost Analysis and use the Tag-based Cost Analysis section. Filter by endpoint tag or request tag to see Cost by Endpoint Tag Groups and Estimated Cost by Request Tag Groups.

To attribute spend with SQL, the table you query depends on the model type:

Service tags propagate to the custom_tags field in system.billing.usage, so you can attribute list-price spend in USD to a team the same way you ranked users in Step 1. Join system.billing.list_prices to convert DBU usage to dollars:

SQL
SELECT
  u.custom_tags['team'] AS team,
  SUM(u.usage_quantity * lp.pricing.effective_list.default) AS list_cost_usd
FROM system.billing.usage u
JOIN system.billing.list_prices lp
  ON  u.sku_name   = lp.sku_name
  AND u.cloud      = lp.cloud
  AND u.usage_unit = lp.usage_unit
  AND u.usage_end_time >= lp.price_start_time
  AND (lp.price_end_time IS NULL OR u.usage_end_time < lp.price_end_time)
WHERE u.billing_origin_product = 'MODEL_SERVING'
  AND u.usage_metadata.ai_gateway.endpoint_name IS NOT NULL
  AND u.custom_tags['team'] IS NOT NULL
  AND u.usage_unit = 'DBU'
  AND lp.currency_code = 'USD'
  AND u.usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY team
ORDER BY list_cost_usd DESC;

Step 5: Analyze cost per token by team or project​

prompt

Genie Code (Agent mode) can do this for you. Try this example prompt:

Query system.billing.usage and system.ai_gateway.usage for the past 30 days to show list-price USD per million tokens by model service, destination model, and the 'team' service tag. Convert DBUs to dollars using system.billing.list_prices at the price effective for each record, and join the two tables on usage date.

Total spend tells you where the budget went, but not whether you're paying a fair rate for it. Tracking inference spend as cost per 1M tokens lets you compare destination models, spot regressions after a routing change, or justify moving traffic to a cheaper model. Join the cost table to system.ai_gateway.usage on the model service, destination model, and team. Both tabs group by the team service tag, so the results line up with Step 3:

Aggregate list-price spend and tokens over the full window, then join on the model service, destination model, and team. Join system.billing.list_prices to convert DBU usage to dollars:

SQL
WITH cost AS (
  SELECT
    u.usage_metadata.ai_gateway.endpoint_name AS endpoint_name,
    u.usage_metadata.ai_gateway.destination_model AS destination_model,
    u.custom_tags['team'] AS team,
    SUM(u.usage_quantity * lp.pricing.effective_list.default) AS list_cost_usd
  FROM system.billing.usage u
  JOIN system.billing.list_prices lp
    ON  u.sku_name   = lp.sku_name
    AND u.cloud      = lp.cloud
    AND u.usage_unit = lp.usage_unit
    AND u.usage_end_time >= lp.price_start_time
    AND (lp.price_end_time IS NULL OR u.usage_end_time < lp.price_end_time)
  WHERE u.billing_origin_product = 'MODEL_SERVING'
    AND u.usage_metadata.ai_gateway.endpoint_name IS NOT NULL
    AND u.custom_tags['team'] IS NOT NULL
    AND u.usage_unit = 'DBU'
    AND lp.currency_code = 'USD'
    AND u.usage_date >= current_date() - INTERVAL 30 DAYS
  GROUP BY ALL
),
tokens AS (
  SELECT
    endpoint_name,
    destination_model,
    endpoint_tags['team'] AS team,
    SUM(total_tokens) AS total_tokens
  FROM system.ai_gateway.usage
  WHERE endpoint_tags['team'] IS NOT NULL
    AND endpoint_name IS NOT NULL
    AND event_time >= current_timestamp() - INTERVAL 30 DAYS
  GROUP BY ALL
)
SELECT
  c.endpoint_name,
  c.destination_model,
  c.team,
  c.list_cost_usd,
  t.total_tokens,
  1000000 * c.list_cost_usd / NULLIF(t.total_tokens, 0) AS list_cost_usd_per_1m_tokens
FROM cost c
JOIN tokens t
  ON  c.endpoint_name     = t.endpoint_name
  AND c.destination_model = t.destination_model
  AND c.team              = t.team
ORDER BY list_cost_usd_per_1m_tokens DESC;
note

Unity AI Gateway doesn't record token usage for non-streaming, non-embedding responses larger than 1 MiB, so cost per token can read high where those responses are common. Compare rates within a workload rather than across unrelated ones.

Next steps​

On this page