Measure Databricks Assistant impact

This article provides information on measuring the impact of Databricks Assistant in terms of adoption, egagement, and reported productivity gains.

Requirements

In order to measure the impact of Databricks Assistant, you need:

  • Account admin privileges required to enable system tables. See Enable system tables.

  • An internal survey to get subjective feedback on Assistant from your team.

Data vs. metrics

You get raw data from the system tables and a survey. To understand Assistant impact, you must analyze and report data as metrics. Metrics are calculated values you use to measure specific aspects or activities related to Assistant impact. This article also refers to metrics as measures.

Tips for measuring Assistant impact

To understand how your organization is using Databricks Assistant, start by measuring its adoption and engagement with Databricks Assistant. This data can be calculated from system tables.

Review your data routinely and make it easily available in a shared dashboard. For a dashboard example and template, see Databricks Assistant system table reference and example.

Assistant impact metrics

The following are recommended measures of Assistant impact, both from system tables and from user feedback. For examples of metrics calculations, download the Assistant events dashboard file from GitHub and read the calculations in JSON. To learn how to import a dashboard file, see Import a dashboard file

Surveying your organization also helps you understand the effectiveness of its engagement with Assistant. See Recommended survey questions.

Measure

Definition

Stage

Data source

Top users overall

Users in a given period who interact most frequently with Assistant

Adoption

Calculate from Databricks Assistant system table data.

Submissions data: per workspace and total

Number of requests submitted to Assistant per workspace and per account

Adoption

Calculate from Databricks Assistant system table data.

Active users by day and month

Unique users who have received and accepted 1+ suggestions or participated in 1+ chats on a given day.

Engagement

Calculate from Databricks Assistant system table data.

Active users per workspace

Unique users in a given workspace who interact with Assistant

Engagement

Calculate from Databricks Assistant system table data.

Top job roles using Databricks Assistant

Numbers of people from each role identified in your org who use Assistant

Engagement

Survey of your organization

Top tasks for Assistant

Most common tasks Assistant helps with

Engagement

Survey of your organization

How often do you use Assistant

Self-reported frequency of Assistant use per user

Engagement

Survey of your organization

Top areas of Assistant use

Self-reported usage areas: SQL editor, notebooks, or both

Engagement

Survey of your organization

User satisfaction with Assistant help

Self-reported satisfaction with Assistant’s answers on a scale of 1-5

User satisfaction

Survey of your organization

Gains in user productivity

Self-reported increase in productivity gained using Assistant on a scale of 1-5

User satisfaction

Survey of your organization

Amount of time users save using Assistant

Self-reported percentage of time saved by using Assistant to complete tasks

User satisfaction

Survey of your organization