What is an AI/BI Genie space?

Preview

This feature is in Public Preview. If Databricks Assistant is enabled at the account level, workspace admins can enable this feature from the Previews page. See Manage Databricks Previews.

AI/BI Genie is a no-code interface powered by DatabricksIQ that allows business users to conduct self-serve data analytics using natural language. Domain experts, such as data analysts, configure Genie spaces with datasets, sample queries, and text guidelines to help Genie translate business questions into analytical queries. After set up, business users can ask questions and generate visualizations to understand operational data. Genie continuously updates its semantic knowledge as your data changes and users pose new questions. See DatabricksIQ-powered features.

Data analysts can prepare a domain-specific Genie space experience for business users by doing the following:

Selecting relevant tables from Unity Catalog and exposing their metadata (table and column descriptions) in the Genie space.
Adding instructions that transfer organization-specific information (business logic and metadata) into the Genie space.

After initial setup, user can incrementally improve Genie’s understanding of their data and semantics over time by saving verified SQL answers as instructions. See Provide instructions.

Example use cases

You can create different Genie spaces to serve various non-technical audiences. The following scenarios describe two possible use cases.

Visualize opportunity status

A sales manager wants to get the current status of open and closed opportunities by stage in their sales pipeline. They can interact with the Genie space using natural language and automatically generate a visualization.

The following gif shows this interaction:

Gif with sample question, response, and auto-generated visualization

Tracking logistics

A logistics company wants to use Genie spaces to help business users from different departments track operational and financial details. They set up a Genie space for their shipment facility managers to track shipments and another for their financial executives to understand their financial health.

How are Genie space responses generated?

Genie spaces generate responses to natural language questions using table and column names and descriptions. The actual data in the tables remains hidden from Genie.

AI/BI Genie uses the names and descriptions to convert natural language questions to an equivalent SQL query. Then, it responds with the results of that query as a table. Genie space authors and business users can inspect the generated SQL query that produces each response.

Technical requirements

Genie spaces use data registered to Unity Catalog.
Genie spaces require a pro or serverless SQL warehouse.
Each workspace can have up too 1,000 Genie spaces.

Required permissions

The following permissions are required to create, edit, and use Genie spaces.

Note

Configuring data and compute access requires elevated permissions generally restricted to an administrator. See Create a SQL warehouse and Manage privileges in Unity Catalog.

Compute permissions: Creators, editors, and users require CAN USE access to the default SQL warehouse designated for the space. Authors select the default SQL warehouse when they create a space. The author can select any SQL warehouse for which they have CAN RUN permissions.
Data access permissions: Any user who interacts with the space needs at least SELECT privileges on the data used in a space.
Genie space permissions: Users need CAN RUN permissions on the Genie space to interact with Genie and the data used in the space. See Genie space ACLs for a complete mapping of privileges and abilities for a Genie space.

What data should I use?

You can create new Genie spaces based on one or more Unity Catalog managed tables. AI/BI Genie uses the metadata attached to Unity Catalog objects to generate responses. Well-annotated datasets, paired with specific instructions that you provide, are key to creating a positive experience for end users.

Databricks recommends the following:

Curate data for analytical consumption: Layer views to reduce the number of columns and add use-case-specific information to increase response quality.
Minimize the number of tables and columns in a Genie space: Bring in only the tables and columns needed to answer questions for a given domain. Unnecessary tables or columns can cause Genie to provide confusing or incorrect answers or display an error message.

Create a new Genie space

New space dialog as described in the list that follows.

When you create a new Genie space, a New space dialog shows the following options:

Title: The title appears in the workspace browser with other workspace objects. Choose a title that will help end users discover your Genie space.
Description: Users see the description when they open the Genie space. Use this text area to describe the spaces’s purpose.
Default warehouse: This compute resource powers the SQL statements generated in the Genie spaces. End users need access to this warehouse to interact with the space. A Genie space can use a Pro or Serverless SQL warehouse. For optimal performance, Databricks recommends using a Serverless SQL warehouse.
Tables: Genie spaces can be based on one or more tables. The Add tables button opens a new dialog where you can select the Unity Catalog tables you want to include.
Sample questions: Sample questions are optional. They appear in chats to provide examples of the types of questions users can ask. They can be added during creation or later from the space’s Settings.

Chat in the Genie space

Most Genie space interactions take place in the chat window.

New chat window as described in the text that follows.

A new chat window includes a curated set of sample questions. These sample questions help users understand what questions they can ask their Genie space. The text field, where users input questions, is near the bottom of the screen.

Responses appear above the text field. After a user enters a question, it is saved to a chat history thread in the left pane.

Chat history

Chat history threads are saved for each user so that they can refer to past questions and answers. Users can also resubmit or revise questions from a chat thread. The New chat button in the left pane starts a new thread.

Each chat thread maintains its context, so the Assistant considers previous questions it has been asked. This allows users to ask follow-up questions to further explore or refocus a result set.

Response structure

The precise response structure varies based on the question. Often, responses include a natural language explanation and a table showing the relevant result set. All responses include the SQL query that was generated to answer the question. Click Show generated code to view the generated query.

The bottom-right side of the response includes optional actions. You can copy the response CSV to your clipboard, download it as a CSV file, add it as an instruction for the Genie space, and upvote or downvote the answer.

A set of Quick actions tiles follow responses that include tabular data. You can use them to generate visualizations.

Quick action tiles that suggest different visualization options.

You can also generate a visualization by describing it in words after getting a query result.

Provide response feedback

Each AI/BI Genie response includes a thumbs-up or thumbs-down feedback button. Databricks recommends encouraging users to provide feedback on the space using this mechanism. The feedback can be used to tune responses and iterate on your space. It is accessible through the Monitoring page. See Monitor the space.

Monitor the space

A Genie space is meant to be a continuous collaboration tool for data teams and business users to accumulate knowledge, not a one-time deployment. Understanding the new questions that business users need answers to is essential for knowledge building. The Monitoring page allows you to view individual questions and responses as well as thumbs-up or thumbs-down feedback provided by users working with the space. Click Monitoring in the Genie space side bar to open the monitoring page.

An example monitoring page showing the described features.

The Monitoring page shows all of the questions and answers that have been asked in the space. You can filter questions by time, rating, user, or status. Monitoring the space can help Genie authors proactively understand the queries raised by business users and how the Genie space responded.

By identifying the questions that Genie struggles with, you can update the Genie space with specific instructions to improve its responses. Click a question to open the question and response text. You can also access the complete chat thread from this view.

Provide instructions

Instructions help to guide Genie’s responses so that it can process the unique jargon, logic, and concepts in a given domain. You can write instructions as example queries or snippets of plain text that help Genie answer questions that space users are likely to ask. Comprehensive instructions are critical to a seamless, intuitive Genie space experience.

Click Instructions in the Genie space side bar to open the instructions pane.

The following examples illustrate various types of instructions:

Company-specific business information:
- “Our fiscal year starts in February”
Values, aliases, or common filters:
- “Always convert to lowercase and use a like operator when applying filters.”
- “Use abbreviations for states in filter values.”
User-defined functions available through Unity Catalog:
- “For quarters use the adventureworks.oneb.get_quarter(date) UDF. The output of get_quarter is the quarter and is either 1,2,3, or 4. Use this to filter the data as needed.
  
  For example, for quarter 3, use where adventureworks.oneb.get_quarter(posted_date)= 3”`”

Sample SQL queries:

You can provide samples of queries that you expect Genie to generate.
Focus on providing samples that highlight logic that is unique to your organization and data, as in the following example:

-- Return our current total open pipeline by region.
-- Opportunities are only considered pipelines if they are tagged as such.
SELECT
  a.region__c AS `Region`,
  sum(o.amount) AS `Open Pipeline`
FROM
  sales.crm.opportunity o
  JOIN sales.crm.accounts a ON o.accountid = a.id
WHERE
  o.forecastcategory = 'Pipeline' AND
  o.stagename NOT ILIKE '%closed%'
GROUP BY ALL;

You can organize Genie space instructions as one long note or group them by related topics for better structure.

Edit generated SQL

You can review and edit the generated SQL in any response. Genie space authors typically have knowledge of the domain and data that allows them to recognize when Genie is generating an incorrect answer. Often, errors can be fixed with a small amount of manual tuning in the generated SQL query itself. To view the generated SQL for any response, click Show generated code to inspect the query.

Users with at least CAN EDIT privileges on a Genie space can edit the generated SQL statement to correct it. After editing, run your query. Then, you can save it as an instruction to teach Genie how to answer in the future. To save your edited query, click Save as instruction.

Best practices for space preparation

Include a set of well-defined questions that you want space users to be able to answer.
Test your Genie space to check response quality. Try the following to see if the model provides the expected response:
- Rephrase the provided questions.
- Ask other questions related to the datasets.
Add and refine Genie space instructions until questions provide the expected response.