Eval Run

View as Markdown

A benchmark evaluation run. The public benchmark API currently evaluates chat-mode responses.

GenieEvalRunResponse object

A benchmark evaluation run. The public benchmark API currently evaluates chat-mode responses.

eval_run_idstringuuidBeta: The unique identifier for the evaluation run.
Example: e1ef34712a29169db030324fd0e1df5f

eval_run_statusstringBeta

Current status of the evaluation run.

Values:

EVALUATION_STATUS_TYPE_UNSPECIFIED
RUNNING
DONE
NOT_STARTED
EVALUATION_FAILED
EVALUATION_CANCELLED
EVALUATION_TIMEOUT

run_by_userint64Beta: User ID who initiated the evaluation run.

created_timestampint64Beta: Timestamp when the evaluation run was created (milliseconds since epoch).

num_questionsint64Beta: Total number of questions in the evaluation run.

num_correctint64Beta: Number of questions answered correctly.

num_needs_reviewint64Beta: Number of questions that need manual review.

num_doneint64Beta: Number of questions that have been completed.

last_updated_timestampint64Beta: Timestamp when the evaluation run was last updated (milliseconds since epoch).

View as Markdown

The GenieEvalRunResponse object

{
  "eval_run_id": "e1ef34712a29169db030324fd0e1df5f",
  "eval_run_status": "RUNNING",
  "run_by_user": 0,
  "created_timestamp": 0,
  "num_questions": 0,
  "num_correct": 0,
  "num_needs_review": 0,
  "num_done": 0,
  "last_updated_timestamp": 0
}

Get a benchmark evaluation run Beta

GET /api/2.0/genie/spaces/{space_id}/eval-runs/{eval_run_id}

Get evaluation run details.

API scopes: genie

Parameters

space_idstringuuidRequiredpath: The ID associated with the Genie space where the evaluation run is located.
Example: e1ef34712a29169db030324fd0e1df5f

eval_run_idstringuuidRequiredpath: Example: e1ef34712a29169db030324fd0e1df5f

Response

Returns the GenieEvalRunResponse object.

View as Markdown

Response

{
  "eval_run_id": "e1ef34712a29169db030324fd0e1df5f",
  "eval_run_status": "RUNNING",
  "run_by_user": 0,
  "created_timestamp": 0,
  "num_questions": 0,
  "num_correct": 0,
  "num_needs_review": 0,
  "num_done": 0,
  "last_updated_timestamp": 0
}

List benchmark evaluation runs Beta

GET /api/2.0/genie/spaces/{space_id}/eval-runs

Lists all evaluation runs in a space.

API scopes: genie

Parameters

space_idstringuuidRequiredpath: The ID associated with the Genie space where the evaluation run is located.
Example: e1ef34712a29169db030324fd0e1df5f

page_sizeint32<= 100query: Maximum number of evaluation runs to return per page
Default: 20

page_tokenstringquery: Token to get the next page of results

Response

Returns a list of GenieEvalRunResponse objects.

View as Markdown

Response

{
  "eval_runs": [
    {
      "eval_run_id": "e1ef34712a29169db030324fd0e1df5f",
      "eval_run_status": "RUNNING",
      "run_by_user": 0,
      "created_timestamp": 0,
      "num_questions": 0,
      "num_correct": 0,
      "num_needs_review": 0,
      "num_done": 0,
      "last_updated_timestamp": 0
    }
  ],
  "next_page_token": "string"
}

Start a benchmark evaluation run Beta

POST /api/2.0/genie/spaces/{space_id}/eval-runs

Creates and runs chat-mode evaluations for multiple benchmark questions in a Genie space.

API scopes: genie

Parameters

space_idstringuuidRequiredpath: The ID associated with the Genie space where the evaluations will be executed.
Example: e1ef34712a29169db030324fd0e1df5f

Request body

benchmark_question_idsarray of string: List of benchmark question IDs to evaluate. These questions must exist in the specified Genie space. If none are specified, then all benchmark questions are evaluated.

Response

Returns the GenieEvalRunResponse object.

View as Markdown

Request

{
  "benchmark_question_ids": [
    "string"
  ]
}

Response

{
  "eval_run_id": "e1ef34712a29169db030324fd0e1df5f",
  "eval_run_status": "RUNNING",
  "run_by_user": 0,
  "created_timestamp": 0,
  "num_questions": 0,
  "num_correct": 0,
  "num_needs_review": 0,
  "num_done": 0,
  "last_updated_timestamp": 0
}

Get benchmark evaluation result details Beta

GET /api/2.0/genie/spaces/{space_id}/eval-runs/{eval_run_id}/results/{result_id}

Get details for evaluation results.

API scopes: genie

Parameters

space_idstringuuidRequiredpath: The ID associated with the Genie space where the evaluation run is located.
Example: e1ef34712a29169db030324fd0e1df5f

eval_run_idstringuuidRequiredpath: The unique identifier for the evaluation run.
Example: e1ef34712a29169db030324fd0e1df5f

result_idstringuuidRequiredpath: The unique identifier for the evaluation result.
Example: e1ef34712a29169db030324fd0e1df5f

Response

result_idstringRequired: The unique identifier for the evaluation result.

space_idstringRequired: The ID of the space the evaluation result belongs to.

benchmark_question_idstringRequired: The ID of the benchmark question that was evaluated.

eval_run_statusstringOutput only

Current status of the evaluation run.

Values:

EVALUATION_STATUS_TYPE_UNSPECIFIED
RUNNING
DONE
NOT_STARTED
EVALUATION_FAILED
EVALUATION_CANCELLED
EVALUATION_TIMEOUT

assessmentstringOutput only

Assessment of the evaluation result: good, bad, or needs review

Values:

GENIE_EVAL_ASSESSMENT_UNSPECIFIED
GOOD
BAD
NEEDS_REVIEW

manual_assessmentbooleanOutput only: Whether this evaluation was manually assessed.

assessment_reasonsarray of stringOutput only

Reasons for the assessment score.

Assessment reasons describe why a Genie response was scored as BAD.

Deterministic values (compared against the ground truth result):

EMPTY_RESULT: Genie's generated SQL results were empty for this benchmark question.
RESULT_MISSING_ROWS: Genie's generated SQL response is missing rows from the provided ground truth SQL.
RESULT_EXTRA_ROWS: Genie's generated SQL response has more rows than the provided ground truth SQL.
RESULT_MISSING_COLUMNS: Genie's generated SQL response is missing columns from the provided ground truth SQL.
RESULT_EXTRA_COLUMNS: Genie's generated SQL response has more columns than the provided ground truth SQL.
SINGLE_CELL_DIFFERENCE: Single value result was produced but differs from ground truth result.
EMPTY_GOOD_SQL: The benchmark SQL returned an empty result.
COLUMN_TYPE_DIFFERENCE: The values between the results match but the column type is different.

LLM judge ratings explain the factors driving BAD results:

LLM_JUDGE_MISSING_OR_INCORRECT_FILTER: Genie's generated SQL is missing a WHERE clause condition or has incorrect filter logic that excludes/includes wrong data.
LLM_JUDGE_INCOMPLETE_OR_PARTIAL_OUTPUT: Genie's generated SQL returns only some of the requested data or columns, missing parts of what the ground truth SQL returns.
LLM_JUDGE_MISINTERPRETATION_OF_USER_REQUEST: Genie's generated SQL fundamentally misunderstands what the user is asking for, addressing the wrong question or goal.
LLM_JUDGE_INSTRUCTION_COMPLIANCE_OR_MISSING_BUSINESS_LOGIC: Genie's generated SQL fails to apply specified instructions or business logic that should be followed.
LLM_JUDGE_INCORRECT_METRIC_CALCULATION: Genie's generated SQL uses incorrect logic or makes wrong assumptions when calculating metrics.
LLM_JUDGE_INCORRECT_TABLE_OR_FIELD_USAGE: Genie's generated SQL references wrong tables, columns, or uses fields that don't match the ground truth SQL's intent.
LLM_JUDGE_INCORRECT_FUNCTION_USAGE: Genie's generated SQL uses SQL functions incorrectly or inappropriately (wrong parameters, wrong function for the task, etc.).
LLM_JUDGE_MISSING_OR_INCORRECT_JOIN: Genie's generated SQL is missing necessary joins between tables or has incorrect join conditions/types that produce wrong results.
LLM_JUDGE_MISSING_OR_INCORRECT_AGGREGATION: Genie's generated SQL is missing GROUP BY clauses or has incorrect grouping that doesn't match the requested aggregation level.
LLM_JUDGE_FORMATTING_ERROR: Genie's generated SQL output has incorrect formatting, ordering (ORDER BY), or presentation issues that don't match expectations.
LLM_JUDGE_OTHER: LLM judge identified an error that doesn't fall into other categories.

Deprecated LLM judge values (kept for backward compatibility, do not use):

LLM_JUDGE_MISSING_JOIN (deprecated)
LLM_JUDGE_WRONG_FILTER (deprecated)
LLM_JUDGE_WRONG_AGGREGATION (deprecated)
LLM_JUDGE_WRONG_COLUMNS (deprecated)
LLM_JUDGE_SYNTAX_ERROR (deprecated)
LLM_JUDGE_SEMANTIC_ERROR (deprecated)

Values:

Show 26 possible values

SCORE_REASON_UNSPECIFIED
EMPTY_RESULT
RESULT_MISSING_ROWS
RESULT_EXTRA_ROWS
RESULT_MISSING_COLUMNS
RESULT_EXTRA_COLUMNS
SINGLE_CELL_DIFFERENCE
EMPTY_GOOD_SQL
COLUMN_TYPE_DIFFERENCE
LLM_JUDGE_MISSING_JOIN
LLM_JUDGE_WRONG_FILTER
LLM_JUDGE_WRONG_AGGREGATION
LLM_JUDGE_WRONG_COLUMNS
LLM_JUDGE_SYNTAX_ERROR
LLM_JUDGE_SEMANTIC_ERROR
LLM_JUDGE_OTHER
LLM_JUDGE_MISSING_OR_INCORRECT_FILTER
LLM_JUDGE_INCOMPLETE_OR_PARTIAL_OUTPUT
LLM_JUDGE_MISINTERPRETATION_OF_USER_REQUEST
LLM_JUDGE_INSTRUCTION_COMPLIANCE_OR_MISSING_BUSINESS_LOGIC
LLM_JUDGE_INCORRECT_METRIC_CALCULATION
LLM_JUDGE_INCORRECT_TABLE_OR_FIELD_USAGE
LLM_JUDGE_INCORRECT_FUNCTION_USAGE
LLM_JUDGE_MISSING_OR_INCORRECT_JOIN
LLM_JUDGE_MISSING_OR_INCORRECT_AGGREGATION
LLM_JUDGE_FORMATTING_ERROR

actual_responsearray of objectOutput only

The actual response generated by Genie.

Show child attributesHide child attributes

responsestringOutput only: The response content (either text or SQL query).

sql_execution_resultobjectOutput only

SQL Statement Execution response.

Show child attributesHide child attributes

statement_idstring: The statement ID is returned upon successfully submitting a SQL statement, and is a required reference for all subsequent calls.

statusobject

Show child attributesHide child attributes

statestring

Statement execution state:

PENDING: waiting for warehouse
RUNNING: running
SUCCEEDED: execution was successful, result data available for fetch
FAILED: execution failed; reason for failure described in accompanying error message
CANCELED: user canceled; can come from explicit cancel call, or timeout with on_wait_timeout=CANCEL
CLOSED: execution successful, and statement closed; result no longer available for fetch

Values:

STATE_UNSPECIFIED
PENDING
RUNNING
SUCCEEDED
FAILED
CANCELED
CLOSED

errorobject

Show child attributesHide child attributes

error_codestring

Error codes returned by Databricks APIs to indicate specific failure conditions.

Values:

Show 81 possible values

UNKNOWN
INTERNAL_ERROR
TEMPORARILY_UNAVAILABLE
IO_ERROR
BAD_REQUEST
SERVICE_UNDER_MAINTENANCE
WORKSPACE_TEMPORARILY_UNAVAILABLE
DEADLINE_EXCEEDED
CANCELLED
RESOURCE_EXHAUSTED
ABORTED
NOT_FOUND
ALREADY_EXISTS
UNAUTHENTICATED
UNAVAILABLE
INVALID_PARAMETER_VALUE
ENDPOINT_NOT_FOUND
MALFORMED_REQUEST
INVALID_STATE
PERMISSION_DENIED
FEATURE_DISABLED
CUSTOMER_UNAUTHORIZED
REQUEST_LIMIT_EXCEEDED
RESOURCE_CONFLICT
UNPARSEABLE_HTTP_ERROR
NOT_IMPLEMENTED
DATA_LOSS
INVALID_STATE_TRANSITION
COULD_NOT_ACQUIRE_LOCK
RESOURCE_ALREADY_EXISTS
RESOURCE_DOES_NOT_EXIST
QUOTA_EXCEEDED
MAX_BLOCK_SIZE_EXCEEDED
MAX_READ_SIZE_EXCEEDED
PARTIAL_DELETE
MAX_LIST_SIZE_EXCEEDED
DRY_RUN_FAILED
RESOURCE_LIMIT_EXCEEDED
DIRECTORY_NOT_EMPTY
DIRECTORY_PROTECTED
MAX_NOTEBOOK_SIZE_EXCEEDED
MAX_CHILD_NODE_SIZE_EXCEEDED
SEARCH_QUERY_TOO_LONG
SEARCH_QUERY_TOO_SHORT
MANAGED_RESOURCE_GROUP_DOES_NOT_EXIST
PERMISSION_NOT_PROPAGATED
DEPLOYMENT_TIMEOUT
GIT_CONFLICT
GIT_UNKNOWN_REF
GIT_SENSITIVE_TOKEN_DETECTED
GIT_URL_NOT_ON_ALLOW_LIST
GIT_REMOTE_ERROR
PROJECTS_OPERATION_TIMEOUT
IPYNB_FILE_IN_REPO
INSECURE_PARTNER_RESPONSE
MALFORMED_PARTNER_RESPONSE
METASTORE_DOES_NOT_EXIST
DAC_DOES_NOT_EXIST
CATALOG_DOES_NOT_EXIST
SCHEMA_DOES_NOT_EXIST
TABLE_DOES_NOT_EXIST
SHARE_DOES_NOT_EXIST
RECIPIENT_DOES_NOT_EXIST
STORAGE_CREDENTIAL_DOES_NOT_EXIST
EXTERNAL_LOCATION_DOES_NOT_EXIST
PRINCIPAL_DOES_NOT_EXIST
PROVIDER_DOES_NOT_EXIST
METASTORE_ALREADY_EXISTS
DAC_ALREADY_EXISTS
CATALOG_ALREADY_EXISTS
SCHEMA_ALREADY_EXISTS
TABLE_ALREADY_EXISTS
SHARE_ALREADY_EXISTS
RECIPIENT_ALREADY_EXISTS
STORAGE_CREDENTIAL_ALREADY_EXISTS
EXTERNAL_LOCATION_ALREADY_EXISTS
PROVIDER_ALREADY_EXISTS
CATALOG_NOT_EMPTY
SCHEMA_NOT_EMPTY
METASTORE_NOT_EMPTY
PROVIDER_SHARE_NOT_ACCESSIBLE

messagestring

stack_tracestring

sql_statestring: SQLSTATE error code returned when the statement execution fails. Only populated when the statement status is FAILED.

manifestobject

Show child attributesHide child attributes

formatstring

Values:

FORMAT_UNSPECIFIED
JSON_ARRAY
ARROW_STREAM
CSV

schemaobject

Show child attributesHide child attributes

column_countint32

columnsarray of object

Show child attributesHide child attributes

namestring: Name of Column.

type_textstring: Full data type specification as SQL/catalogString text.

type_namestring

Values:

Show 24 possible values

BOOLEAN
BYTE
SHORT
INT
LONG
FLOAT
DOUBLE
DATE
TIMESTAMP
STRING
BINARY
DECIMAL
INTERVAL
ARRAY
STRUCT
MAP
CHAR
NULL
USER_DEFINED_TYPE
TIMESTAMP_NTZ
VARIANT
GEOMETRY
GEOGRAPHY
TABLE_TYPE

positionint32: Ordinal position of column (starting at position 0).

type_precisionint32: Digits of precision; required for DecimalTypes.

type_scaleint32: Digits to right of decimal; Required for DecimalTypes.

type_interval_typestring: Format of IntervalType.

type_jsonstring: Full data type specification, JSON-serialized.

commentstring: User-provided free-form text description.

nullableboolean: Whether field may be Null (default: true).
Default: true

partition_indexint32: Partition index for column.

maskobject

Show child attributesHide child attributes

function_namestring: The full name of the column mask SQL UDF.

using_column_namesarray of string: The list of additional table columns to be passed as input to the column mask function. The first arg of the mask function should be of the type of the column being masked and the types of the rest of the args should match the types of columns in 'using_column_names'.

using_argumentsarray of objectDeprecated

The list of table columns or literals to be passed as additional arguments to a column mask function, carrying the type (column reference vs constant literal) of each argument. Deprecated: use using_column_names instead.

Show child attributesHide child attributes

columnstring: A column reference.

constantstring: A constant literal.

total_chunk_countint32: The total number of chunks that the result set has been divided into.

chunksarray of object

Array of result set chunk metadata.

Show child attributesHide child attributes

chunk_indexint32: The position within the sequence of result set chunks.

row_offsetint64: The starting row offset within the result set.

row_countint64: The number of rows within the result chunk.

byte_countint64: The number of bytes in the result chunk. This field is not available when using INLINE disposition.

next_chunk_indexint32: When fetching, provides the chunk_index for the next chunk. If absent, indicates there are no more chunks. The next chunk can be fetched with a statementexecution/getstatementresultchunkn request.

next_chunk_internal_linkstring: When fetching, provides a link to fetch the next chunk. If absent, indicates there are no more chunks. This link is an absolute path to be joined with your $DATABRICKS_HOST, and should be treated as an opaque link. This is an alternative to using next_chunk_index.

total_row_countint64: The total number of rows in the result set.

total_byte_countint64: The total number of bytes in the result set. This field is not available when using INLINE disposition.

truncatedboolean: Indicates whether the result is truncated due to row_limit or byte_limit.

resultobject

Show child attributesHide child attributes

external_linksarray of object

Show child attributesHide child attributes

external_linkstring: A short-lived cloud-storage URL pointing to a chunk of result data, hosted by an external service, with a short expiration time (<= 15 minutes). As this URL contains a temporary credential, it should be considered sensitive and the client should not expose this URL in a log.
AWS
On AWS, this is a presigned URL.
Azure
On Azure, this is a SAS URL.
GCP
On GCP, this is a signed URL.

expirationstringdate-time: Indicates the date-time that the given external link will expire and becomes invalid, after which point a new external_link must be requested.

chunk_indexint32: The position within the sequence of result set chunks.

row_offsetint64: The starting row offset within the result set.

row_countint64: The number of rows within the result chunk.

byte_countint64: The number of bytes in the result chunk. This field is not available when using INLINE disposition.

next_chunk_indexint32: When fetching, provides the chunk_index for the next chunk. If absent, indicates there are no more chunks. The next chunk can be fetched with a statementexecution/getstatementresultchunkn request.

next_chunk_internal_linkstring: When fetching, provides a link to fetch the next chunk. If absent, indicates there are no more chunks. This link is an absolute path to be joined with your $DATABRICKS_HOST, and should be treated as an opaque link. This is an alternative to using next_chunk_index.

data_arrayarray of object

The JSON_ARRAY format is an array of arrays of values, where each non-null value is formatted as a string. Null values are encoded as JSON null.

Show child attributesHide child attributes

valuesarray of object

Repeated field of dynamically typed values.

Show child attributesHide child attributes

null_valuestring

Represents a null value.

Values:

NULL_VALUE

number_valuedouble: Represents a double value.

string_valuestring: Represents a string value.

bool_valueboolean: Represents a boolean value.

struct_valueobject

Represents a structured value.

Show child attributesHide child attributes

fieldsarray of object

Unordered map of dynamically typed values.

Show child attributesHide child attributes

keystring

chunk_indexint32: The position within the sequence of result set chunks.

row_offsetint64: The starting row offset within the result set.

row_countint64: The number of rows within the result chunk.

byte_countint64: The number of bytes in the result chunk. This field is not available when using INLINE disposition.

next_chunk_indexint32: When fetching, provides the chunk_index for the next chunk. If absent, indicates there are no more chunks. The next chunk can be fetched with a statementexecution/getstatementresultchunkn request.

next_chunk_internal_linkstring: When fetching, provides a link to fetch the next chunk. If absent, indicates there are no more chunks. This link is an absolute path to be joined with your $DATABRICKS_HOST, and should be treated as an opaque link. This is an alternative to using next_chunk_index.

response_typestringOutput only

Type of response

Values:

GENIE_EVAL_RESPONSE_TYPE_UNSPECIFIED
TEXT
SQL

expected_responsearray of objectOutput only

The expected responses from the benchmark.

Show child attributesHide child attributes

responsestringOutput only: The response content (either text or SQL query).

sql_execution_resultobjectOutput only

SQL Statement Execution response.

Show child attributesHide child attributes

statement_idstring: The statement ID is returned upon successfully submitting a SQL statement, and is a required reference for all subsequent calls.

statusobject

Show child attributesHide child attributes

statestring

Statement execution state:

PENDING: waiting for warehouse
RUNNING: running
SUCCEEDED: execution was successful, result data available for fetch
FAILED: execution failed; reason for failure described in accompanying error message
CANCELED: user canceled; can come from explicit cancel call, or timeout with on_wait_timeout=CANCEL
CLOSED: execution successful, and statement closed; result no longer available for fetch

Values:

STATE_UNSPECIFIED
PENDING
RUNNING
SUCCEEDED
FAILED
CANCELED
CLOSED

errorobject

Show child attributesHide child attributes

error_codestring

Error codes returned by Databricks APIs to indicate specific failure conditions.

Values:

Show 81 possible values

UNKNOWN
INTERNAL_ERROR
TEMPORARILY_UNAVAILABLE
IO_ERROR
BAD_REQUEST
SERVICE_UNDER_MAINTENANCE
WORKSPACE_TEMPORARILY_UNAVAILABLE
DEADLINE_EXCEEDED
CANCELLED
RESOURCE_EXHAUSTED
ABORTED
NOT_FOUND
ALREADY_EXISTS
UNAUTHENTICATED
UNAVAILABLE
INVALID_PARAMETER_VALUE
ENDPOINT_NOT_FOUND
MALFORMED_REQUEST
INVALID_STATE
PERMISSION_DENIED
FEATURE_DISABLED
CUSTOMER_UNAUTHORIZED
REQUEST_LIMIT_EXCEEDED
RESOURCE_CONFLICT
UNPARSEABLE_HTTP_ERROR
NOT_IMPLEMENTED
DATA_LOSS
INVALID_STATE_TRANSITION
COULD_NOT_ACQUIRE_LOCK
RESOURCE_ALREADY_EXISTS
RESOURCE_DOES_NOT_EXIST
QUOTA_EXCEEDED
MAX_BLOCK_SIZE_EXCEEDED
MAX_READ_SIZE_EXCEEDED
PARTIAL_DELETE
MAX_LIST_SIZE_EXCEEDED
DRY_RUN_FAILED
RESOURCE_LIMIT_EXCEEDED
DIRECTORY_NOT_EMPTY
DIRECTORY_PROTECTED
MAX_NOTEBOOK_SIZE_EXCEEDED
MAX_CHILD_NODE_SIZE_EXCEEDED
SEARCH_QUERY_TOO_LONG
SEARCH_QUERY_TOO_SHORT
MANAGED_RESOURCE_GROUP_DOES_NOT_EXIST
PERMISSION_NOT_PROPAGATED
DEPLOYMENT_TIMEOUT
GIT_CONFLICT
GIT_UNKNOWN_REF
GIT_SENSITIVE_TOKEN_DETECTED
GIT_URL_NOT_ON_ALLOW_LIST
GIT_REMOTE_ERROR
PROJECTS_OPERATION_TIMEOUT
IPYNB_FILE_IN_REPO
INSECURE_PARTNER_RESPONSE
MALFORMED_PARTNER_RESPONSE
METASTORE_DOES_NOT_EXIST
DAC_DOES_NOT_EXIST
CATALOG_DOES_NOT_EXIST
SCHEMA_DOES_NOT_EXIST
TABLE_DOES_NOT_EXIST
SHARE_DOES_NOT_EXIST
RECIPIENT_DOES_NOT_EXIST
STORAGE_CREDENTIAL_DOES_NOT_EXIST
EXTERNAL_LOCATION_DOES_NOT_EXIST
PRINCIPAL_DOES_NOT_EXIST
PROVIDER_DOES_NOT_EXIST
METASTORE_ALREADY_EXISTS
DAC_ALREADY_EXISTS
CATALOG_ALREADY_EXISTS
SCHEMA_ALREADY_EXISTS
TABLE_ALREADY_EXISTS
SHARE_ALREADY_EXISTS
RECIPIENT_ALREADY_EXISTS
STORAGE_CREDENTIAL_ALREADY_EXISTS
EXTERNAL_LOCATION_ALREADY_EXISTS
PROVIDER_ALREADY_EXISTS
CATALOG_NOT_EMPTY
SCHEMA_NOT_EMPTY
METASTORE_NOT_EMPTY
PROVIDER_SHARE_NOT_ACCESSIBLE

messagestring

stack_tracestring

sql_statestring: SQLSTATE error code returned when the statement execution fails. Only populated when the statement status is FAILED.

manifestobject

Show child attributesHide child attributes

formatstring

Values:

FORMAT_UNSPECIFIED
JSON_ARRAY
ARROW_STREAM
CSV

schemaobject

Show child attributesHide child attributes

column_countint32

columnsarray of object

Show child attributesHide child attributes

namestring: Name of Column.

type_textstring: Full data type specification as SQL/catalogString text.

type_namestring

Values:

Show 24 possible values

BOOLEAN
BYTE
SHORT
INT
LONG
FLOAT
DOUBLE
DATE
TIMESTAMP
STRING
BINARY
DECIMAL
INTERVAL
ARRAY
STRUCT
MAP
CHAR
NULL
USER_DEFINED_TYPE
TIMESTAMP_NTZ
VARIANT
GEOMETRY
GEOGRAPHY
TABLE_TYPE

positionint32: Ordinal position of column (starting at position 0).

type_precisionint32: Digits of precision; required for DecimalTypes.

type_scaleint32: Digits to right of decimal; Required for DecimalTypes.

type_interval_typestring: Format of IntervalType.

type_jsonstring: Full data type specification, JSON-serialized.

commentstring: User-provided free-form text description.

nullableboolean: Whether field may be Null (default: true).
Default: true

partition_indexint32: Partition index for column.

maskobject

Show child attributesHide child attributes

function_namestring: The full name of the column mask SQL UDF.

using_column_namesarray of string: The list of additional table columns to be passed as input to the column mask function. The first arg of the mask function should be of the type of the column being masked and the types of the rest of the args should match the types of columns in 'using_column_names'.

using_argumentsarray of objectDeprecated

Show child attributesHide child attributes

columnstring: A column reference.

constantstring: A constant literal.

total_chunk_countint32: The total number of chunks that the result set has been divided into.

chunksarray of object

Array of result set chunk metadata.

Show child attributesHide child attributes

chunk_indexint32: The position within the sequence of result set chunks.

row_offsetint64: The starting row offset within the result set.

row_countint64: The number of rows within the result chunk.

byte_countint64: The number of bytes in the result chunk. This field is not available when using INLINE disposition.

next_chunk_indexint32: When fetching, provides the chunk_index for the next chunk. If absent, indicates there are no more chunks. The next chunk can be fetched with a statementexecution/getstatementresultchunkn request.

next_chunk_internal_linkstring: When fetching, provides a link to fetch the next chunk. If absent, indicates there are no more chunks. This link is an absolute path to be joined with your $DATABRICKS_HOST, and should be treated as an opaque link. This is an alternative to using next_chunk_index.

total_row_countint64: The total number of rows in the result set.

total_byte_countint64: The total number of bytes in the result set. This field is not available when using INLINE disposition.

truncatedboolean: Indicates whether the result is truncated due to row_limit or byte_limit.

resultobject

Show child attributesHide child attributes

external_linksarray of object

Show child attributesHide child attributes

external_linkstring: A short-lived cloud-storage URL pointing to a chunk of result data, hosted by an external service, with a short expiration time (<= 15 minutes). As this URL contains a temporary credential, it should be considered sensitive and the client should not expose this URL in a log.
AWS
On AWS, this is a presigned URL.
Azure
On Azure, this is a SAS URL.
GCP
On GCP, this is a signed URL.

expirationstringdate-time: Indicates the date-time that the given external link will expire and becomes invalid, after which point a new external_link must be requested.

chunk_indexint32: The position within the sequence of result set chunks.

row_offsetint64: The starting row offset within the result set.

row_countint64: The number of rows within the result chunk.

byte_countint64: The number of bytes in the result chunk. This field is not available when using INLINE disposition.

next_chunk_indexint32: When fetching, provides the chunk_index for the next chunk. If absent, indicates there are no more chunks. The next chunk can be fetched with a statementexecution/getstatementresultchunkn request.

next_chunk_internal_linkstring: When fetching, provides a link to fetch the next chunk. If absent, indicates there are no more chunks. This link is an absolute path to be joined with your $DATABRICKS_HOST, and should be treated as an opaque link. This is an alternative to using next_chunk_index.

data_arrayarray of object

The JSON_ARRAY format is an array of arrays of values, where each non-null value is formatted as a string. Null values are encoded as JSON null.

Show child attributesHide child attributes

valuesarray of object

Repeated field of dynamically typed values.

Show child attributesHide child attributes

null_valuestring

Represents a null value.

Values:

NULL_VALUE

number_valuedouble: Represents a double value.

string_valuestring: Represents a string value.

bool_valueboolean: Represents a boolean value.

struct_valueobject

Represents a structured value.

Show child attributesHide child attributes

fieldsarray of object

Unordered map of dynamically typed values.

Show child attributesHide child attributes

keystring

chunk_indexint32: The position within the sequence of result set chunks.

row_offsetint64: The starting row offset within the result set.

row_countint64: The number of rows within the result chunk.

byte_countint64: The number of bytes in the result chunk. This field is not available when using INLINE disposition.

next_chunk_indexint32: When fetching, provides the chunk_index for the next chunk. If absent, indicates there are no more chunks. The next chunk can be fetched with a statementexecution/getstatementresultchunkn request.

next_chunk_internal_linkstring: When fetching, provides a link to fetch the next chunk. If absent, indicates there are no more chunks. This link is an absolute path to be joined with your $DATABRICKS_HOST, and should be treated as an opaque link. This is an alternative to using next_chunk_index.

response_typestringOutput only

Type of response

Values:

GENIE_EVAL_RESPONSE_TYPE_UNSPECIFIED
TEXT
SQL

View as Markdown

Response

{
  "result_id": "string",
  "space_id": "string",
  "benchmark_question_id": "string",
  "eval_run_status": "RUNNING",
  "assessment": "GOOD",
  "manual_assessment": true,
  "assessment_reasons": [
    "EMPTY_RESULT"
  ],
  "actual_response": [
    {
      "response": "string",
      "sql_execution_result": {},
      "response_type": "TEXT"
    }
  ],
  "expected_response": [
    {
      "response": "string",
      "sql_execution_result": {},
      "response_type": "TEXT"
    }
  ]
}

List results for a benchmark evaluation run Beta

GET /api/2.0/genie/spaces/{space_id}/eval-runs/{eval_run_id}/results

List evaluation results for a specific evaluation run.

API scopes: genie

Parameters

space_idstringuuidRequiredpath: The ID associated with the Genie space where the evaluation run is located.
Example: e1ef34712a29169db030324fd0e1df5f

eval_run_idstringuuidRequiredpath: The unique identifier for the evaluation run.
Example: e1ef34712a29169db030324fd0e1df5f

page_sizeint32<= 100query: Maximum number of eval results to return per page.
Default: 20

page_tokenstringquery: Opaque token to retrieve the next page of results.

Response

eval_resultsarray of object

List of evaluation results for the specified run.

Show child attributesHide child attributes

result_idstringuuidRequired: Unique identifier for this evaluation result.
Example: e1ef34712a29169db030324fd0e1df5f

space_idstringuuidRequired: The ID of the space the evaluation result belongs to.
Example: e1ef34712a29169db030324fd0e1df5f

benchmark_question_idstringuuidRequired: The ID of the benchmark question that was evaluated.
Example: e1ef34712a29169db030324fd0e1df5f

statusstringOutput only

Current status of this evaluation result.

Values:

EVALUATION_STATUS_TYPE_UNSPECIFIED
RUNNING
DONE
NOT_STARTED
EVALUATION_FAILED
EVALUATION_CANCELLED
EVALUATION_TIMEOUT

questionstringOutput only: Stored snapshot of original benchmark question text.

benchmark_answerstringOutput only: Stored snapshot of original benchmark answer text.

created_by_userint64Output only: User ID who created evaluation result.

next_page_tokenstring: The token to use for retrieving the next page of results.

View as Markdown

Response

{
  "eval_results": [
    {
      "result_id": "e1ef34712a29169db030324fd0e1df5f",
      "space_id": "e1ef34712a29169db030324fd0e1df5f",
      "benchmark_question_id": "e1ef34712a29169db030324fd0e1df5f",
      "status": "RUNNING",
      "question": "string",
      "benchmark_answer": "string",
      "created_by_user": 0
    }
  ],
  "next_page_token": "string"
}

GenieEvalRunResponse object​

Get a benchmark evaluation run Beta​

Parameters​

Response​

List benchmark evaluation runs Beta​

Parameters​

Response​

Start a benchmark evaluation run Beta​

Parameters​

Request body​

Response​

Get benchmark evaluation result details Beta​

Parameters​

Response​

List results for a benchmark evaluation run Beta​

Parameters​

Response​

GenieEvalRunResponse object

Get a benchmark evaluation run Beta

Parameters

Response

List benchmark evaluation runs Beta

Parameters

Response

Start a benchmark evaluation run Beta

Parameters

Request body

Response

Get benchmark evaluation result details Beta

Parameters

Response

List results for a benchmark evaluation run Beta

Parameters

Response