Skip to main content

Eval Run

View as Markdown

A benchmark evaluation run. A run asks Genie each selected benchmark question in a Genie space, compares every response with the benchmark SQL, and records one evaluation result per question. Runs execute asynchronously; poll the run until its status is terminal, then list its results. All evaluation run methods require CAN EDIT permission on the space. The public benchmark API currently evaluates chat-mode responses.

GenieEvalRunResponse object​

A benchmark evaluation run. A run asks Genie each selected benchmark question in a Genie space, compares every response with the benchmark SQL, and records one evaluation result per question. Runs execute asynchronously; poll the run until its status is terminal, then list its results. All evaluation run methods require CAN EDIT permission on the space. The public benchmark API currently evaluates chat-mode responses.

eval_run_idstringuuidBeta

The unique identifier for the evaluation run.

Example: e1ef34712a29169db030324fd0e1df5f

eval_run_statusstringBeta

Current status of the run, derived from the statuses of its results. Unless a result has timed out or the run has been cancelled, the run is RUNNING while any question has not finished. It is DONE once every question has finished, whether it was scored or failed. EVALUATION_CANCELLED means the run was cancelled before all questions finished, and EVALUATION_TIMEOUT means the background evaluation job stopped before they did. DONE, EVALUATION_CANCELLED and EVALUATION_TIMEOUT are terminal run statuses. NOT_STARTED and EVALUATION_FAILED appear only on individual results.

Values:

  • RUNNING
  • DONE
  • NOT_STARTED
  • EVALUATION_FAILED
  • EVALUATION_CANCELLED
  • EVALUATION_TIMEOUT
run_by_userint64Beta

User ID who initiated the evaluation run.

created_timestampint64Beta

Timestamp when the evaluation run was created (milliseconds since epoch).

num_questionsint64Beta

Total number of questions in the evaluation run.

num_correctint64Beta

Number of questions whose result is assessed GOOD, including results assessed manually.

num_needs_reviewint64Beta

Number of finished questions whose result is assessed NEEDS_REVIEW and has not been reviewed manually yet.

num_doneint64Beta

Number of questions that have finished, either scored (DONE) or failed (EVALUATION_FAILED). Cancelled and timed-out questions are not counted.

last_updated_timestampint64Beta

Timestamp when the evaluation run was last updated (milliseconds since epoch).

Get a benchmark evaluation run Beta​

GET /api/2.0/genie/spaces/{space_id}/eval-runs/{eval_run_id}

Gets the current status and summary counts of a benchmark evaluation run. Poll this method until eval_run_status is terminal: DONE, EVALUATION_CANCELLED or EVALUATION_TIMEOUT.

API scopes: genie

Parameters​

space_idstringuuidRequiredpath

The ID associated with the Genie space where the evaluation run is located.

Example: e1ef34712a29169db030324fd0e1df5f

eval_run_idstringuuidRequiredpath

The ID of the evaluation run, as returned when the run was created.

Example: e1ef34712a29169db030324fd0e1df5f

Response​

Returns the GenieEvalRunResponse object.

List benchmark evaluation runs Beta​

GET /api/2.0/genie/spaces/{space_id}/eval-runs

Lists the benchmark evaluation runs in a Genie space with the current status and summary counts of each run.

API scopes: genie

Parameters​

space_idstringuuidRequiredpath

The ID associated with the Genie space where the evaluation run is located.

Example: e1ef34712a29169db030324fd0e1df5f

page_sizeint32<= 100query

Maximum number of evaluation runs to return per page

Default: 20

page_tokenstringquery

Token to get the next page of results

Response​

Returns a list of GenieEvalRunResponse objects.

Start a benchmark evaluation run Beta​

POST /api/2.0/genie/spaces/{space_id}/eval-runs

Starts a benchmark evaluation run that asks Genie each selected benchmark question in a Genie space and compares every response with the benchmark SQL. The run executes asynchronously: the response is the new run with its current status. If it is still running, poll with the get method until the status is terminal.

API scopes: genie

Parameters​

space_idstringuuidRequiredpath

The ID associated with the Genie space where the evaluations will be executed.

Example: e1ef34712a29169db030324fd0e1df5f

Request body​

benchmark_question_idsarray of string

List of benchmark question IDs to evaluate. These questions must exist in the specified Genie space. If none are specified, then all benchmark questions are evaluated.

Response​

Returns the GenieEvalRunResponse object.

Get benchmark evaluation result details Beta​

GET /api/2.0/genie/spaces/{space_id}/eval-runs/{eval_run_id}/results/{result_id}

Gets the full result for one benchmark question in an evaluation run: its assessment and the reasons for it, Genie's response with the SQL execution result, and the expected response from the benchmark. The result must belong to the given run.

API scopes: genie

Parameters​

space_idstringuuidRequiredpath

The ID associated with the Genie space where the evaluation run is located.

Example: e1ef34712a29169db030324fd0e1df5f

eval_run_idstringuuidRequiredpath

The unique identifier for the evaluation run.

Example: e1ef34712a29169db030324fd0e1df5f

result_idstringuuidRequiredpath

The unique identifier for the evaluation result.

Example: e1ef34712a29169db030324fd0e1df5f

Response​

result_idstringRequired

The unique identifier for the evaluation result.

space_idstringRequired

The ID of the space the evaluation result belongs to.

benchmark_question_idstringRequired

The ID of the benchmark question that was evaluated.

eval_run_statusstringOutput only

Current status of this evaluation result, not of the whole run. Once the status is DONE, the response and automatic assessment are complete. A user can still change the assessment manually.

Values:

  • RUNNING
  • DONE
  • NOT_STARTED
  • EVALUATION_FAILED
  • EVALUATION_CANCELLED
  • EVALUATION_TIMEOUT
assessmentstringOutput only

Outcome of comparing Genie's response with the benchmark. GOOD means the results matched, BAD means they differed, and NEEDS_REVIEW means the comparison was inconclusive and a person should review the result. The value is only meaningful once the result status is DONE.

Values:

  • GOOD
  • BAD
  • NEEDS_REVIEW
manual_assessmentbooleanOutput only

Whether a user set the assessment manually after reviewing the result. A manual assessment replaces the automatic one and clears assessment_reasons.

assessment_reasonsarray of stringOutput only

Reasons for the assessment score.

Assessment reasons describe why a Genie response was scored as BAD.

Deterministic values (compared against the ground truth result):

  • EMPTY_RESULT: Genie's generated SQL results were empty for this benchmark question.
  • RESULT_MISSING_ROWS: Genie's generated SQL response is missing rows from the provided ground truth SQL.
  • RESULT_EXTRA_ROWS: Genie's generated SQL response has more rows than the provided ground truth SQL.
  • RESULT_MISSING_COLUMNS: Genie's generated SQL response is missing columns from the provided ground truth SQL.
  • RESULT_EXTRA_COLUMNS: Genie's generated SQL response has more columns than the provided ground truth SQL.
  • SINGLE_CELL_DIFFERENCE: Single value result was produced but differs from ground truth result.
  • EMPTY_GOOD_SQL: The benchmark SQL returned an empty result.
  • COLUMN_TYPE_DIFFERENCE: The values between the results match but the column type is different.

LLM judge ratings explain the factors driving BAD results:

  • LLM_JUDGE_MISSING_OR_INCORRECT_FILTER: Genie's generated SQL is missing a WHERE clause condition or has incorrect filter logic that excludes/includes wrong data.
  • LLM_JUDGE_INCOMPLETE_OR_PARTIAL_OUTPUT: Genie's generated SQL returns only some of the requested data or columns, missing parts of what the ground truth SQL returns.
  • LLM_JUDGE_MISINTERPRETATION_OF_USER_REQUEST: Genie's generated SQL fundamentally misunderstands what the user is asking for, addressing the wrong question or goal.
  • LLM_JUDGE_INSTRUCTION_COMPLIANCE_OR_MISSING_BUSINESS_LOGIC: Genie's generated SQL fails to apply specified instructions or business logic that should be followed.
  • LLM_JUDGE_INCORRECT_METRIC_CALCULATION: Genie's generated SQL uses incorrect logic or makes wrong assumptions when calculating metrics.
  • LLM_JUDGE_INCORRECT_TABLE_OR_FIELD_USAGE: Genie's generated SQL references wrong tables, columns, or uses fields that don't match the ground truth SQL's intent.
  • LLM_JUDGE_INCORRECT_FUNCTION_USAGE: Genie's generated SQL uses SQL functions incorrectly or inappropriately (wrong parameters, wrong function for the task, etc.).
  • LLM_JUDGE_MISSING_OR_INCORRECT_JOIN: Genie's generated SQL is missing necessary joins between tables or has incorrect join conditions/types that produce wrong results.
  • LLM_JUDGE_MISSING_OR_INCORRECT_AGGREGATION: Genie's generated SQL is missing GROUP BY clauses or has incorrect grouping that doesn't match the requested aggregation level.
  • LLM_JUDGE_FORMATTING_ERROR: Genie's generated SQL output has incorrect formatting, ordering (ORDER BY), or presentation issues that don't match expectations.
  • LLM_JUDGE_OTHER: LLM judge identified an error that doesn't fall into other categories.

Deprecated LLM judge values (kept for backward compatibility, do not use):

  • LLM_JUDGE_MISSING_JOIN (deprecated)
  • LLM_JUDGE_WRONG_FILTER (deprecated)
  • LLM_JUDGE_WRONG_AGGREGATION (deprecated)
  • LLM_JUDGE_WRONG_COLUMNS (deprecated)
  • LLM_JUDGE_SYNTAX_ERROR (deprecated)
  • LLM_JUDGE_SEMANTIC_ERROR (deprecated)

Values:

Show 25 possible values
  • EMPTY_RESULT
  • RESULT_MISSING_ROWS
  • RESULT_EXTRA_ROWS
  • RESULT_MISSING_COLUMNS
  • RESULT_EXTRA_COLUMNS
  • SINGLE_CELL_DIFFERENCE
  • EMPTY_GOOD_SQL
  • COLUMN_TYPE_DIFFERENCE
  • LLM_JUDGE_MISSING_JOIN
  • LLM_JUDGE_WRONG_FILTER
  • LLM_JUDGE_WRONG_AGGREGATION
  • LLM_JUDGE_WRONG_COLUMNS
  • LLM_JUDGE_SYNTAX_ERROR
  • LLM_JUDGE_SEMANTIC_ERROR
  • LLM_JUDGE_OTHER
  • LLM_JUDGE_MISSING_OR_INCORRECT_FILTER
  • LLM_JUDGE_INCOMPLETE_OR_PARTIAL_OUTPUT
  • LLM_JUDGE_MISINTERPRETATION_OF_USER_REQUEST
  • LLM_JUDGE_INSTRUCTION_COMPLIANCE_OR_MISSING_BUSINESS_LOGIC
  • LLM_JUDGE_INCORRECT_METRIC_CALCULATION
  • LLM_JUDGE_INCORRECT_TABLE_OR_FIELD_USAGE
  • LLM_JUDGE_INCORRECT_FUNCTION_USAGE
  • LLM_JUDGE_MISSING_OR_INCORRECT_JOIN
  • LLM_JUDGE_MISSING_OR_INCORRECT_AGGREGATION
  • LLM_JUDGE_FORMATTING_ERROR
actual_responsearray of objectOutput only

Genie's response to the benchmark question. A SQL entry carries the generated statement and its execution result, and a TEXT entry carries Genie's text answer. The list is empty when Genie produced no response.

Show child attributesHide child attributes
responsestringOutput only

The content of this entry is the SQL statement when response_type is SQL, or Genie's text answer when it is TEXT.

sql_execution_resultobjectOutput only

Result of executing the SQL statement, only set when response_type is SQL. The value has the structure of a Statement Execution API response.

Show child attributesHide child attributes
statement_idstring

The statement ID is returned upon successfully submitting a SQL statement, and is a required reference for all subsequent calls.

statusobject

The execution state of the statement and, if it failed, the error details.

Show child attributesHide child attributes
statestring

Statement execution state:

  • PENDING: waiting for warehouse
  • RUNNING: running
  • SUCCEEDED: execution was successful, result data available for fetch
  • FAILED: execution failed; reason for failure described in accompanying error message
  • CANCELED: user canceled; can come from explicit cancel call, or timeout with on_wait_timeout=CANCEL
  • CLOSED: execution successful, and statement closed; result no longer available for fetch

Values:

  • PENDING
  • RUNNING
  • SUCCEEDED
  • FAILED
  • CANCELED
  • CLOSED
errorobject

The error that caused the statement to fail. Only populated when state is FAILED.

Show child attributesHide child attributes
error_codestring

Error codes returned by Databricks APIs to indicate specific failure conditions.

Values:

Show 81 possible values
  • UNKNOWN
  • INTERNAL_ERROR
  • TEMPORARILY_UNAVAILABLE
  • IO_ERROR
  • BAD_REQUEST
  • SERVICE_UNDER_MAINTENANCE
  • WORKSPACE_TEMPORARILY_UNAVAILABLE
  • DEADLINE_EXCEEDED
  • CANCELLED
  • RESOURCE_EXHAUSTED
  • ABORTED
  • NOT_FOUND
  • ALREADY_EXISTS
  • UNAUTHENTICATED
  • UNAVAILABLE
  • INVALID_PARAMETER_VALUE
  • ENDPOINT_NOT_FOUND
  • MALFORMED_REQUEST
  • INVALID_STATE
  • PERMISSION_DENIED
  • FEATURE_DISABLED
  • CUSTOMER_UNAUTHORIZED
  • REQUEST_LIMIT_EXCEEDED
  • RESOURCE_CONFLICT
  • UNPARSEABLE_HTTP_ERROR
  • NOT_IMPLEMENTED
  • DATA_LOSS
  • INVALID_STATE_TRANSITION
  • COULD_NOT_ACQUIRE_LOCK
  • RESOURCE_ALREADY_EXISTS
  • RESOURCE_DOES_NOT_EXIST
  • QUOTA_EXCEEDED
  • MAX_BLOCK_SIZE_EXCEEDED
  • MAX_READ_SIZE_EXCEEDED
  • PARTIAL_DELETE
  • MAX_LIST_SIZE_EXCEEDED
  • DRY_RUN_FAILED
  • RESOURCE_LIMIT_EXCEEDED
  • DIRECTORY_NOT_EMPTY
  • DIRECTORY_PROTECTED
  • MAX_NOTEBOOK_SIZE_EXCEEDED
  • MAX_CHILD_NODE_SIZE_EXCEEDED
  • SEARCH_QUERY_TOO_LONG
  • SEARCH_QUERY_TOO_SHORT
  • MANAGED_RESOURCE_GROUP_DOES_NOT_EXIST
  • PERMISSION_NOT_PROPAGATED
  • DEPLOYMENT_TIMEOUT
  • GIT_CONFLICT
  • GIT_UNKNOWN_REF
  • GIT_SENSITIVE_TOKEN_DETECTED
  • GIT_URL_NOT_ON_ALLOW_LIST
  • GIT_REMOTE_ERROR
  • PROJECTS_OPERATION_TIMEOUT
  • IPYNB_FILE_IN_REPO
  • INSECURE_PARTNER_RESPONSE
  • MALFORMED_PARTNER_RESPONSE
  • METASTORE_DOES_NOT_EXIST
  • DAC_DOES_NOT_EXIST
  • CATALOG_DOES_NOT_EXIST
  • SCHEMA_DOES_NOT_EXIST
  • TABLE_DOES_NOT_EXIST
  • SHARE_DOES_NOT_EXIST
  • RECIPIENT_DOES_NOT_EXIST
  • STORAGE_CREDENTIAL_DOES_NOT_EXIST
  • EXTERNAL_LOCATION_DOES_NOT_EXIST
  • PRINCIPAL_DOES_NOT_EXIST
  • PROVIDER_DOES_NOT_EXIST
  • METASTORE_ALREADY_EXISTS
  • DAC_ALREADY_EXISTS
  • CATALOG_ALREADY_EXISTS
  • SCHEMA_ALREADY_EXISTS
  • TABLE_ALREADY_EXISTS
  • SHARE_ALREADY_EXISTS
  • RECIPIENT_ALREADY_EXISTS
  • STORAGE_CREDENTIAL_ALREADY_EXISTS
  • EXTERNAL_LOCATION_ALREADY_EXISTS
  • PROVIDER_ALREADY_EXISTS
  • CATALOG_NOT_EMPTY
  • SCHEMA_NOT_EMPTY
  • METASTORE_NOT_EMPTY
  • PROVIDER_SHARE_NOT_ACCESSIBLE
messagestring
stack_tracestring
sql_statestring

SQLSTATE error code returned when the statement execution fails. Only populated when the statement status is FAILED.

manifestobject

The schema and metadata of the result set. Present when status.state is SUCCEEDED.

Show child attributesHide child attributes
formatstring

The format of the result data, as requested with format.

Values:

  • JSON_ARRAY
  • ARROW_STREAM
  • CSV
schemaobject

The schema of the result set: the ordered list of column descriptions.

Show child attributesHide child attributes
column_countint32

The number of columns in the result set.

columnsarray of object

The column descriptions, in result set order.

Show child attributesHide child attributes
namestring

Name of Column.

type_textstring

Full data type specification as SQL/catalogString text.

type_namestring

The name of the column's data type, for example STRING, DECIMAL or STRUCT. The complete type, including parameters such as the precision of a decimal or the element type of an array, is in type_text.

Values:

Show 24 possible values
  • BOOLEAN
  • BYTE
  • SHORT
  • INT
  • LONG
  • FLOAT
  • DOUBLE
  • DATE
  • TIMESTAMP
  • STRING
  • BINARY
  • DECIMAL
  • INTERVAL
  • ARRAY
  • STRUCT
  • MAP
  • CHAR
  • NULL
  • USER_DEFINED_TYPE
  • TIMESTAMP_NTZ
  • VARIANT
  • GEOMETRY
  • GEOGRAPHY
  • TABLE_TYPE
positionint32

Ordinal position of column (starting at position 0).

type_precisionint32

Digits of precision; required for DecimalTypes.

type_scaleint32

Digits to right of decimal; Required for DecimalTypes.

type_interval_typestring

Format of IntervalType.

type_jsonstring

Full data type specification, JSON-serialized.

commentstring

User-provided free-form text description.

nullableboolean

Whether field may be Null (default: true).

Default: true

partition_indexint32

Partition index for column.

maskobject

The column mask function applied to this column, if any. A column mask is a SQL user-defined function that transforms the values a query returns for the column. See https://docs.databricks.com/aws/en/tables/row-and-column-filters for details.

Show child attributesHide child attributes
function_namestring

The full name of the column mask SQL UDF.

using_column_namesarray of string

The list of additional table columns to be passed as input to the column mask function. The first arg of the mask function should be of the type of the column being masked and the types of the rest of the args should match the types of columns in 'using_column_names'.

using_argumentsarray of objectDeprecated

The list of table columns or literals to be passed as additional arguments to a column mask function, carrying the type (column reference vs constant literal) of each argument. Deprecated: use using_column_names instead.

Show child attributesHide child attributes
columnstring

A column reference.

constantstring

A constant literal.

total_chunk_countint32

The total number of chunks that the result set has been divided into.

chunksarray of object

Array of result set chunk metadata.

Show child attributesHide child attributes
chunk_indexint32

The position within the sequence of result set chunks.

row_offsetint64

The starting row offset within the result set.

row_countint64

The number of rows within the result chunk.

byte_countint64

The number of bytes in the result chunk. This field is not available when using INLINE disposition.

next_chunk_indexint32

When fetching, provides the chunk_index for the next chunk. If absent, indicates there are no more chunks. The next chunk can be fetched with a statementexecution/getStatementResultChunkN request.

total_row_countint64

The total number of rows in the result set.

total_byte_countint64

The total number of bytes in the result set. This field is not available when using INLINE disposition.

truncatedboolean

Indicates whether the result is truncated due to row_limit or byte_limit.

resultobject

Contains the result data of a single chunk when using INLINE disposition. When using EXTERNAL_LINKS disposition, the array external_links is used instead to provide short-lived cloud-storage URLs to the result data in cloud storage. Exactly one of these alternatives is used. Calls to getResultData return the link for the requested chunk; executeStatement and getStatementResult responses can contain links for multiple chunks.

AWS

On AWS, external links are presigned URLs.

Azure

On Azure, external links are SAS URLs.

GCP

On GCP, external links are signed URLs.

Show child attributesHide child attributes
data_arrayarray of array of object

The JSON_ARRAY format is an array of arrays of values, where each non-null value is formatted as a string. Null values are encoded as JSON null.

chunk_indexint32

The position within the sequence of result set chunks.

row_offsetint64

The starting row offset within the result set.

row_countint64

The number of rows within the result chunk.

byte_countint64

The number of bytes in the result chunk. This field is not available when using INLINE disposition.

next_chunk_indexint32

When fetching, provides the chunk_index for the next chunk. If absent, indicates there are no more chunks. The next chunk can be fetched with a statementexecution/getStatementResultChunkN request.

response_typestringOutput only

Whether this entry is a SQL statement or a text answer.

Values:

  • TEXT
  • SQL
expected_responsearray of objectOutput only

The expected answer from the benchmark question: one SQL entry with the benchmark SQL statement and its execution result.

Show child attributesHide child attributes
responsestringOutput only

The content of this entry is the SQL statement when response_type is SQL, or Genie's text answer when it is TEXT.

sql_execution_resultobjectOutput only

Result of executing the SQL statement, only set when response_type is SQL. The value has the structure of a Statement Execution API response.

Show child attributesHide child attributes
statement_idstring

The statement ID is returned upon successfully submitting a SQL statement, and is a required reference for all subsequent calls.

statusobject

The execution state of the statement and, if it failed, the error details.

Show child attributesHide child attributes
statestring

Statement execution state:

  • PENDING: waiting for warehouse
  • RUNNING: running
  • SUCCEEDED: execution was successful, result data available for fetch
  • FAILED: execution failed; reason for failure described in accompanying error message
  • CANCELED: user canceled; can come from explicit cancel call, or timeout with on_wait_timeout=CANCEL
  • CLOSED: execution successful, and statement closed; result no longer available for fetch

Values:

  • PENDING
  • RUNNING
  • SUCCEEDED
  • FAILED
  • CANCELED
  • CLOSED
errorobject

The error that caused the statement to fail. Only populated when state is FAILED.

Show child attributesHide child attributes
error_codestring

Error codes returned by Databricks APIs to indicate specific failure conditions.

Values:

Show 81 possible values
  • UNKNOWN
  • INTERNAL_ERROR
  • TEMPORARILY_UNAVAILABLE
  • IO_ERROR
  • BAD_REQUEST
  • SERVICE_UNDER_MAINTENANCE
  • WORKSPACE_TEMPORARILY_UNAVAILABLE
  • DEADLINE_EXCEEDED
  • CANCELLED
  • RESOURCE_EXHAUSTED
  • ABORTED
  • NOT_FOUND
  • ALREADY_EXISTS
  • UNAUTHENTICATED
  • UNAVAILABLE
  • INVALID_PARAMETER_VALUE
  • ENDPOINT_NOT_FOUND
  • MALFORMED_REQUEST
  • INVALID_STATE
  • PERMISSION_DENIED
  • FEATURE_DISABLED
  • CUSTOMER_UNAUTHORIZED
  • REQUEST_LIMIT_EXCEEDED
  • RESOURCE_CONFLICT
  • UNPARSEABLE_HTTP_ERROR
  • NOT_IMPLEMENTED
  • DATA_LOSS
  • INVALID_STATE_TRANSITION
  • COULD_NOT_ACQUIRE_LOCK
  • RESOURCE_ALREADY_EXISTS
  • RESOURCE_DOES_NOT_EXIST
  • QUOTA_EXCEEDED
  • MAX_BLOCK_SIZE_EXCEEDED
  • MAX_READ_SIZE_EXCEEDED
  • PARTIAL_DELETE
  • MAX_LIST_SIZE_EXCEEDED
  • DRY_RUN_FAILED
  • RESOURCE_LIMIT_EXCEEDED
  • DIRECTORY_NOT_EMPTY
  • DIRECTORY_PROTECTED
  • MAX_NOTEBOOK_SIZE_EXCEEDED
  • MAX_CHILD_NODE_SIZE_EXCEEDED
  • SEARCH_QUERY_TOO_LONG
  • SEARCH_QUERY_TOO_SHORT
  • MANAGED_RESOURCE_GROUP_DOES_NOT_EXIST
  • PERMISSION_NOT_PROPAGATED
  • DEPLOYMENT_TIMEOUT
  • GIT_CONFLICT
  • GIT_UNKNOWN_REF
  • GIT_SENSITIVE_TOKEN_DETECTED
  • GIT_URL_NOT_ON_ALLOW_LIST
  • GIT_REMOTE_ERROR
  • PROJECTS_OPERATION_TIMEOUT
  • IPYNB_FILE_IN_REPO
  • INSECURE_PARTNER_RESPONSE
  • MALFORMED_PARTNER_RESPONSE
  • METASTORE_DOES_NOT_EXIST
  • DAC_DOES_NOT_EXIST
  • CATALOG_DOES_NOT_EXIST
  • SCHEMA_DOES_NOT_EXIST
  • TABLE_DOES_NOT_EXIST
  • SHARE_DOES_NOT_EXIST
  • RECIPIENT_DOES_NOT_EXIST
  • STORAGE_CREDENTIAL_DOES_NOT_EXIST
  • EXTERNAL_LOCATION_DOES_NOT_EXIST
  • PRINCIPAL_DOES_NOT_EXIST
  • PROVIDER_DOES_NOT_EXIST
  • METASTORE_ALREADY_EXISTS
  • DAC_ALREADY_EXISTS
  • CATALOG_ALREADY_EXISTS
  • SCHEMA_ALREADY_EXISTS
  • TABLE_ALREADY_EXISTS
  • SHARE_ALREADY_EXISTS
  • RECIPIENT_ALREADY_EXISTS
  • STORAGE_CREDENTIAL_ALREADY_EXISTS
  • EXTERNAL_LOCATION_ALREADY_EXISTS
  • PROVIDER_ALREADY_EXISTS
  • CATALOG_NOT_EMPTY
  • SCHEMA_NOT_EMPTY
  • METASTORE_NOT_EMPTY
  • PROVIDER_SHARE_NOT_ACCESSIBLE
messagestring
stack_tracestring
sql_statestring

SQLSTATE error code returned when the statement execution fails. Only populated when the statement status is FAILED.

manifestobject

The schema and metadata of the result set. Present when status.state is SUCCEEDED.

Show child attributesHide child attributes
formatstring

The format of the result data, as requested with format.

Values:

  • JSON_ARRAY
  • ARROW_STREAM
  • CSV
schemaobject

The schema of the result set: the ordered list of column descriptions.

Show child attributesHide child attributes
column_countint32

The number of columns in the result set.

columnsarray of object

The column descriptions, in result set order.

Show child attributesHide child attributes
namestring

Name of Column.

type_textstring

Full data type specification as SQL/catalogString text.

type_namestring

The name of the column's data type, for example STRING, DECIMAL or STRUCT. The complete type, including parameters such as the precision of a decimal or the element type of an array, is in type_text.

Values:

Show 24 possible values
  • BOOLEAN
  • BYTE
  • SHORT
  • INT
  • LONG
  • FLOAT
  • DOUBLE
  • DATE
  • TIMESTAMP
  • STRING
  • BINARY
  • DECIMAL
  • INTERVAL
  • ARRAY
  • STRUCT
  • MAP
  • CHAR
  • NULL
  • USER_DEFINED_TYPE
  • TIMESTAMP_NTZ
  • VARIANT
  • GEOMETRY
  • GEOGRAPHY
  • TABLE_TYPE
positionint32

Ordinal position of column (starting at position 0).

type_precisionint32

Digits of precision; required for DecimalTypes.

type_scaleint32

Digits to right of decimal; Required for DecimalTypes.

type_interval_typestring

Format of IntervalType.

type_jsonstring

Full data type specification, JSON-serialized.

commentstring

User-provided free-form text description.

nullableboolean

Whether field may be Null (default: true).

Default: true

partition_indexint32

Partition index for column.

maskobject

The column mask function applied to this column, if any. A column mask is a SQL user-defined function that transforms the values a query returns for the column. See https://docs.databricks.com/aws/en/tables/row-and-column-filters for details.

Show child attributesHide child attributes
function_namestring

The full name of the column mask SQL UDF.

using_column_namesarray of string

The list of additional table columns to be passed as input to the column mask function. The first arg of the mask function should be of the type of the column being masked and the types of the rest of the args should match the types of columns in 'using_column_names'.

using_argumentsarray of objectDeprecated

The list of table columns or literals to be passed as additional arguments to a column mask function, carrying the type (column reference vs constant literal) of each argument. Deprecated: use using_column_names instead.

Show child attributesHide child attributes
columnstring

A column reference.

constantstring

A constant literal.

total_chunk_countint32

The total number of chunks that the result set has been divided into.

chunksarray of object

Array of result set chunk metadata.

Show child attributesHide child attributes
chunk_indexint32

The position within the sequence of result set chunks.

row_offsetint64

The starting row offset within the result set.

row_countint64

The number of rows within the result chunk.

byte_countint64

The number of bytes in the result chunk. This field is not available when using INLINE disposition.

next_chunk_indexint32

When fetching, provides the chunk_index for the next chunk. If absent, indicates there are no more chunks. The next chunk can be fetched with a statementexecution/getStatementResultChunkN request.

total_row_countint64

The total number of rows in the result set.

total_byte_countint64

The total number of bytes in the result set. This field is not available when using INLINE disposition.

truncatedboolean

Indicates whether the result is truncated due to row_limit or byte_limit.

resultobject

Contains the result data of a single chunk when using INLINE disposition. When using EXTERNAL_LINKS disposition, the array external_links is used instead to provide short-lived cloud-storage URLs to the result data in cloud storage. Exactly one of these alternatives is used. Calls to getResultData return the link for the requested chunk; executeStatement and getStatementResult responses can contain links for multiple chunks.

AWS

On AWS, external links are presigned URLs.

Azure

On Azure, external links are SAS URLs.

GCP

On GCP, external links are signed URLs.

Show child attributesHide child attributes
data_arrayarray of array of object

The JSON_ARRAY format is an array of arrays of values, where each non-null value is formatted as a string. Null values are encoded as JSON null.

chunk_indexint32

The position within the sequence of result set chunks.

row_offsetint64

The starting row offset within the result set.

row_countint64

The number of rows within the result chunk.

byte_countint64

The number of bytes in the result chunk. This field is not available when using INLINE disposition.

next_chunk_indexint32

When fetching, provides the chunk_index for the next chunk. If absent, indicates there are no more chunks. The next chunk can be fetched with a statementexecution/getStatementResultChunkN request.

response_typestringOutput only

Whether this entry is a SQL statement or a text answer.

Values:

  • TEXT
  • SQL

List results for a benchmark evaluation run Beta​

GET /api/2.0/genie/spaces/{space_id}/eval-runs/{eval_run_id}/results

Lists the results of a benchmark evaluation run, one per benchmark question. Each entry carries the result status and a snapshot of the question and benchmark answer; the assessment and Genie's response are available through the result details method.

API scopes: genie

Parameters​

space_idstringuuidRequiredpath

The ID associated with the Genie space where the evaluation run is located.

Example: e1ef34712a29169db030324fd0e1df5f

eval_run_idstringuuidRequiredpath

The unique identifier for the evaluation run.

Example: e1ef34712a29169db030324fd0e1df5f

page_sizeint32<= 100query

Maximum number of eval results to return per page.

Default: 20

page_tokenstringquery

Opaque token to retrieve the next page of results.

Response​

eval_resultsarray of object

List of evaluation results for the specified run.

Show child attributesHide child attributes
result_idstringuuidRequired

Unique identifier for this evaluation result.

Example: e1ef34712a29169db030324fd0e1df5f

space_idstringuuidRequired

The ID of the space the evaluation result belongs to.

Example: e1ef34712a29169db030324fd0e1df5f

benchmark_question_idstringuuidRequired

The ID of the benchmark question that was evaluated.

Example: e1ef34712a29169db030324fd0e1df5f

statusstringOutput only

Current status of this result. NOT_STARTED and RUNNING mean the question has not finished. DONE means Genie answered and the response was assessed. EVALUATION_FAILED means Genie could not answer the question or the benchmark SQL could not run. EVALUATION_CANCELLED and EVALUATION_TIMEOUT mean the run was cancelled or its background job stopped before this question finished.

Values:

  • RUNNING
  • DONE
  • NOT_STARTED
  • EVALUATION_FAILED
  • EVALUATION_CANCELLED
  • EVALUATION_TIMEOUT
questionstringOutput only

Text of the benchmark question as it was when the run started. Later edits to the question do not change it.

benchmark_answerstringOutput only

Benchmark SQL statement that defines the expected answer, as it was when the run started.

created_by_userint64Output only

ID of the user who started the evaluation run.

next_page_tokenstring

The token to use for retrieving the next page of results.