Eval Run

View as Markdown

GenieEvalRunResponse object

eval_run_idstringBeta: The unique identifier for the evaluation run.
Example: e1ef34712a29169db030324fd0e1df5f

eval_run_statusstringBeta: Current status of the evaluation run.
Values: EVALUATION_STATUS_TYPE_UNSPECIFIED, RUNNING, DONE, NOT_STARTED, EVALUATION_FAILED, EVALUATION_CANCELLED, EVALUATION_TIMEOUT

run_by_userint64Beta: User ID who initiated the evaluation run.

created_timestampint64Beta: Timestamp when the evaluation run was created (milliseconds since epoch).

num_questionsint64Beta: Total number of questions in the evaluation run.

num_correctint64Beta: Number of questions answered correctly.

num_needs_reviewint64Beta: Number of questions that need manual review.

num_doneint64Beta: Number of questions that have been completed.

last_updated_timestampint64Beta: Timestamp when the evaluation run was last updated (milliseconds since epoch).

View as Markdown

The GenieEvalRunResponse object

{
  "eval_run_id": "e1ef34712a29169db030324fd0e1df5f",
  "eval_run_status": "string",
  "run_by_user": 0,
  "created_timestamp": 0,
  "num_questions": 0,
  "num_correct": 0,
  "num_needs_review": 0,
  "num_done": 0,
  "last_updated_timestamp": 0
}

Get Beta

GET /api/2.0/genie/spaces/{space_id}/eval-runs/{eval_run_id}

Get evaluation run details.

API scopes: genie

Parameters

space_idstringpath: The ID associated with the Genie space where the evaluation run is located.
Example: e1ef34712a29169db030324fd0e1df5f

eval_run_idstringpath: Example: e1ef34712a29169db030324fd0e1df5f

Response

Returns the GenieEvalRunResponse object.

View as Markdown

Response

{
  "eval_run_id": "e1ef34712a29169db030324fd0e1df5f",
  "eval_run_status": "string",
  "run_by_user": 0,
  "created_timestamp": 0,
  "num_questions": 0,
  "num_correct": 0,
  "num_needs_review": 0,
  "num_done": 0,
  "last_updated_timestamp": 0
}

List Beta

GET /api/2.0/genie/spaces/{space_id}/eval-runs

Lists all evaluation runs in a space.

API scopes: genie

Parameters

space_idstringpath: The ID associated with the Genie space where the evaluation run is located.
Example: e1ef34712a29169db030324fd0e1df5f

page_sizeint32query: Maximum number of evaluation runs to return per page
Default: 20
Constraints: <= 100

page_tokenstringquery: Token to get the next page of results

Response

Returns a list of GenieEvalRunResponse objects.

View as Markdown

Response

{
  "eval_runs": [
    {
      "eval_run_id": "e1ef34712a29169db030324fd0e1df5f",
      "eval_run_status": "string",
      "run_by_user": 0,
      "created_timestamp": 0,
      "num_questions": 0,
      "num_correct": 0,
      "num_needs_review": 0,
      "num_done": 0,
      "last_updated_timestamp": 0
    }
  ],
  "next_page_token": "string"
}

Create Beta

POST /api/2.0/genie/spaces/{space_id}/eval-runs

Create and run evaluations for multiple benchmark questions in a Genie space.

API scopes: genie

Parameters

space_idstringpath: The ID associated with the Genie space where the evaluations will be executed.
Example: e1ef34712a29169db030324fd0e1df5f

Request body

benchmark_question_idsarray of string: List of benchmark question IDs to evaluate. These questions must exist in the specified Genie space. If none are specified, then all benchmark questions are evaluated.

Response

Returns the GenieEvalRunResponse object.

View as Markdown

Request

{
  "benchmark_question_ids": [
    "string"
  ]
}

Response

{
  "eval_run_id": "e1ef34712a29169db030324fd0e1df5f",
  "eval_run_status": "string",
  "run_by_user": 0,
  "created_timestamp": 0,
  "num_questions": 0,
  "num_correct": 0,
  "num_needs_review": 0,
  "num_done": 0,
  "last_updated_timestamp": 0
}

Genie Get Eval Result Details Beta

GET /api/2.0/genie/spaces/{space_id}/eval-runs/{eval_run_id}/results/{result_id}

Get details for evaluation results.

API scopes: genie

Parameters

space_idstringpath: The ID associated with the Genie space where the evaluation run is located.
Example: e1ef34712a29169db030324fd0e1df5f

eval_run_idstringpath: The unique identifier for the evaluation run.
Example: e1ef34712a29169db030324fd0e1df5f

result_idstringpath: The unique identifier for the evaluation result.
Example: e1ef34712a29169db030324fd0e1df5f

Response

result_idstring: The unique identifier for the evaluation result.

space_idstring: The ID of the space the evaluation result belongs to.

benchmark_question_idstring: The ID of the benchmark question that was evaluated.

eval_run_statusstring: Current status of the evaluation run.
Values: EVALUATION_STATUS_TYPE_UNSPECIFIED, RUNNING, DONE, NOT_STARTED, EVALUATION_FAILED, EVALUATION_CANCELLED, EVALUATION_TIMEOUT

assessmentstring: Assessment of the evaluation result: good, bad, or needs review
Values: GENIE_EVAL_ASSESSMENT_UNSPECIFIED, GOOD, BAD, NEEDS_REVIEW

manual_assessmentboolean: Whether this evaluation was manually assessed.

assessment_reasonsarray of string

Reasons for the assessment score.

Assessment reasons describe why a Genie response was scored as BAD.

Deterministic values (compared against the ground truth result):

EMPTY_RESULT: Genie's generated SQL results were empty for this benchmark question.
RESULT_MISSING_ROWS: Genie's generated SQL response is missing rows from the provided ground truth SQL.
RESULT_EXTRA_ROWS: Genie's generated SQL response has more rows than the provided ground truth SQL.
RESULT_MISSING_COLUMNS: Genie's generated SQL response is missing columns from the provided ground truth SQL.
RESULT_EXTRA_COLUMNS: Genie's generated SQL response has more columns than the provided ground truth SQL.
SINGLE_CELL_DIFFERENCE: Single value result was produced but differs from ground truth result.
EMPTY_GOOD_SQL: The benchmark SQL returned an empty result.
COLUMN_TYPE_DIFFERENCE: The values between the results match but the column type is different.

LLM judge ratings explain the factors driving BAD results:

LLM_JUDGE_MISSING_OR_INCORRECT_FILTER: Genie's generated SQL is missing a WHERE clause condition or has incorrect filter logic that excludes/includes wrong data.
LLM_JUDGE_INCOMPLETE_OR_PARTIAL_OUTPUT: Genie's generated SQL returns only some of the requested data or columns, missing parts of what the ground truth SQL returns.
LLM_JUDGE_MISINTERPRETATION_OF_USER_REQUEST: Genie's generated SQL fundamentally misunderstands what the user is asking for, addressing the wrong question or goal.
LLM_JUDGE_INSTRUCTION_COMPLIANCE_OR_MISSING_BUSINESS_LOGIC: Genie's generated SQL fails to apply specified instructions or business logic that should be followed.
LLM_JUDGE_INCORRECT_METRIC_CALCULATION: Genie's generated SQL uses incorrect logic or makes wrong assumptions when calculating metrics.
LLM_JUDGE_INCORRECT_TABLE_OR_FIELD_USAGE: Genie's generated SQL references wrong tables, columns, or uses fields that don't match the ground truth SQL's intent.
LLM_JUDGE_INCORRECT_FUNCTION_USAGE: Genie's generated SQL uses SQL functions incorrectly or inappropriately (wrong parameters, wrong function for the task, etc.).
LLM_JUDGE_MISSING_OR_INCORRECT_JOIN: Genie's generated SQL is missing necessary joins between tables or has incorrect join conditions/types that produce wrong results.
LLM_JUDGE_MISSING_OR_INCORRECT_AGGREGATION: Genie's generated SQL is missing GROUP BY clauses or has incorrect grouping that doesn't match the requested aggregation level.
LLM_JUDGE_FORMATTING_ERROR: Genie's generated SQL output has incorrect formatting, ordering (ORDER BY), or presentation issues that don't match expectations.
LLM_JUDGE_OTHER: LLM judge identified an error that doesn't fall into other categories.

Deprecated LLM judge values (kept for backward compatibility, do not use):

LLM_JUDGE_MISSING_JOIN (deprecated)
LLM_JUDGE_WRONG_FILTER (deprecated)
LLM_JUDGE_WRONG_AGGREGATION (deprecated)
LLM_JUDGE_WRONG_COLUMNS (deprecated)
LLM_JUDGE_SYNTAX_ERROR (deprecated)
LLM_JUDGE_SEMANTIC_ERROR (deprecated)

Values: SCORE_REASON_UNSPECIFIED, EMPTY_RESULT, RESULT_MISSING_ROWS, RESULT_EXTRA_ROWS, RESULT_MISSING_COLUMNS, RESULT_EXTRA_COLUMNS, SINGLE_CELL_DIFFERENCE, EMPTY_GOOD_SQL, COLUMN_TYPE_DIFFERENCE, LLM_JUDGE_MISSING_JOIN, LLM_JUDGE_WRONG_FILTER, LLM_JUDGE_WRONG_AGGREGATION, LLM_JUDGE_WRONG_COLUMNS, LLM_JUDGE_SYNTAX_ERROR, LLM_JUDGE_SEMANTIC_ERROR, LLM_JUDGE_OTHER, LLM_JUDGE_MISSING_OR_INCORRECT_FILTER, LLM_JUDGE_INCOMPLETE_OR_PARTIAL_OUTPUT, LLM_JUDGE_MISINTERPRETATION_OF_USER_REQUEST, LLM_JUDGE_INSTRUCTION_COMPLIANCE_OR_MISSING_BUSINESS_LOGIC, LLM_JUDGE_INCORRECT_METRIC_CALCULATION, LLM_JUDGE_INCORRECT_TABLE_OR_FIELD_USAGE, LLM_JUDGE_INCORRECT_FUNCTION_USAGE, LLM_JUDGE_MISSING_OR_INCORRECT_JOIN, LLM_JUDGE_MISSING_OR_INCORRECT_AGGREGATION, LLM_JUDGE_FORMATTING_ERROR

actual_responsearray of object

The actual response generated by Genie.

Show child attributesHide child attributes

responsestring: The response content (either text or SQL query).

sql_execution_resultobject

SQL Statement Execution response.

Show child attributesHide child attributes

statement_idstring: The statement ID is returned upon successfully submitting a SQL statement, and is a required reference for all subsequent calls.

statusobject

Show child attributesHide child attributes

statestring

Statement execution state:

PENDING: waiting for warehouse
RUNNING: running
SUCCEEDED: execution was successful, result data available for fetch
FAILED: execution failed; reason for failure described in accompanying error message
CANCELED: user canceled; can come from explicit cancel call, or timeout with on_wait_timeout=CANCEL
CLOSED: execution successful, and statement closed; result no longer available for fetch

Values: STATE_UNSPECIFIED, PENDING, RUNNING, SUCCEEDED, FAILED, CANCELED, CLOSED

errorobject

sql_statestring: SQLSTATE error code returned when the statement execution fails. Only populated when the statement status is FAILED.

manifestobject

Show child attributesHide child attributes

formatstring: Values: FORMAT_UNSPECIFIED, JSON_ARRAY, ARROW_STREAM, CSV

schemaobject

total_chunk_countint32: The total number of chunks that the result set has been divided into.

chunksarray of object: Array of result set chunk metadata.

total_row_countint64: The total number of rows in the result set.

total_byte_countint64: The total number of bytes in the result set. This field is not available when using INLINE disposition.

truncatedboolean: Indicates whether the result is truncated due to row_limit or byte_limit.

resultobject

Show child attributesHide child attributes

external_linksarray of object

data_arrayarray of object: The JSON_ARRAY format is an array of arrays of values, where each non-null value is formatted as a string. Null values are encoded as JSON null.

chunk_indexint32: The position within the sequence of result set chunks.

row_offsetint64: The starting row offset within the result set.

row_countint64: The number of rows within the result chunk.

byte_countint64: The number of bytes in the result chunk. This field is not available when using INLINE disposition.

next_chunk_indexint32: When fetching, provides the chunk_index for the next chunk. If absent, indicates there are no more chunks. The next chunk can be fetched with a statementexecution/getstatementresultchunkn request.

next_chunk_internal_linkstring: When fetching, provides a link to fetch the next chunk. If absent, indicates there are no more chunks. This link is an absolute path to be joined with your $DATABRICKS_HOST, and should be treated as an opaque link. This is an alternative to using next_chunk_index.

response_typestring: Type of response
Values: GENIE_EVAL_RESPONSE_TYPE_UNSPECIFIED, TEXT, SQL

expected_responsearray of object

The expected responses from the benchmark.

Show child attributesHide child attributes

responsestring: The response content (either text or SQL query).

sql_execution_resultobject

SQL Statement Execution response.

Show child attributesHide child attributes

statement_idstring: The statement ID is returned upon successfully submitting a SQL statement, and is a required reference for all subsequent calls.

statusobject

Show child attributesHide child attributes

statestring

Statement execution state:

PENDING: waiting for warehouse
RUNNING: running
SUCCEEDED: execution was successful, result data available for fetch
FAILED: execution failed; reason for failure described in accompanying error message
CANCELED: user canceled; can come from explicit cancel call, or timeout with on_wait_timeout=CANCEL
CLOSED: execution successful, and statement closed; result no longer available for fetch

Values: STATE_UNSPECIFIED, PENDING, RUNNING, SUCCEEDED, FAILED, CANCELED, CLOSED

errorobject

sql_statestring: SQLSTATE error code returned when the statement execution fails. Only populated when the statement status is FAILED.

manifestobject

Show child attributesHide child attributes

formatstring: Values: FORMAT_UNSPECIFIED, JSON_ARRAY, ARROW_STREAM, CSV

schemaobject

total_chunk_countint32: The total number of chunks that the result set has been divided into.

chunksarray of object: Array of result set chunk metadata.

total_row_countint64: The total number of rows in the result set.

total_byte_countint64: The total number of bytes in the result set. This field is not available when using INLINE disposition.

truncatedboolean: Indicates whether the result is truncated due to row_limit or byte_limit.

resultobject

Show child attributesHide child attributes

external_linksarray of object

data_arrayarray of object: The JSON_ARRAY format is an array of arrays of values, where each non-null value is formatted as a string. Null values are encoded as JSON null.

chunk_indexint32: The position within the sequence of result set chunks.

row_offsetint64: The starting row offset within the result set.

row_countint64: The number of rows within the result chunk.

byte_countint64: The number of bytes in the result chunk. This field is not available when using INLINE disposition.

next_chunk_indexint32: When fetching, provides the chunk_index for the next chunk. If absent, indicates there are no more chunks. The next chunk can be fetched with a statementexecution/getstatementresultchunkn request.

next_chunk_internal_linkstring: When fetching, provides a link to fetch the next chunk. If absent, indicates there are no more chunks. This link is an absolute path to be joined with your $DATABRICKS_HOST, and should be treated as an opaque link. This is an alternative to using next_chunk_index.

response_typestring: Type of response
Values: GENIE_EVAL_RESPONSE_TYPE_UNSPECIFIED, TEXT, SQL

View as Markdown

Response

{
  "result_id": "string",
  "space_id": "string",
  "benchmark_question_id": "string",
  "eval_run_status": "string",
  "assessment": "string",
  "manual_assessment": true,
  "assessment_reasons": [
    "string"
  ],
  "actual_response": [
    {
      "response": "string",
      "sql_execution_result": {},
      "response_type": "string"
    }
  ],
  "expected_response": [
    {
      "response": "string",
      "sql_execution_result": {},
      "response_type": "string"
    }
  ]
}

Genie List Eval Results Beta

GET /api/2.0/genie/spaces/{space_id}/eval-runs/{eval_run_id}/results

List evaluation results for a specific evaluation run.

API scopes: genie

Parameters

space_idstringpath: The ID associated with the Genie space where the evaluation run is located.
Example: e1ef34712a29169db030324fd0e1df5f

eval_run_idstringpath: The unique identifier for the evaluation run.
Example: e1ef34712a29169db030324fd0e1df5f

page_sizeint32query: Maximum number of eval results to return per page.
Default: 20
Constraints: <= 100

page_tokenstringquery: Opaque token to retrieve the next page of results.

Response

eval_resultsarray of object

List of evaluation results for the specified run.

Show child attributesHide child attributes

result_idstring: Unique identifier for this evaluation result.
Example: e1ef34712a29169db030324fd0e1df5f

space_idstring: The ID of the space the evaluation result belongs to.
Example: e1ef34712a29169db030324fd0e1df5f

benchmark_question_idstring: The ID of the benchmark question that was evaluated.
Example: e1ef34712a29169db030324fd0e1df5f

statusstring: Current status of this evaluation result.
Values: EVALUATION_STATUS_TYPE_UNSPECIFIED, RUNNING, DONE, NOT_STARTED, EVALUATION_FAILED, EVALUATION_CANCELLED, EVALUATION_TIMEOUT

questionstring: Stored snapshot of original benchmark question text.

benchmark_answerstring: Stored snapshot of original benchmark answer text.

created_by_userint64: User ID who created evaluation result.

next_page_tokenstring: The token to use for retrieving the next page of results.

View as Markdown

Response

{
  "eval_results": [
    {
      "result_id": "e1ef34712a29169db030324fd0e1df5f",
      "space_id": "e1ef34712a29169db030324fd0e1df5f",
      "benchmark_question_id": "e1ef34712a29169db030324fd0e1df5f",
      "status": "string",
      "question": "string",
      "benchmark_answer": "string",
      "created_by_user": 0
    }
  ],
  "next_page_token": "string"
}

GenieEvalRunResponse object​

Get Beta​

Parameters​

Response​

List Beta​

Parameters​

Response​

Create Beta​

Parameters​

Request body​

Response​

Genie Get Eval Result Details Beta​

Parameters​

Response​

Genie List Eval Results Beta​

Parameters​

Response​

GenieEvalRunResponse object

Get Beta

Parameters

Response

List Beta

Parameters

Response

Create Beta

Parameters

Request body

Response

Genie Get Eval Result Details Beta

Parameters

Response

Genie List Eval Results Beta

Parameters

Response