Skip to main content

Vector Index

View as Markdown

VectorIndex object

namestring

Name of the index

endpoint_namestring

Name of the endpoint associated with the index

primary_keystring

Primary key of the index

index_typestring

Values: DELTA_SYNC, DIRECT_ACCESS

direct_access_index_specobject
Show child attributesHide child attributes
embedding_vector_columnsarray of object

The columns that contain the embedding vectors. The format should be array[double].

Show child attributesHide child attributes
namestring

Name of the column

embedding_dimensionint32

Dimension of the embedding vector

schema_jsonstring

The schema of the index in JSON format. Supported types are integer, long, float, double, boolean, string, date, timestamp. Supported types for vector column: array<float>, array<double>,`.

embedding_source_columnsarray of object

The columns that contain the embedding source. The format should be array[double].

Show child attributesHide child attributes
namestring

Name of the column

embedding_model_endpoint_namestring

Name of the embedding model endpoint, used by default for both ingestion and querying.

model_endpoint_name_for_querystring

Name of the embedding model endpoint which, if specified, is used for querying (not ingestion).

delta_sync_index_specobject
Show child attributesHide child attributes
source_tablestring

The name of the source table.

embedding_source_columnsarray of object

The columns that contain the embedding source.

Show child attributesHide child attributes
namestring

Name of the column

embedding_model_endpoint_namestring

Name of the embedding model endpoint, used by default for both ingestion and querying.

model_endpoint_name_for_querystring

Name of the embedding model endpoint which, if specified, is used for querying (not ingestion).

embedding_vector_columnsarray of object

The columns that contain the embedding vectors.

Show child attributesHide child attributes
namestring

Name of the column

embedding_dimensionint32

Dimension of the embedding vector

pipeline_typestring

Pipeline execution mode.

  • TRIGGERED: If the pipeline uses the triggered execution mode, the system stops processing after successfully refreshing the source table in the pipeline once, ensuring the table is updated based on the data available when the update started.
  • CONTINUOUS: If the pipeline uses continuous execution, the pipeline processes new data as it arrives in the source table to keep vector index fresh.

Values: TRIGGERED, CONTINUOUS

pipeline_idstring

The ID of the pipeline that is used to sync the index.

embedding_writeback_tablestring

[Optional] Name of the Delta table to sync the vector index contents and computed embeddings to.

columns_to_syncarray of string

[Optional] Select the columns to sync with the vector index. If you leave this field blank, all columns from the source table are synced with the index. The primary key column and embedding source column or embedding vector column are always synced.

columns_to_indexarray of string

[Optional] Alias for columns_to_sync. Select the columns to include in the vector index. If you leave this field blank, all columns from the source table are included. The primary key column and embedding source column or embedding vector column are always included. Only one of columns_to_sync or columns_to_index may be specified.

statusobject
Show child attributesHide child attributes
messagestring

Message associated with the index status

indexed_row_countint64

Number of rows indexed

readyboolean

Whether the index is ready for search

index_urlstring

Index API Url to be used to perform operations on the index

creatorstring

The user who created the index.

index_subtypestring

The subtype of the index.

Values: VECTOR, FULL_TEXT, HYBRID

Get

GET /api/2.0/vector-search/indexes/{name}

Get an index.

API scopes: vector-search

Parameters

namestringpath

Name of the index

ensure_reranker_compatiblebooleanquery

If true, the URL returned for the index is guaranteed to be compatible with the reranker. Currently this means we return the CP URL regardless of how the index is being accessed. If not set or set to false, the URL may still be compatible with the reranker depending on what URL we return.

Response

Returns the VectorIndex object.

List

GET /api/2.0/vector-search/indexes

List all indexes in the given endpoint.

API scopes: vector-search

Parameters

endpoint_namestringquery

Name of the endpoint

page_tokenstringquery

Token for pagination

Response

vector_indexesarray of object
Show child attributesHide child attributes
namestring

Name of the index

endpoint_namestring

Name of the endpoint associated with the index

primary_keystring

Primary key of the index

index_typestring

Values: DELTA_SYNC, DIRECT_ACCESS

direct_access_index_specobjectRequired
Show child attributesHide child attributes
embedding_vector_columnsarray of object

The columns that contain the embedding vectors. The format should be array[double].

Show child attributesHide child attributes
namestring

Name of the column

embedding_dimensionint32

Dimension of the embedding vector

schema_jsonstring

The schema of the index in JSON format. Supported types are integer, long, float, double, boolean, string, date, timestamp. Supported types for vector column: array<float>, array<double>,`.

embedding_source_columnsarray of object

The columns that contain the embedding source. The format should be array[double].

Show child attributesHide child attributes
namestring

Name of the column

embedding_model_endpoint_namestringRequired

Name of the embedding model endpoint, used by default for both ingestion and querying.

model_endpoint_name_for_querystring

Name of the embedding model endpoint which, if specified, is used for querying (not ingestion).

delta_sync_index_specobjectRequired
Show child attributesHide child attributes
source_tablestring

The name of the source table.

embedding_source_columnsarray of object

The columns that contain the embedding source.

Show child attributesHide child attributes
namestring

Name of the column

embedding_model_endpoint_namestringRequired

Name of the embedding model endpoint, used by default for both ingestion and querying.

model_endpoint_name_for_querystring

Name of the embedding model endpoint which, if specified, is used for querying (not ingestion).

embedding_vector_columnsarray of object

The columns that contain the embedding vectors.

Show child attributesHide child attributes
namestring

Name of the column

embedding_dimensionint32

Dimension of the embedding vector

pipeline_typestring

Pipeline execution mode.

  • TRIGGERED: If the pipeline uses the triggered execution mode, the system stops processing after successfully refreshing the source table in the pipeline once, ensuring the table is updated based on the data available when the update started.
  • CONTINUOUS: If the pipeline uses continuous execution, the pipeline processes new data as it arrives in the source table to keep vector index fresh.

Values: TRIGGERED, CONTINUOUS

pipeline_idstring

The ID of the pipeline that is used to sync the index.

embedding_writeback_tablestring

[Optional] Name of the Delta table to sync the vector index contents and computed embeddings to.

columns_to_syncarray of string

[Optional] Select the columns to sync with the vector index. If you leave this field blank, all columns from the source table are synced with the index. The primary key column and embedding source column or embedding vector column are always synced.

columns_to_indexarray of string

[Optional] Alias for columns_to_sync. Select the columns to include in the vector index. If you leave this field blank, all columns from the source table are included. The primary key column and embedding source column or embedding vector column are always included. Only one of columns_to_sync or columns_to_index may be specified.

statusobject
Show child attributesHide child attributes
messagestring

Message associated with the index status

indexed_row_countint64

Number of rows indexed

readyboolean

Whether the index is ready for search

index_urlstring

Index API Url to be used to perform operations on the index

creatorstring

The user who created the index.

index_subtypestring

The subtype of the index.

Values: VECTOR, FULL_TEXT, HYBRID

next_page_tokenstring

A token that can be used to get the next page of results. If not present, there are no more results to show.

Create

POST /api/2.0/vector-search/indexes

Create a new index.

API scopes: vector-search

Request body

namestring

Name of the index

endpoint_namestring

Name of the endpoint to be used for serving the index

primary_keystring

Primary key of the index

index_typestring

Values: DELTA_SYNC, DIRECT_ACCESS

direct_access_index_specobjectRequired

Specification for Direct Vector Access Index. Required if index_type is DIRECT_ACCESS.

Show child attributesHide child attributes
embedding_vector_columnsarray of object

The columns that contain the embedding vectors. The format should be array[double].

Show child attributesHide child attributes
namestring

Name of the column

embedding_dimensionint32

Dimension of the embedding vector

schema_jsonstring

The schema of the index in JSON format. Supported types are integer, long, float, double, boolean, string, date, timestamp. Supported types for vector column: array<float>, array<double>,`.

embedding_source_columnsarray of object

The columns that contain the embedding source. The format should be array[double].

Show child attributesHide child attributes
namestring

Name of the column

embedding_model_endpoint_namestringRequired

Name of the embedding model endpoint, used by default for both ingestion and querying.

model_endpoint_name_for_querystring

Name of the embedding model endpoint which, if specified, is used for querying (not ingestion).

delta_sync_index_specobjectRequired

Specification for Delta Sync Index. Required if index_type is DELTA_SYNC.

Show child attributesHide child attributes
source_tablestring

The name of the source table.

embedding_source_columnsarray of object

The columns that contain the embedding source.

Show child attributesHide child attributes
namestring

Name of the column

embedding_model_endpoint_namestringRequired

Name of the embedding model endpoint, used by default for both ingestion and querying.

model_endpoint_name_for_querystring

Name of the embedding model endpoint which, if specified, is used for querying (not ingestion).

embedding_vector_columnsarray of object

The columns that contain the embedding vectors.

Show child attributesHide child attributes
namestring

Name of the column

embedding_dimensionint32

Dimension of the embedding vector

pipeline_typestring

Pipeline execution mode.

  • TRIGGERED: If the pipeline uses the triggered execution mode, the system stops processing after successfully refreshing the source table in the pipeline once, ensuring the table is updated based on the data available when the update started.
  • CONTINUOUS: If the pipeline uses continuous execution, the pipeline processes new data as it arrives in the source table to keep vector index fresh.

Values: TRIGGERED, CONTINUOUS

pipeline_idstring

The ID of the pipeline that is used to sync the index.

embedding_writeback_tablestring

[Optional] Name of the Delta table to sync the vector index contents and computed embeddings to.

columns_to_syncarray of string

[Optional] Select the columns to sync with the vector index. If you leave this field blank, all columns from the source table are synced with the index. The primary key column and embedding source column or embedding vector column are always synced.

columns_to_indexarray of string

[Optional] Alias for columns_to_sync. Select the columns to include in the vector index. If you leave this field blank, all columns from the source table are included. The primary key column and embedding source column or embedding vector column are always included. Only one of columns_to_sync or columns_to_index may be specified.

index_subtypestring

The subtype of the index. Use HYBRID or FULL_TEXT. VECTOR is not supported.

Values: VECTOR, FULL_TEXT, HYBRID

Response

Returns the VectorIndex object.

Delete

DELETE /api/2.0/vector-search/indexes/{name}

Delete an index.

API scopes: vector-search

Parameters

namestringpath

Name of the index

Delete Data Vector Index

DELETE /api/2.0/vector-search/indexes/{name}/delete-data

Handles the deletion of data from a specified vector index.

API scopes: vector-search

Parameters

namestringpath

Name of the vector index where data is to be deleted. Must be a Direct Vector Access Index.

primary_keysarray of stringquery

List of primary keys for the data to be deleted.

Response

statusstring

Status of the delete operation.

Values: SUCCESS, PARTIAL_SUCCESS, FAILURE

resultobject

Result of the upsert or delete operation.

Show child attributesHide child attributes
success_row_countint64

Count of successfully processed rows.

failed_primary_keysarray of string

List of primary keys for rows that failed to process.

Query Vector Index

POST /api/2.0/vector-search/indexes/{name}/query

Query the specified vector index.

API scopes: vector-search

Parameters

namestringpath

Name of the vector index to query.

Request body

num_resultsint32

Number of results to return. Defaults to 10.

columnsarray of string

List of column names to include in the response.

filters_jsonstring

JSON string representing query filters.

Example filters:

  • {"id <": 5}: Filter for id less than 5.
  • {"id >": 5}: Filter for id greater than 5.
  • {"id <=": 5}: Filter for id less than equal to 5.
  • {"id >=": 5}: Filter for id greater than equal to 5.
  • {"id": 5}: Filter for id equal to 5.
query_vectorarray of float

Query vector. Required for Direct Vector Access Index and Delta Sync Index using self-managed vectors.

query_textstring

Query text. Required for Delta Sync Index using model endpoint.

score_thresholdfloat

Threshold for the approximate nearest neighbor search. Defaults to 0.0.

query_typestring

The query type to use. Choices are ANN and HYBRID and FULL_TEXT. Defaults to ANN.

rerankerobject

If set, the top 50 results are reranked with the Databricks Reranker model before returning the num_results results to the user. The setting columns_to_rerank selects which columns are used for reranking. For each datapoint, the columns selected are concatenated before being sent to the reranking model. See https://docs.databricks.com/aws/en/vector-search/query-vector-search#rerank for more information.

Show child attributesHide child attributes
modelstring

Reranker identifier:

  • When model_type=BASE/UNSPECIFIED: must be "databricks_reranker".
  • When model_type=FINETUNED: the Model Serving endpoint name hosting a finetuned reranker.
parametersobject

Parameters that control how the reranker processes the query results.

Show child attributesHide child attributes
columns_to_rerankarray of string
query_columnsarray of string

Text columns to search for query_text. When empty, all text columns are searched.

sort_columnsarray of string

Sort results by column values instead of the default relevance ordering. Each clause has the form "<column> ASC" or "<column> DESC", for example ["rating DESC", "price ASC"].

facetsarray of string

Facets to compute over the matched results. Each entry has one of these forms: "<column>" - top 10 distinct values by count "<column> TOP <n>" - top n distinct values, where n > 0 "<column> BUCKETS [[from,to],...]" - inclusive numeric ranges TOP and BUCKETS are case-insensitive. A column may appear at most once.

Response

manifestobject

Metadata about the result set.

Show child attributesHide child attributes
column_countint32

Number of columns in the result set.

columnsarray of object

Information about each column in the result set.

Show child attributesHide child attributes
namestring

Name of the column.

type_textstring

Data type of the column (e.g., "string", "int", "array<float>")

facet_column_countint32

Number of columns in facet_result.

facet_columnsarray of object

Information about each column in facet_result.

Show child attributesHide child attributes
namestring

Name of the column.

type_textstring

Data type of the column (e.g., "string", "int", "array<float>")

resultobject

Data returned in the query result.

Show child attributesHide child attributes
row_countint32

Number of rows in the result set.

data_arrayarray of array of object

Data rows returned in the query.

next_page_tokenstring

[Optional] Token that can be used in QueryVectorIndexNextPage API to get next page of results. If more than 1000 results satisfy the query, they are returned in groups of 1000. Empty value means no more results. The maximum number of results that can be returned is 10,000.

facet_resultobject

Facet aggregation rows returned by a query.

Show child attributesHide child attributes
facet_row_countint32

Number of facet rows returned.

facet_arrayarray of array of object

Facet rows. Each row is [facet_column_name, value_or_range, count].

Query Vector Index Next Page

POST /api/2.0/vector-search/indexes/{name}/query-next-page

Use next_page_token returned from previous QueryVectorIndex or QueryVectorIndexNextPage request to fetch next page of results.

API scopes: vector-search

Parameters

namestringpath

Name of the vector index to query.

Request body

endpoint_namestring

Name of the endpoint.

page_tokenstring

Page token returned from previous QueryVectorIndex or QueryVectorIndexNextPage API.

Response

manifestobject

Metadata about the result set.

Show child attributesHide child attributes
column_countint32

Number of columns in the result set.

columnsarray of object

Information about each column in the result set.

Show child attributesHide child attributes
namestring

Name of the column.

type_textstring

Data type of the column (e.g., "string", "int", "array<float>")

facet_column_countint32

Number of columns in facet_result.

facet_columnsarray of object

Information about each column in facet_result.

Show child attributesHide child attributes
namestring

Name of the column.

type_textstring

Data type of the column (e.g., "string", "int", "array<float>")

resultobject

Data returned in the query result.

Show child attributesHide child attributes
row_countint32

Number of rows in the result set.

data_arrayarray of array of object

Data rows returned in the query.

next_page_tokenstring

[Optional] Token that can be used in QueryVectorIndexNextPage API to get next page of results. If more than 1000 results satisfy the query, they are returned in groups of 1000. Empty value means no more results. The maximum number of results that can be returned is 10,000.

facet_resultobject

Facet aggregation rows returned by a query.

Show child attributesHide child attributes
facet_row_countint32

Number of facet rows returned.

facet_arrayarray of array of object

Facet rows. Each row is [facet_column_name, value_or_range, count].

Scan Vector Index

POST /api/2.0/vector-search/indexes/{name}/scan

Scan the specified vector index and return the first num_results entries after the exclusive primary_key.

API scopes: vector-search

Parameters

namestringpath

Name of the vector index to scan.

Request body

num_resultsint32

Number of results to return. Defaults to 10.

last_primary_keystring

Primary key of the last entry returned in the previous scan.

Response

dataarray of object

List of data entries

Show child attributesHide child attributes
fieldsarray of object

Data entry, corresponding to a row in a vector index.

Show child attributesHide child attributes
keystring

Column name.

valueobject

Column value, nullable.

Show child attributesHide child attributes
number_valuedoubleRequired
string_valuestringRequired
bool_valuebooleanRequired
struct_valueobjectRequired
list_valueobjectRequired
last_primary_keystring

Primary key of the last entry.

Sync Vector Index

POST /api/2.0/vector-search/indexes/{name}/sync

Triggers a synchronization process for a specified vector index.

API scopes: vector-search

Parameters

namestringpath

Name of the vector index to synchronize. Must be a Delta Sync Index.

Upsert Data Vector Index

POST /api/2.0/vector-search/indexes/{name}/upsert-data

Handles the upserting of data into a specified vector index.

API scopes: vector-search

Parameters

namestringpath

Name of the vector index where data is to be upserted. Must be a Direct Vector Access Index.

Request body

inputs_jsonstring

JSON string representing the data to be upserted.

Response

statusstring

Status of the upsert operation.

Values: SUCCESS, PARTIAL_SUCCESS, FAILURE

resultobject

Result of the upsert or delete operation.

Show child attributesHide child attributes
success_row_countint64

Count of successfully processed rows.

failed_primary_keysarray of string

List of primary keys for rows that failed to process.