Serving Endpoint

View as Markdown

GetGET /api/2.0/serving-endpoints/{name}CreatePOST /api/2.0/serving-endpointsUpdatePUT /api/2.0/serving-endpoints/{name}/configDeleteDELETE /api/2.0/serving-endpoints/{name}Create Provisioned Throughput Inference EndpointPOST /api/2.0/serving-endpoints/ptGet Export Endpoint MetricsGET /api/2.0/serving-endpoints/{name}/metricsGet Inference Endpoint SchemaGET /api/2.0/serving-endpoints/{name}/openapiGet Served Model Build LogsGET /api/2.0/serving-endpoints/{name}/served-models/{served_model_name}/build-logsGet Served Model LogsGET /api/2.0/serving-endpoints/{name}/served-models/{served_model_name}/logsPatch Inference Endpoint Telemetry ConfigPATCH /api/2.0/serving-endpoints/{name}/telemetry-configPut Inference Endpoint Rate LimitsPUT /api/2.0/serving-endpoints/{name}/rate-limitsPut Provisioned Throughput Inference Endpoint ConfigPUT /api/2.0/serving-endpoints/pt/{name}/configUpdate Inference Endpoint NotificationsPATCH /api/2.0/serving-endpoints/{name}/notificationsPut Inference Endpoint Ai GatewayPUT /api/2.0/serving-endpoints/{name}/ai-gatewayPatch Inference Endpoint TagsPATCH /api/2.0/serving-endpoints/{name}/tags

Get

GET /api/2.0/serving-endpoints/{name}

Retrieves the details for a single serving endpoint.

API scopes: model-serving

Parameters

namestringpath

The name of the serving endpoint. This field is required.

Example: feed-ads

Constraints: [ 1 .. 63 ] characters

Response

namestring

The name of the serving endpoint.

Example: feed-ads

creatorstring

The email of the user who created the serving endpoint.

Example: alice@company.com

creation_timestampint64

The timestamp when the endpoint was created in Unix time.

last_updated_timestampint64

The timestamp when the endpoint was last updated by a user in Unix time.

stateobject

Information corresponding to the state of the serving endpoint.

Show child attributesHide child attributes

readystring

The state of an endpoint, indicating whether or not the endpoint is queryable. An endpoint is READY if all of the served entities in its active configuration are ready. If any of the actively served entities are in a non-ready state, the endpoint state will be NOT_READY.

Values: READY_STATE_UNSPECIFIED, READY, NOT_READY

config_updatestring

The state of an endpoint's config update. This informs the user if the pending_config is in progress, if the update failed, or if there is no update in progress. Note that if the endpoint's config_update state value is IN_PROGRESS, another update can not be made until the update completes or fails.

Values: CONFIG_UPDATE_STATE_UNSPECIFIED, NOT_UPDATING, IN_PROGRESS, UPDATE_FAILED, UPDATE_CANCELED

configobject

The config that is currently being served by the endpoint.

Show child attributesHide child attributes

config_versionint64

The config version that the serving endpoint is currently serving.

served_entitiesarray of object

The list of served entities under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

served_modelsarray of object

(Deprecated, use served_entities instead) The list of served models under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

traffic_configobject

The traffic configuration associated with the serving endpoint config.

Show child attributesHide child attributes

routesarray of object

The list of routes that define traffic to each served entity.

Show child attributesHide child attributes

served_model_namestring

The name of the served model this route configures traffic for.

Example: ads-model-3

traffic_percentageint32

The percentage of endpoint traffic to send to this route. It must be an integer between 0 and 100 inclusive.

Example: 100

served_entity_namestring

Example: ads-model-3

auto_capture_configobject

Configuration for legacy Inference Tables which automatically log requests and responses to Unity Catalog. Deprecated: please use AI Gateway inference tables instead. See https://docs.databricks.com/aws/en/ai-gateway/inference-tables.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog name if the inference table is already enabled.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. NOTE: On update, you cannot change the schema name if the inference table is already enabled.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.

Example: my-prefix-

stateobject

Show child attributesHide child attributes

payload_tableobject

enabledboolean

Indicates whether the inference table is enabled.

Example: true

pending_configobject

The config that the endpoint is attempting to update to.

Show child attributesHide child attributes

served_entitiesarray of object

The list of served entities belonging to the last issued update to the serving endpoint.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

served_modelsarray of object

(Deprecated, use served_entities instead) The list of served models belonging to the last issued update to the serving endpoint.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

traffic_configobject

The traffic config defining how invocations to the serving endpoint should be routed.

Show child attributesHide child attributes

routesarray of object

The list of routes that define traffic to each served entity.

Show child attributesHide child attributes

served_model_namestring

The name of the served model this route configures traffic for.

Example: ads-model-3

traffic_percentageint32

The percentage of endpoint traffic to send to this route. It must be an integer between 0 and 100 inclusive.

Example: 100

served_entity_namestring

Example: ads-model-3

config_versionint32

The config version that the serving endpoint is currently serving.

start_timeint64

The timestamp when the update to the pending config started.

auto_capture_configobject

Configuration for legacy Inference Tables which automatically log requests and responses to Unity Catalog. Deprecated: please use AI Gateway inference tables instead. See https://docs.databricks.com/aws/en/ai-gateway/inference-tables.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog name if the inference table is already enabled.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. NOTE: On update, you cannot change the schema name if the inference table is already enabled.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.

Example: my-prefix-

stateobject

Show child attributesHide child attributes

payload_tableobject

enabledboolean

Indicates whether the inference table is enabled.

Example: true

idstring

System-generated ID of the endpoint. This is used to refer to the endpoint in the Permissions API

Example: 88fd3f75a0d24b0380ddc40484d7a31b

permission_levelstring

The permission level of the principal making the request.

Values: CAN_MANAGE, CAN_QUERY, CAN_VIEW

Example: CAN_MANAGE

tagsarray of object

Tags attached to the serving endpoint.

Show child attributesHide child attributes

keystring

Key field for a serving endpoint tag.

Example: team

valuestring

Optional value field for a serving endpoint tag.

Example: data science

taskstring

The task type of the serving endpoint.

Example: model-serving-task

route_optimizedboolean

Boolean representing if route optimization has been enabled for the endpoint

Example: true

endpoint_urlstring

Endpoint invocation url if route optimization is enabled for endpoint

data_plane_infoobject

Information required to query DataPlane APIs.

Show child attributesHide child attributes

query_infoobject

Information required to query DataPlane API 'query' endpoint.

Show child attributesHide child attributes

endpoint_urlstring

The URL of the endpoint for this operation in the dataplane.

authorization_detailsstring

Authorization details as a string.

ai_gatewayobject

The AI Gateway configuration for the serving endpoint. NOTE: External model, provisioned throughput, and pay-per-token endpoints are fully supported; agent endpoints currently only support inference tables.

Show child attributesHide child attributes

usage_tracking_configobject

Configuration to enable usage tracking using system tables. These tables allow you to monitor operational usage on endpoints and their associated costs.

Show child attributesHide child attributes

enabledboolean

Whether to enable usage tracking.

Example: true

inference_table_configobject

Configuration for payload logging using inference tables. Use these tables to monitor and audit data being sent to and received from model APIs and to improve model quality.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the catalog name.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the schema name.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you have to disable inference table first in order to change the prefix name.

Example: my-prefix

enabledboolean

Indicates whether the inference table is enabled.

Example: true

rate_limitsarray of object

Configuration for rate limits which can be set to limit endpoint traffic.

Show child attributesHide child attributes

callsint64

Used to specify how many calls are allowed for a key within the renewal_period.

Example: 15

keystring

Key field for a rate limit. Currently, 'user', 'user_group, 'service_principal', and 'endpoint' are supported, with 'endpoint' being the default if not specified.

renewal_periodstring

Renewal period field for a rate limit. Currently, only 'minute' is supported.

principalstring

Principal field for a user, user group, or service principal to apply rate limiting to. Accepts a user email, group name, or service principal application ID.

Example: user@test.com

tokensint64

Used to specify how many tokens are allowed for a key within the renewal_period.

Example: 10000

guardrailsobject

Configuration for AI Guardrails to prevent unwanted data and unsafe data in requests and responses.

Show child attributesHide child attributes

inputobject

Configuration for input guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

outputobject

Configuration for output guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

fallback_configobject

Configuration for traffic fallback which auto fallbacks to other served entities if the request to a served entity fails with certain error codes, to increase availability.

Show child attributesHide child attributes

enabledboolean

Whether to enable traffic fallback. When a served entity in the serving endpoint returns specific error codes (e.g. 500), the request will automatically be round-robin attempted with other served entities in the same endpoint, following the order of served entity list, until a successful response is returned. If all attempts fail, return the last response with the error code.

Example: true

budget_policy_idstring

The budget policy associated with the endpoint.

email_notificationsobject

Email notification settings.

Show child attributesHide child attributes

on_update_successarray of string

A list of email addresses to be notified when an endpoint successfully updates its configuration or state.

Example: user.name@databricks.com

on_update_failurearray of string

A list of email addresses to be notified when an endpoint fails to update its configuration or state.

Example: user.name@databricks.com

descriptionstring

Description of the serving model

telemetry_configobject

Telemetry configuration for the endpoint, including inference-table payload logging.

Show child attributesHide child attributes

telemetry_profile_idstringRequired

The ID of an existing telemetry profile to apply to this endpoint. Provide this to reuse a telemetry profile that has already been created, instead of specifying table_names.

table_namesobjectRequired

The Unity Catalog tables to which endpoint telemetry (logs, traces, and metrics) is exported. Provide this to create a new telemetry profile for the endpoint from the given tables.

Show child attributesHide child attributes

logs_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported logs.

metrics_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported metrics.

traces_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported traces (spans).

annotations_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported annotations.

inference_table_configobject

Configuration for inference table payload logging, including sampling.

Show child attributesHide child attributes

sampling_fractiondouble

Fraction of requests sampled for payload logging, in the range [0.0, 1.0], where 1.0 logs all requests.

namestring

The full name of the inference table created for this endpoint.

enabled_telemetry_featuresarray of string

The telemetry signals to enable for this endpoint. If empty or omitted, all signals are enabled; otherwise only the listed signals are enabled.

Values: TELEMETRY_FEATURE_LOGS, TELEMETRY_FEATURE_TRACES, TELEMETRY_FEATURE_METRICS, TELEMETRY_FEATURE_INFERENCE_TABLE

View as Markdown

Response

External model endpointFeature Serving endpointFoundation Model EndpointModels in Unity Catalog endpointWorkspace model endpoint

{
  "ai_gateway": {
    "guardrails": {
      "input": {
        "invalid_keywords": [
          "keyword1",
          "keyword2"
        ],
        "pii": {
          "behavior": "BLOCK"
        },
        "safety": true,
        "valid_topics": [
          "topic1",
          "topic2"
        ]
      },
      "output": {
        "invalid_keywords": [
          "keyword1",
          "keyword2"
        ],
        "pii": {
          "behavior": "BLOCK"
        },
        "safety": true,
        "valid_topics": [
          "topic1",
          "topic2"
        ]
      }
    },
    "inference_table_config": {
      "catalog_name": "my-catalog",
      "enabled": true,
      "schema_name": "my-schema",
      "table_name_prefix": "my-prefix"
    },
    "rate_limits": [
      {
        "calls": 100,
        "key": "user",
        "renewal_period": "minute"
      }
    ],
    "usage_tracking_config": {
      "enabled": true
    }
  },
  "config": {
    "config_version": 1,
    "served_entities": [
      {
        "creation_timestamp": 1699617587000,
        "creator": "user@email.com",
        "external_model": {
          "name": "text-embedding-ada-002",
          "openai_config": {
            "openai_api_key": "{{secrets/my_scope/my_openai_api_key}}"
          },
          "provider": "openai",
          "task": "llm/v1/embeddings"
        },
        "name": "openai_embeddings",
        "state": {
          "deployment": "DEPLOYMENT_READY",
          "deployment_state_message": ""
        }
      }
    ],
    "traffic_config": {
      "routes": [
        {
          "served_model_name": "openai_embeddings",
          "traffic_percentage": 100
        }
      ]
    }
  },
  "creation_timestamp": 1699617587000,
  "creator": "user@email.com",
  "id": "69962db6b9db47c4a8a222d2ac79d7f8",
  "last_updated_timestamp": 1699617587000,
  "name": "openai_endpoint",
  "permission_level": "CAN_MANAGE",
  "route_optimized": false,
  "state": {
    "ready": "READY"
  },
  "tags": {
    "key": "team",
    "value": "gen-ai"
  }
}

Create

POST /api/2.0/serving-endpoints

Create a new serving endpoint.

API scopes: model-serving

Request body

namestring

The name of the serving endpoint. This field is required and must be unique across a <Workspace>. An endpoint name can consist of alphanumeric characters, dashes, and underscores.

Example: feed-ads

configobject

The core config of the serving endpoint.

Show child attributesHide child attributes

served_entitiesarray of object

The list of served entities under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

served_modelsarray of object

(Deprecated, use served_entities instead) The list of served models under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

traffic_configobject

The traffic configuration associated with the serving endpoint config.

Show child attributesHide child attributes

routesarray of object

The list of routes that define traffic to each served entity.

Show child attributesHide child attributes

served_model_namestring

The name of the served model this route configures traffic for.

Example: ads-model-3

traffic_percentageint32

The percentage of endpoint traffic to send to this route. It must be an integer between 0 and 100 inclusive.

Example: 100

served_entity_namestring

Example: ads-model-3

auto_capture_configobject

Configuration for legacy Inference Tables which automatically log requests and responses to Unity Catalog. Deprecated: please use AI Gateway inference tables instead. See https://docs.databricks.com/aws/en/ai-gateway/inference-tables.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog name if the inference table is already enabled.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. NOTE: On update, you cannot change the schema name if the inference table is already enabled.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.

Example: my-prefix-

stateobject

Show child attributesHide child attributes

payload_tableobject

enabledboolean

Indicates whether the inference table is enabled.

Example: true

tagsarray of object

Tags to be attached to the serving endpoint and automatically propagated to billing logs.

Show child attributesHide child attributes

keystring

Key field for a serving endpoint tag.

Example: team

valuestring

Optional value field for a serving endpoint tag.

Example: data science

route_optimizedboolean

Enable route optimization for the serving endpoint.

Example: true

rate_limitsarray of object

Rate limits to be applied to the serving endpoint. NOTE: this field is deprecated, please use AI Gateway to manage rate limits.

Show child attributesHide child attributes

callsint64

Used to specify how many calls are allowed for a key within the renewal_period.

Example: 15

keystring

Key field for a serving endpoint rate limit. Currently, only 'user' and 'endpoint' are supported, with 'endpoint' being the default if not specified.

renewal_periodstring

Renewal period field for a serving endpoint rate limit. Currently, only 'minute' is supported.

ai_gatewayobject

The AI Gateway configuration for the serving endpoint. NOTE: External model, provisioned throughput, and pay-per-token endpoints are fully supported; agent endpoints currently only support inference tables.

Show child attributesHide child attributes

usage_tracking_configobject

Configuration to enable usage tracking using system tables. These tables allow you to monitor operational usage on endpoints and their associated costs.

Show child attributesHide child attributes

enabledboolean

Whether to enable usage tracking.

Example: true

inference_table_configobject

Configuration for payload logging using inference tables. Use these tables to monitor and audit data being sent to and received from model APIs and to improve model quality.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the catalog name.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the schema name.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you have to disable inference table first in order to change the prefix name.

Example: my-prefix

enabledboolean

Indicates whether the inference table is enabled.

Example: true

rate_limitsarray of object

Configuration for rate limits which can be set to limit endpoint traffic.

Show child attributesHide child attributes

callsint64

Used to specify how many calls are allowed for a key within the renewal_period.

Example: 15

keystring

Key field for a rate limit. Currently, 'user', 'user_group, 'service_principal', and 'endpoint' are supported, with 'endpoint' being the default if not specified.

renewal_periodstring

Renewal period field for a rate limit. Currently, only 'minute' is supported.

principalstring

Principal field for a user, user group, or service principal to apply rate limiting to. Accepts a user email, group name, or service principal application ID.

Example: user@test.com

tokensint64

Used to specify how many tokens are allowed for a key within the renewal_period.

Example: 10000

guardrailsobject

Configuration for AI Guardrails to prevent unwanted data and unsafe data in requests and responses.

Show child attributesHide child attributes

inputobject

Configuration for input guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

outputobject

Configuration for output guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

fallback_configobject

Configuration for traffic fallback which auto fallbacks to other served entities if the request to a served entity fails with certain error codes, to increase availability.

Show child attributesHide child attributes

enabledboolean

Whether to enable traffic fallback. When a served entity in the serving endpoint returns specific error codes (e.g. 500), the request will automatically be round-robin attempted with other served entities in the same endpoint, following the order of served entity list, until a successful response is returned. If all attempts fail, return the last response with the error code.

Example: true

budget_policy_idstring

The budget policy to be applied to the serving endpoint.

email_notificationsobject

Email notification settings.

Show child attributesHide child attributes

on_update_successarray of string

A list of email addresses to be notified when an endpoint successfully updates its configuration or state.

Example: user.name@databricks.com

on_update_failurearray of string

A list of email addresses to be notified when an endpoint fails to update its configuration or state.

Example: user.name@databricks.com

descriptionstring

Constraints: <= 1000 characters

telemetry_configobject

Configuration for persisting endpoint telemetry (logs, traces, and metrics) to Unity Catalog tables.

Show child attributesHide child attributes

telemetry_profile_idstringRequired

The ID of an existing telemetry profile to apply to this endpoint. Provide this to reuse a telemetry profile that has already been created, instead of specifying table_names.

table_namesobjectRequired

The Unity Catalog tables to which endpoint telemetry (logs, traces, and metrics) is exported. Provide this to create a new telemetry profile for the endpoint from the given tables.

Show child attributesHide child attributes

logs_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported logs.

metrics_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported metrics.

traces_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported traces (spans).

annotations_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported annotations.

inference_table_configobject

Configuration for inference table payload logging, including sampling.

Show child attributesHide child attributes

sampling_fractiondouble

Fraction of requests sampled for payload logging, in the range [0.0, 1.0], where 1.0 logs all requests.

namestring

The full name of the inference table created for this endpoint.

enabled_telemetry_featuresarray of string

The telemetry signals to enable for this endpoint. If empty or omitted, all signals are enabled; otherwise only the listed signals are enabled.

Values: TELEMETRY_FEATURE_LOGS, TELEMETRY_FEATURE_TRACES, TELEMETRY_FEATURE_METRICS, TELEMETRY_FEATURE_INFERENCE_TABLE

Response

namestring

The name of the serving endpoint.

Example: feed-ads

creatorstring

The email of the user who created the serving endpoint.

Example: alice@company.com

creation_timestampint64

The timestamp when the endpoint was created in Unix time.

last_updated_timestampint64

The timestamp when the endpoint was last updated by a user in Unix time.

stateobject

Information corresponding to the state of the serving endpoint.

Show child attributesHide child attributes

readystring

The state of an endpoint, indicating whether or not the endpoint is queryable. An endpoint is READY if all of the served entities in its active configuration are ready. If any of the actively served entities are in a non-ready state, the endpoint state will be NOT_READY.

Values: READY_STATE_UNSPECIFIED, READY, NOT_READY

config_updatestring

The state of an endpoint's config update. This informs the user if the pending_config is in progress, if the update failed, or if there is no update in progress. Note that if the endpoint's config_update state value is IN_PROGRESS, another update can not be made until the update completes or fails.

Values: CONFIG_UPDATE_STATE_UNSPECIFIED, NOT_UPDATING, IN_PROGRESS, UPDATE_FAILED, UPDATE_CANCELED

configobject

The config that is currently being served by the endpoint.

Show child attributesHide child attributes

config_versionint64

The config version that the serving endpoint is currently serving.

served_entitiesarray of object

The list of served entities under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

served_modelsarray of object

(Deprecated, use served_entities instead) The list of served models under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

traffic_configobject

The traffic configuration associated with the serving endpoint config.

Show child attributesHide child attributes

routesarray of object

The list of routes that define traffic to each served entity.

Show child attributesHide child attributes

served_model_namestring

The name of the served model this route configures traffic for.

Example: ads-model-3

traffic_percentageint32

The percentage of endpoint traffic to send to this route. It must be an integer between 0 and 100 inclusive.

Example: 100

served_entity_namestring

Example: ads-model-3

auto_capture_configobject

Configuration for legacy Inference Tables which automatically log requests and responses to Unity Catalog. Deprecated: please use AI Gateway inference tables instead. See https://docs.databricks.com/aws/en/ai-gateway/inference-tables.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog name if the inference table is already enabled.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. NOTE: On update, you cannot change the schema name if the inference table is already enabled.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.

Example: my-prefix-

stateobject

Show child attributesHide child attributes

payload_tableobject

enabledboolean

Indicates whether the inference table is enabled.

Example: true

pending_configobject

The config that the endpoint is attempting to update to.

Show child attributesHide child attributes

served_entitiesarray of object

The list of served entities belonging to the last issued update to the serving endpoint.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

served_modelsarray of object

(Deprecated, use served_entities instead) The list of served models belonging to the last issued update to the serving endpoint.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

traffic_configobject

The traffic config defining how invocations to the serving endpoint should be routed.

Show child attributesHide child attributes

routesarray of object

The list of routes that define traffic to each served entity.

Show child attributesHide child attributes

served_model_namestring

The name of the served model this route configures traffic for.

Example: ads-model-3

traffic_percentageint32

The percentage of endpoint traffic to send to this route. It must be an integer between 0 and 100 inclusive.

Example: 100

served_entity_namestring

Example: ads-model-3

config_versionint32

The config version that the serving endpoint is currently serving.

start_timeint64

The timestamp when the update to the pending config started.

auto_capture_configobject

Configuration for legacy Inference Tables which automatically log requests and responses to Unity Catalog. Deprecated: please use AI Gateway inference tables instead. See https://docs.databricks.com/aws/en/ai-gateway/inference-tables.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog name if the inference table is already enabled.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. NOTE: On update, you cannot change the schema name if the inference table is already enabled.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.

Example: my-prefix-

stateobject

Show child attributesHide child attributes

payload_tableobject

enabledboolean

Indicates whether the inference table is enabled.

Example: true

idstring

System-generated ID of the endpoint. This is used to refer to the endpoint in the Permissions API

Example: 88fd3f75a0d24b0380ddc40484d7a31b

permission_levelstring

The permission level of the principal making the request.

Values: CAN_MANAGE, CAN_QUERY, CAN_VIEW

Example: CAN_MANAGE

tagsarray of object

Tags attached to the serving endpoint.

Show child attributesHide child attributes

keystring

Key field for a serving endpoint tag.

Example: team

valuestring

Optional value field for a serving endpoint tag.

Example: data science

taskstring

The task type of the serving endpoint.

Example: model-serving-task

route_optimizedboolean

Boolean representing if route optimization has been enabled for the endpoint

Example: true

endpoint_urlstring

Endpoint invocation url if route optimization is enabled for endpoint

data_plane_infoobject

Information required to query DataPlane APIs.

Show child attributesHide child attributes

query_infoobject

Information required to query DataPlane API 'query' endpoint.

Show child attributesHide child attributes

endpoint_urlstring

The URL of the endpoint for this operation in the dataplane.

authorization_detailsstring

Authorization details as a string.

ai_gatewayobject

The AI Gateway configuration for the serving endpoint. NOTE: External model, provisioned throughput, and pay-per-token endpoints are fully supported; agent endpoints currently only support inference tables.

Show child attributesHide child attributes

usage_tracking_configobject

Configuration to enable usage tracking using system tables. These tables allow you to monitor operational usage on endpoints and their associated costs.

Show child attributesHide child attributes

enabledboolean

Whether to enable usage tracking.

Example: true

inference_table_configobject

Configuration for payload logging using inference tables. Use these tables to monitor and audit data being sent to and received from model APIs and to improve model quality.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the catalog name.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the schema name.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you have to disable inference table first in order to change the prefix name.

Example: my-prefix

enabledboolean

Indicates whether the inference table is enabled.

Example: true

rate_limitsarray of object

Configuration for rate limits which can be set to limit endpoint traffic.

Show child attributesHide child attributes

callsint64

Used to specify how many calls are allowed for a key within the renewal_period.

Example: 15

keystring

Key field for a rate limit. Currently, 'user', 'user_group, 'service_principal', and 'endpoint' are supported, with 'endpoint' being the default if not specified.

renewal_periodstring

Renewal period field for a rate limit. Currently, only 'minute' is supported.

principalstring

Principal field for a user, user group, or service principal to apply rate limiting to. Accepts a user email, group name, or service principal application ID.

Example: user@test.com

tokensint64

Used to specify how many tokens are allowed for a key within the renewal_period.

Example: 10000

guardrailsobject

Configuration for AI Guardrails to prevent unwanted data and unsafe data in requests and responses.

Show child attributesHide child attributes

inputobject

Configuration for input guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

outputobject

Configuration for output guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

fallback_configobject

Configuration for traffic fallback which auto fallbacks to other served entities if the request to a served entity fails with certain error codes, to increase availability.

Show child attributesHide child attributes

enabledboolean

Whether to enable traffic fallback. When a served entity in the serving endpoint returns specific error codes (e.g. 500), the request will automatically be round-robin attempted with other served entities in the same endpoint, following the order of served entity list, until a successful response is returned. If all attempts fail, return the last response with the error code.

Example: true

budget_policy_idstring

The budget policy associated with the endpoint.

email_notificationsobject

Email notification settings.

Show child attributesHide child attributes

on_update_successarray of string

A list of email addresses to be notified when an endpoint successfully updates its configuration or state.

Example: user.name@databricks.com

on_update_failurearray of string

A list of email addresses to be notified when an endpoint fails to update its configuration or state.

Example: user.name@databricks.com

descriptionstring

Description of the serving model

telemetry_configobject

Telemetry configuration for the endpoint, including inference-table payload logging.

Show child attributesHide child attributes

telemetry_profile_idstringRequired

The ID of an existing telemetry profile to apply to this endpoint. Provide this to reuse a telemetry profile that has already been created, instead of specifying table_names.

table_namesobjectRequired

The Unity Catalog tables to which endpoint telemetry (logs, traces, and metrics) is exported. Provide this to create a new telemetry profile for the endpoint from the given tables.

Show child attributesHide child attributes

logs_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported logs.

metrics_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported metrics.

traces_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported traces (spans).

annotations_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported annotations.

inference_table_configobject

Configuration for inference table payload logging, including sampling.

Show child attributesHide child attributes

sampling_fractiondouble

Fraction of requests sampled for payload logging, in the range [0.0, 1.0], where 1.0 logs all requests.

namestring

The full name of the inference table created for this endpoint.

enabled_telemetry_featuresarray of string

The telemetry signals to enable for this endpoint. If empty or omitted, all signals are enabled; otherwise only the listed signals are enabled.

Values: TELEMETRY_FEATURE_LOGS, TELEMETRY_FEATURE_TRACES, TELEMETRY_FEATURE_METRICS, TELEMETRY_FEATURE_INFERENCE_TABLE

View as Markdown

Request

External model endpointFeature Serving endpointModels in Unity Catalog endpointWorkspace model endpoint

{
  "ai_gateway": {
    "guardrails": {
      "input": {
        "invalid_keywords": [
          "keyword1",
          "keyword2"
        ],
        "pii": {
          "behavior": "BLOCK"
        },
        "safety": true,
        "valid_topics": [
          "topic1",
          "topic2"
        ]
      },
      "output": {
        "invalid_keywords": [
          "keyword1",
          "keyword2"
        ],
        "pii": {
          "behavior": "BLOCK"
        },
        "safety": true,
        "valid_topics": [
          "topic1",
          "topic2"
        ]
      }
    },
    "inference_table_config": {
      "catalog_name": "my-catalog",
      "enabled": true,
      "schema_name": "my-schema",
      "table_name_prefix": "my-prefix"
    },
    "rate_limits": [
      {
        "calls": 100,
        "key": "user",
        "renewal_period": "minute"
      }
    ],
    "usage_tracking_config": {
      "enabled": true
    }
  },
  "config": {
    "served_entities": [
      {
        "external_model": {
          "name": "text-embedding-ada-002",
          "openai_config": {
            "openai_api_key": "{{secrets/my_scope/my_openai_api_key}}"
          },
          "provider": "openai",
          "task": "llm/v1/embeddings"
        },
        "name": "openai_embeddings"
      }
    ]
  },
  "name": "openai_endpoint",
  "tags": [
    {
      "key": "team",
      "value": "gen-ai"
    }
  ]
}

Response

External model endpointFeature Serving endpointModels in Unity Catalog endpointWorkspace model endpoint

{
  "ai_gateway": {
    "guardrails": {
      "input": {
        "invalid_keywords": [
          "keyword1",
          "keyword2"
        ],
        "pii": {
          "behavior": "BLOCK"
        },
        "safety": true,
        "valid_topics": [
          "topic1",
          "topic2"
        ]
      },
      "output": {
        "invalid_keywords": [
          "keyword1",
          "keyword2"
        ],
        "pii": {
          "behavior": "BLOCK"
        },
        "safety": true,
        "valid_topics": [
          "topic1",
          "topic2"
        ]
      }
    },
    "inference_table_config": {
      "catalog_name": "my-catalog",
      "enabled": true,
      "schema_name": "my-schema",
      "table_name_prefix": "my-prefix"
    },
    "rate_limits": [
      {
        "calls": 100,
        "key": "user",
        "renewal_period": "minute"
      }
    ],
    "usage_tracking_config": {
      "enabled": true
    }
  },
  "config": {
    "config_version": 1,
    "served_entities": [
      {
        "creation_timestamp": 1699617587000,
        "creator": "user@email.com",
        "external_model": {
          "name": "text-embedding-ada-002",
          "openai_config": {
            "openai_api_key": "{{secrets/my_scope/my_openai_api_key}}"
          },
          "provider": "openai",
          "task": "llm/v1/embeddings"
        },
        "name": "openai_embeddings",
        "state": {
          "deployment": "DEPLOYMENT_READY",
          "deployment_state_message": ""
        }
      }
    ],
    "traffic_config": {
      "routes": [
        {
          "served_model_name": "openai_embeddings",
          "traffic_percentage": 100
        }
      ]
    }
  },
  "creation_timestamp": 1699617587000,
  "creator": "user@email.com",
  "id": "69962db6b9db47c4a8a222d2ac79d7f8",
  "last_updated_timestamp": 1699617587000,
  "name": "openai_endpoint",
  "permission_level": "CAN_MANAGE",
  "route_optimized": false,
  "state": {
    "ready": "READY"
  },
  "tags": {
    "key": "team",
    "value": "gen-ai"
  }
}

Update

PUT /api/2.0/serving-endpoints/{name}/config

Updates any combination of the serving endpoint's served entities, the compute configuration of those served entities, and the endpoint's traffic config. An endpoint that already has an update in progress can not be updated until the current update completes or fails.

API scopes: model-serving

Parameters

namestringpath

The name of the serving endpoint to update. This field is required.

Example: feed-ads

Constraints: [ 1 .. 63 ] characters

Request body

served_entitiesarray of object

The list of served entities under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

Show child attributesHide child attributes

ai21labs_api_keystring

The <Databricks> secret key reference for an AI21 Labs API key. If you prefer to paste your API key directly, see ai21labs_api_key_plaintext. You must provide an API key using one of the following fields: ai21labs_api_key or ai21labs_api_key_plaintext.

Example: {{secrets/my_scope/my_ai21labs_api_key}}

ai21labs_api_key_plaintextstring

An AI21 Labs API key provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see ai21labs_api_key. You must provide an API key using one of the following fields: ai21labs_api_key or ai21labs_api_key_plaintext.

Example: Your API Key

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

Show child attributesHide child attributes

anthropic_api_keystring

The <Databricks> secret key reference for an Anthropic API key. If you prefer to paste your API key directly, see anthropic_api_key_plaintext. You must provide an API key using one of the following fields: anthropic_api_key or anthropic_api_key_plaintext.

Example: {{secrets/my_scope/my_anthropic_api_key}}

anthropic_api_key_plaintextstring

The Anthropic API key provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see anthropic_api_key. You must provide an API key using one of the following fields: anthropic_api_key or anthropic_api_key_plaintext.

Example: Your API Key

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

Show child attributesHide child attributes

aws_regionstring

The AWS region to use. Bedrock has to be enabled there.

Example: myAwsRegion

aws_access_key_idstring

The <Databricks> secret key reference for an AWS access key ID with permissions to interact with Bedrock services. If you prefer to paste your API key directly, see aws_access_key_id_plaintext. You must provide an API key using one of the following fields: aws_access_key_id or aws_access_key_id_plaintext.

Example: {{secrets/my_scope/my_aws_access_key_id}}

aws_secret_access_keystring

The <Databricks> secret key reference for an AWS secret access key paired with the access key ID, with permissions to interact with Bedrock services. If you prefer to paste your API key directly, see aws_secret_access_key_plaintext. You must provide an API key using one of the following fields: aws_secret_access_key or aws_secret_access_key_plaintext.

Example: {{secrets/my_scope/my_aws_secret_access_key}}

bedrock_providerstring

The underlying provider in Amazon Bedrock. Supported values (case insensitive) include: Anthropic, Cohere, AI21Labs, Amazon.

aws_access_key_id_plaintextstring

An AWS access key ID with permissions to interact with Bedrock services provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see aws_access_key_id. You must provide an API key using one of the following fields: aws_access_key_id or aws_access_key_id_plaintext.

Example: Your AWS Access Key ID

aws_secret_access_key_plaintextstring

An AWS secret access key paired with the access key ID, with permissions to interact with Bedrock services provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see aws_secret_access_key. You must provide an API key using one of the following fields: aws_secret_access_key or aws_secret_access_key_plaintext.

Example: Your AWS Secret Access Key

instance_profile_arnstring

ARN of the instance profile that the external model will use to access AWS resources. You must authenticate using an instance profile or access keys. If you prefer to authenticate using access keys, see aws_access_key_id, aws_access_key_id_plaintext, aws_secret_access_key and aws_secret_access_key_plaintext.

Example: arn:aws:iam::123456789012:instance-profile/my-instance-profile

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

Show child attributesHide child attributes

cohere_api_keystring

The <Databricks> secret key reference for a Cohere API key. If you prefer to paste your API key directly, see cohere_api_key_plaintext. You must provide an API key using one of the following fields: cohere_api_key or cohere_api_key_plaintext.

Example: {{secrets/my_scope/my_cohere_api_key}}

cohere_api_key_plaintextstring

The Cohere API key provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see cohere_api_key. You must provide an API key using one of the following fields: cohere_api_key or cohere_api_key_plaintext.

Example: Your API Key

cohere_api_basestring

This is an optional field to provide a customized base URL for the Cohere API. If left unspecified, the standard Cohere base URL is used.

Example: https://api.cohere.ai/v1

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

Show child attributesHide child attributes

private_keystring

The <Databricks> secret key reference for a private key for the service account which has access to the Google Cloud Vertex AI Service. See Best practices for managing service account keys. If you prefer to paste your API key directly, see private_key_plaintext. You must provide an API key using one of the following fields: private_key or private_key_plaintext

Example: {{secrets/my_scope/my_google_cloud_vertex_ai_api_key}}

project_idstring

This is the Google Cloud project id that the service account is associated with.

Example: your-project-id

regionstring

This is the region for the Google Cloud Vertex AI Service. See supported regions for more details. Some models are only available in specific regions.

Example: us-central1

private_key_plaintextstring

The private key for the service account which has access to the Google Cloud Vertex AI Service provided as a plaintext secret. See Best practices for managing service account keys. If you prefer to reference your key using Databricks Secrets, see private_key. You must provide an API key using one of the following fields: private_key or private_key_plaintext.

Example: Your API Key

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

Show child attributesHide child attributes

databricks_api_tokenstring

The <Databricks> secret key reference for a Databricks API token that corresponds to a user or service principal with Can Query access to the model serving endpoint pointed to by this external model. If you prefer to paste your API key directly, see databricks_api_token_plaintext. You must provide an API key using one of the following fields: databricks_api_token or databricks_api_token_plaintext.

Example: {{secrets/my_scope/my_databricks_api_token}}

databricks_workspace_urlstring

The URL of the <Databricks> workspace containing the model serving endpoint pointed to by this external model.

Example: https://my-databricks-workspace.com

databricks_api_token_plaintextstring

The Databricks API token that corresponds to a user or service principal with Can Query access to the model serving endpoint pointed to by this external model provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see databricks_api_token. You must provide an API key using one of the following fields: databricks_api_token or databricks_api_token_plaintext.

Example: Your Databricks API Token

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

Show child attributesHide child attributes

openai_api_keystring

The <Databricks> secret key reference for an OpenAI API key using the OpenAI or Azure service. If you prefer to paste your API key directly, see openai_api_key_plaintext. You must provide an API key using one of the following fields: openai_api_key or openai_api_key_plaintext.

Example: {{secrets/my_scope/my_openai_api_key}}

openai_api_typestring

This is an optional field to specify the type of OpenAI API to use. For Azure OpenAI, this field is required, and adjust this parameter to represent the preferred security access validation protocol. For access token validation, use azure. For authentication using Azure Active Directory (Azure AD) use, azuread.

Example: azure

openai_api_basestring

This is a field to provide a customized base URl for the OpenAI API. For Azure OpenAI, this field is required, and is the base URL for the Azure OpenAI API service provided by Azure. For other OpenAI API types, this field is optional, and if left unspecified, the standard OpenAI base URL is used.

Example: https://api.openai.com/v1

openai_api_versionstring

This is an optional field to specify the OpenAI API version. For Azure OpenAI, this field is required, and is the version of the Azure OpenAI service to utilize, specified by a date.

Example: 2023-11-01

openai_deployment_namestring

This field is only required for Azure OpenAI and is the name of the deployment resource for the Azure OpenAI service.

Example: my_deployment_resource

openai_organizationstring

This is an optional field to specify the organization in OpenAI or Azure OpenAI.

Example: Databricks

microsoft_entra_tenant_idstring

This field is only required for Azure AD OpenAI and is the Microsoft Entra Tenant ID.

Example: 12345678-abcd-1234-5678-12345678abcd

microsoft_entra_client_idstring

This field is only required for Azure AD OpenAI and is the Microsoft Entra Client ID.

Example: 12345678-abcd-1234-5678-12345678abcd

microsoft_entra_client_secretstring

The <Databricks> secret key reference for a client secret used for Microsoft Entra ID authentication. If you prefer to paste your client secret directly, see microsoft_entra_client_secret_plaintext. You must provide an API key using one of the following fields: microsoft_entra_client_secret or microsoft_entra_client_secret_plaintext.

Example: {{secrets/my_scope/my_microsoft_entra_client_secret}}

openai_api_key_plaintextstring

The OpenAI API key using the OpenAI or Azure service provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see openai_api_key. You must provide an API key using one of the following fields: openai_api_key or openai_api_key_plaintext.

Example: Your API Key

microsoft_entra_client_secret_plaintextstring

The client secret used for Microsoft Entra ID authentication provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see microsoft_entra_client_secret. You must provide an API key using one of the following fields: microsoft_entra_client_secret or microsoft_entra_client_secret_plaintext.

Example: Your Microsoft Entra Client Secret

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

Show child attributesHide child attributes

palm_api_keystring

The <Databricks> secret key reference for a PaLM API key. If you prefer to paste your API key directly, see palm_api_key_plaintext. You must provide an API key using one of the following fields: palm_api_key or palm_api_key_plaintext.

Example: {{secrets/my_scope/my_palm_api_key}}

palm_api_key_plaintextstring

The PaLM API key provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see palm_api_key. You must provide an API key using one of the following fields: palm_api_key or palm_api_key_plaintext.

Example: Your API Key

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

Show child attributesHide child attributes

custom_provider_urlstring

This is a field to provide the URL of the custom provider API.

Example: https://custom-provider.com

bearer_token_authobject

This is a field to provide bearer token authentication for the custom provider API. You can only specify one authentication method.

api_key_authobject

This is a field to provide API key authentication for the custom provider API. You can only specify one authentication method.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

served_modelsarray of object

(Deprecated, use served_entities instead) The list of served models under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

Show child attributesHide child attributes

ai21labs_api_keystring

The <Databricks> secret key reference for an AI21 Labs API key. If you prefer to paste your API key directly, see ai21labs_api_key_plaintext. You must provide an API key using one of the following fields: ai21labs_api_key or ai21labs_api_key_plaintext.

Example: {{secrets/my_scope/my_ai21labs_api_key}}

ai21labs_api_key_plaintextstring

An AI21 Labs API key provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see ai21labs_api_key. You must provide an API key using one of the following fields: ai21labs_api_key or ai21labs_api_key_plaintext.

Example: Your API Key

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

Show child attributesHide child attributes

anthropic_api_keystring

The <Databricks> secret key reference for an Anthropic API key. If you prefer to paste your API key directly, see anthropic_api_key_plaintext. You must provide an API key using one of the following fields: anthropic_api_key or anthropic_api_key_plaintext.

Example: {{secrets/my_scope/my_anthropic_api_key}}

anthropic_api_key_plaintextstring

The Anthropic API key provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see anthropic_api_key. You must provide an API key using one of the following fields: anthropic_api_key or anthropic_api_key_plaintext.

Example: Your API Key

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

Show child attributesHide child attributes

aws_regionstring

The AWS region to use. Bedrock has to be enabled there.

Example: myAwsRegion

aws_access_key_idstring

The <Databricks> secret key reference for an AWS access key ID with permissions to interact with Bedrock services. If you prefer to paste your API key directly, see aws_access_key_id_plaintext. You must provide an API key using one of the following fields: aws_access_key_id or aws_access_key_id_plaintext.

Example: {{secrets/my_scope/my_aws_access_key_id}}

aws_secret_access_keystring

The <Databricks> secret key reference for an AWS secret access key paired with the access key ID, with permissions to interact with Bedrock services. If you prefer to paste your API key directly, see aws_secret_access_key_plaintext. You must provide an API key using one of the following fields: aws_secret_access_key or aws_secret_access_key_plaintext.

Example: {{secrets/my_scope/my_aws_secret_access_key}}

bedrock_providerstring

The underlying provider in Amazon Bedrock. Supported values (case insensitive) include: Anthropic, Cohere, AI21Labs, Amazon.

aws_access_key_id_plaintextstring

An AWS access key ID with permissions to interact with Bedrock services provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see aws_access_key_id. You must provide an API key using one of the following fields: aws_access_key_id or aws_access_key_id_plaintext.

Example: Your AWS Access Key ID

aws_secret_access_key_plaintextstring

An AWS secret access key paired with the access key ID, with permissions to interact with Bedrock services provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see aws_secret_access_key. You must provide an API key using one of the following fields: aws_secret_access_key or aws_secret_access_key_plaintext.

Example: Your AWS Secret Access Key

instance_profile_arnstring

ARN of the instance profile that the external model will use to access AWS resources. You must authenticate using an instance profile or access keys. If you prefer to authenticate using access keys, see aws_access_key_id, aws_access_key_id_plaintext, aws_secret_access_key and aws_secret_access_key_plaintext.

Example: arn:aws:iam::123456789012:instance-profile/my-instance-profile

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

Show child attributesHide child attributes

cohere_api_keystring

The <Databricks> secret key reference for a Cohere API key. If you prefer to paste your API key directly, see cohere_api_key_plaintext. You must provide an API key using one of the following fields: cohere_api_key or cohere_api_key_plaintext.

Example: {{secrets/my_scope/my_cohere_api_key}}

cohere_api_key_plaintextstring

The Cohere API key provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see cohere_api_key. You must provide an API key using one of the following fields: cohere_api_key or cohere_api_key_plaintext.

Example: Your API Key

cohere_api_basestring

This is an optional field to provide a customized base URL for the Cohere API. If left unspecified, the standard Cohere base URL is used.

Example: https://api.cohere.ai/v1

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

Show child attributesHide child attributes

private_keystring

The <Databricks> secret key reference for a private key for the service account which has access to the Google Cloud Vertex AI Service. See Best practices for managing service account keys. If you prefer to paste your API key directly, see private_key_plaintext. You must provide an API key using one of the following fields: private_key or private_key_plaintext

Example: {{secrets/my_scope/my_google_cloud_vertex_ai_api_key}}

project_idstring

This is the Google Cloud project id that the service account is associated with.

Example: your-project-id

regionstring

This is the region for the Google Cloud Vertex AI Service. See supported regions for more details. Some models are only available in specific regions.

Example: us-central1

private_key_plaintextstring

The private key for the service account which has access to the Google Cloud Vertex AI Service provided as a plaintext secret. See Best practices for managing service account keys. If you prefer to reference your key using Databricks Secrets, see private_key. You must provide an API key using one of the following fields: private_key or private_key_plaintext.

Example: Your API Key

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

Show child attributesHide child attributes

databricks_api_tokenstring

The <Databricks> secret key reference for a Databricks API token that corresponds to a user or service principal with Can Query access to the model serving endpoint pointed to by this external model. If you prefer to paste your API key directly, see databricks_api_token_plaintext. You must provide an API key using one of the following fields: databricks_api_token or databricks_api_token_plaintext.

Example: {{secrets/my_scope/my_databricks_api_token}}

databricks_workspace_urlstring

The URL of the <Databricks> workspace containing the model serving endpoint pointed to by this external model.

Example: https://my-databricks-workspace.com

databricks_api_token_plaintextstring

The Databricks API token that corresponds to a user or service principal with Can Query access to the model serving endpoint pointed to by this external model provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see databricks_api_token. You must provide an API key using one of the following fields: databricks_api_token or databricks_api_token_plaintext.

Example: Your Databricks API Token

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

Show child attributesHide child attributes

openai_api_keystring

The <Databricks> secret key reference for an OpenAI API key using the OpenAI or Azure service. If you prefer to paste your API key directly, see openai_api_key_plaintext. You must provide an API key using one of the following fields: openai_api_key or openai_api_key_plaintext.

Example: {{secrets/my_scope/my_openai_api_key}}

openai_api_typestring

This is an optional field to specify the type of OpenAI API to use. For Azure OpenAI, this field is required, and adjust this parameter to represent the preferred security access validation protocol. For access token validation, use azure. For authentication using Azure Active Directory (Azure AD) use, azuread.

Example: azure

openai_api_basestring

This is a field to provide a customized base URl for the OpenAI API. For Azure OpenAI, this field is required, and is the base URL for the Azure OpenAI API service provided by Azure. For other OpenAI API types, this field is optional, and if left unspecified, the standard OpenAI base URL is used.

Example: https://api.openai.com/v1

openai_api_versionstring

This is an optional field to specify the OpenAI API version. For Azure OpenAI, this field is required, and is the version of the Azure OpenAI service to utilize, specified by a date.

Example: 2023-11-01

openai_deployment_namestring

This field is only required for Azure OpenAI and is the name of the deployment resource for the Azure OpenAI service.

Example: my_deployment_resource

openai_organizationstring

This is an optional field to specify the organization in OpenAI or Azure OpenAI.

Example: Databricks

microsoft_entra_tenant_idstring

This field is only required for Azure AD OpenAI and is the Microsoft Entra Tenant ID.

Example: 12345678-abcd-1234-5678-12345678abcd

microsoft_entra_client_idstring

This field is only required for Azure AD OpenAI and is the Microsoft Entra Client ID.

Example: 12345678-abcd-1234-5678-12345678abcd

microsoft_entra_client_secretstring

The <Databricks> secret key reference for a client secret used for Microsoft Entra ID authentication. If you prefer to paste your client secret directly, see microsoft_entra_client_secret_plaintext. You must provide an API key using one of the following fields: microsoft_entra_client_secret or microsoft_entra_client_secret_plaintext.

Example: {{secrets/my_scope/my_microsoft_entra_client_secret}}

openai_api_key_plaintextstring

The OpenAI API key using the OpenAI or Azure service provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see openai_api_key. You must provide an API key using one of the following fields: openai_api_key or openai_api_key_plaintext.

Example: Your API Key

microsoft_entra_client_secret_plaintextstring

The client secret used for Microsoft Entra ID authentication provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see microsoft_entra_client_secret. You must provide an API key using one of the following fields: microsoft_entra_client_secret or microsoft_entra_client_secret_plaintext.

Example: Your Microsoft Entra Client Secret

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

Show child attributesHide child attributes

palm_api_keystring

The <Databricks> secret key reference for a PaLM API key. If you prefer to paste your API key directly, see palm_api_key_plaintext. You must provide an API key using one of the following fields: palm_api_key or palm_api_key_plaintext.

Example: {{secrets/my_scope/my_palm_api_key}}

palm_api_key_plaintextstring

The PaLM API key provided as a plaintext string. If you prefer to reference your key using Databricks Secrets, see palm_api_key. You must provide an API key using one of the following fields: palm_api_key or palm_api_key_plaintext.

Example: Your API Key

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

Show child attributesHide child attributes

custom_provider_urlstring

This is a field to provide the URL of the custom provider API.

Example: https://custom-provider.com

bearer_token_authobject

This is a field to provide bearer token authentication for the custom provider API. You can only specify one authentication method.

api_key_authobject

This is a field to provide API key authentication for the custom provider API. You can only specify one authentication method.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

traffic_configobject

The traffic configuration associated with the serving endpoint config.

Show child attributesHide child attributes

routesarray of object

The list of routes that define traffic to each served entity.

Show child attributesHide child attributes

served_model_namestring

The name of the served model this route configures traffic for.

Example: ads-model-3

traffic_percentageint32

The percentage of endpoint traffic to send to this route. It must be an integer between 0 and 100 inclusive.

Example: 100

served_entity_namestring

Example: ads-model-3

auto_capture_configobject

Configuration for legacy Inference Tables which automatically log requests and responses to Unity Catalog. Deprecated: please use AI Gateway inference tables instead. See https://docs.databricks.com/aws/en/ai-gateway/inference-tables.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog name if the inference table is already enabled.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. NOTE: On update, you cannot change the schema name if the inference table is already enabled.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.

Example: my-prefix-

stateobject

Show child attributesHide child attributes

payload_tableobject

Show child attributesHide child attributes

namestring

statusstring

status_messagestring

enabledboolean

Indicates whether the inference table is enabled.

Example: true

Response

namestring

The name of the serving endpoint.

Example: feed-ads

creatorstring

The email of the user who created the serving endpoint.

Example: alice@company.com

creation_timestampint64

The timestamp when the endpoint was created in Unix time.

last_updated_timestampint64

The timestamp when the endpoint was last updated by a user in Unix time.

stateobject

Information corresponding to the state of the serving endpoint.

Show child attributesHide child attributes

readystring

The state of an endpoint, indicating whether or not the endpoint is queryable. An endpoint is READY if all of the served entities in its active configuration are ready. If any of the actively served entities are in a non-ready state, the endpoint state will be NOT_READY.

Values: READY_STATE_UNSPECIFIED, READY, NOT_READY

config_updatestring

The state of an endpoint's config update. This informs the user if the pending_config is in progress, if the update failed, or if there is no update in progress. Note that if the endpoint's config_update state value is IN_PROGRESS, another update can not be made until the update completes or fails.

Values: CONFIG_UPDATE_STATE_UNSPECIFIED, NOT_UPDATING, IN_PROGRESS, UPDATE_FAILED, UPDATE_CANCELED

configobject

The config that is currently being served by the endpoint.

Show child attributesHide child attributes

config_versionint64

The config version that the serving endpoint is currently serving.

served_entitiesarray of object

The list of served entities under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

served_modelsarray of object

(Deprecated, use served_entities instead) The list of served models under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

traffic_configobject

The traffic configuration associated with the serving endpoint config.

Show child attributesHide child attributes

routesarray of object

The list of routes that define traffic to each served entity.

Show child attributesHide child attributes

served_model_namestring

The name of the served model this route configures traffic for.

Example: ads-model-3

traffic_percentageint32

The percentage of endpoint traffic to send to this route. It must be an integer between 0 and 100 inclusive.

Example: 100

served_entity_namestring

Example: ads-model-3

auto_capture_configobject

Configuration for legacy Inference Tables which automatically log requests and responses to Unity Catalog. Deprecated: please use AI Gateway inference tables instead. See https://docs.databricks.com/aws/en/ai-gateway/inference-tables.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog name if the inference table is already enabled.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. NOTE: On update, you cannot change the schema name if the inference table is already enabled.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.

Example: my-prefix-

stateobject

Show child attributesHide child attributes

payload_tableobject

enabledboolean

Indicates whether the inference table is enabled.

Example: true

pending_configobject

The config that the endpoint is attempting to update to.

Show child attributesHide child attributes

served_entitiesarray of object

The list of served entities belonging to the last issued update to the serving endpoint.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

served_modelsarray of object

(Deprecated, use served_entities instead) The list of served models belonging to the last issued update to the serving endpoint.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

traffic_configobject

The traffic config defining how invocations to the serving endpoint should be routed.

Show child attributesHide child attributes

routesarray of object

The list of routes that define traffic to each served entity.

Show child attributesHide child attributes

served_model_namestring

The name of the served model this route configures traffic for.

Example: ads-model-3

traffic_percentageint32

The percentage of endpoint traffic to send to this route. It must be an integer between 0 and 100 inclusive.

Example: 100

served_entity_namestring

Example: ads-model-3

config_versionint32

The config version that the serving endpoint is currently serving.

start_timeint64

The timestamp when the update to the pending config started.

auto_capture_configobject

Configuration for legacy Inference Tables which automatically log requests and responses to Unity Catalog. Deprecated: please use AI Gateway inference tables instead. See https://docs.databricks.com/aws/en/ai-gateway/inference-tables.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog name if the inference table is already enabled.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. NOTE: On update, you cannot change the schema name if the inference table is already enabled.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.

Example: my-prefix-

stateobject

Show child attributesHide child attributes

payload_tableobject

enabledboolean

Indicates whether the inference table is enabled.

Example: true

idstring

System-generated ID of the endpoint. This is used to refer to the endpoint in the Permissions API

Example: 88fd3f75a0d24b0380ddc40484d7a31b

permission_levelstring

The permission level of the principal making the request.

Values: CAN_MANAGE, CAN_QUERY, CAN_VIEW

Example: CAN_MANAGE

tagsarray of object

Tags attached to the serving endpoint.

Show child attributesHide child attributes

keystring

Key field for a serving endpoint tag.

Example: team

valuestring

Optional value field for a serving endpoint tag.

Example: data science

taskstring

The task type of the serving endpoint.

Example: model-serving-task

route_optimizedboolean

Boolean representing if route optimization has been enabled for the endpoint

Example: true

endpoint_urlstring

Endpoint invocation url if route optimization is enabled for endpoint

data_plane_infoobject

Information required to query DataPlane APIs.

Show child attributesHide child attributes

query_infoobject

Information required to query DataPlane API 'query' endpoint.

Show child attributesHide child attributes

endpoint_urlstring

The URL of the endpoint for this operation in the dataplane.

authorization_detailsstring

Authorization details as a string.

ai_gatewayobject

The AI Gateway configuration for the serving endpoint. NOTE: External model, provisioned throughput, and pay-per-token endpoints are fully supported; agent endpoints currently only support inference tables.

Show child attributesHide child attributes

usage_tracking_configobject

Configuration to enable usage tracking using system tables. These tables allow you to monitor operational usage on endpoints and their associated costs.

Show child attributesHide child attributes

enabledboolean

Whether to enable usage tracking.

Example: true

inference_table_configobject

Configuration for payload logging using inference tables. Use these tables to monitor and audit data being sent to and received from model APIs and to improve model quality.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the catalog name.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the schema name.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you have to disable inference table first in order to change the prefix name.

Example: my-prefix

enabledboolean

Indicates whether the inference table is enabled.

Example: true

rate_limitsarray of object

Configuration for rate limits which can be set to limit endpoint traffic.

Show child attributesHide child attributes

callsint64

Used to specify how many calls are allowed for a key within the renewal_period.

Example: 15

keystring

Key field for a rate limit. Currently, 'user', 'user_group, 'service_principal', and 'endpoint' are supported, with 'endpoint' being the default if not specified.

renewal_periodstring

Renewal period field for a rate limit. Currently, only 'minute' is supported.

principalstring

Principal field for a user, user group, or service principal to apply rate limiting to. Accepts a user email, group name, or service principal application ID.

Example: user@test.com

tokensint64

Used to specify how many tokens are allowed for a key within the renewal_period.

Example: 10000

guardrailsobject

Configuration for AI Guardrails to prevent unwanted data and unsafe data in requests and responses.

Show child attributesHide child attributes

inputobject

Configuration for input guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

outputobject

Configuration for output guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

fallback_configobject

Configuration for traffic fallback which auto fallbacks to other served entities if the request to a served entity fails with certain error codes, to increase availability.

Show child attributesHide child attributes

enabledboolean

Whether to enable traffic fallback. When a served entity in the serving endpoint returns specific error codes (e.g. 500), the request will automatically be round-robin attempted with other served entities in the same endpoint, following the order of served entity list, until a successful response is returned. If all attempts fail, return the last response with the error code.

Example: true

budget_policy_idstring

The budget policy associated with the endpoint.

email_notificationsobject

Email notification settings.

Show child attributesHide child attributes

on_update_successarray of string

A list of email addresses to be notified when an endpoint successfully updates its configuration or state.

Example: user.name@databricks.com

on_update_failurearray of string

A list of email addresses to be notified when an endpoint fails to update its configuration or state.

Example: user.name@databricks.com

descriptionstring

Description of the serving model

telemetry_configobject

Telemetry configuration for the endpoint, including inference-table payload logging.

Show child attributesHide child attributes

telemetry_profile_idstringRequired

The ID of an existing telemetry profile to apply to this endpoint. Provide this to reuse a telemetry profile that has already been created, instead of specifying table_names.

table_namesobjectRequired

The Unity Catalog tables to which endpoint telemetry (logs, traces, and metrics) is exported. Provide this to create a new telemetry profile for the endpoint from the given tables.

Show child attributesHide child attributes

logs_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported logs.

metrics_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported metrics.

traces_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported traces (spans).

annotations_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported annotations.

inference_table_configobject

Configuration for inference table payload logging, including sampling.

Show child attributesHide child attributes

sampling_fractiondouble

Fraction of requests sampled for payload logging, in the range [0.0, 1.0], where 1.0 logs all requests.

namestring

The full name of the inference table created for this endpoint.

enabled_telemetry_featuresarray of string

The telemetry signals to enable for this endpoint. If empty or omitted, all signals are enabled; otherwise only the listed signals are enabled.

Values: TELEMETRY_FEATURE_LOGS, TELEMETRY_FEATURE_TRACES, TELEMETRY_FEATURE_METRICS, TELEMETRY_FEATURE_INFERENCE_TABLE

View as Markdown

Request

External model endpointFeature Serving endpointModels in Unity Catalog endpointWorkspace model endpoint

{
  "served_entities": [
    {
      "external_model": {
        "name": "gpt-4",
        "openai_config": {
          "openai_api_key": "{{secrets/my_scope/my_openai_api_key}}"
        },
        "provider": "openai",
        "task": "llm/v1/chat"
      },
      "name": "openapi_chat"
    }
  ]
}

Response

External model endpointFeature Serving endpointModels in Unity Catalog endpointWorkspace model endpoint

{
  "config": {
    "config_version": 1,
    "served_entities": [
      {
        "creation_timestamp": 1699617587000,
        "creator": "user@email.com",
        "external_model": {
          "name": "gpt-4",
          "openai_config": {
            "openai_api_key": "{{secrets/my_scope/my_openai_api_key}}"
          },
          "provider": "openai",
          "task": "llm/v1/chat"
        },
        "name": "openai_chat",
        "state": {
          "deployment": "DEPLOYMENT_READY",
          "deployment_state_message": ""
        }
      }
    ],
    "traffic_config": {
      "routes": [
        {
          "served_model_name": "openai_chat",
          "traffic_percentage": 100
        }
      ]
    }
  },
  "creation_timestamp": 1699617587000,
  "creator": "user@email.com",
  "id": "69962db6b9db47c4a8a222d2ac79d7f8",
  "last_updated_timestamp": 1699617587000,
  "name": "openai_endpoint",
  "permission_level": "CAN_MANAGE",
  "route_optimized": false,
  "state": {
    "ready": "READY"
  }
}

Delete

DELETE /api/2.0/serving-endpoints/{name}

Delete a serving endpoint.

API scopes: model-serving

Parameters

namestringpath

View as Markdown

Response

{}

Create Provisioned Throughput Inference Endpoint

POST /api/2.0/serving-endpoints/pt

Create a new PT serving endpoint.

API scopes: model-serving

Request body

namestring

The name of the serving endpoint. This field is required and must be unique across a <Workspace>. An endpoint name can consist of alphanumeric characters, dashes, and underscores.

Example: feed-ads

configobject

The core config of the serving endpoint.

Show child attributesHide child attributes

served_entitiesarray of object

The list of served entities under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

provisioned_model_unitsint64

The number of model units to be provisioned.

Example: 1000

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

traffic_configobject

Show child attributesHide child attributes

routesarray of object

The list of routes that define traffic to each served entity.

Show child attributesHide child attributes

served_model_namestring

The name of the served model this route configures traffic for.

Example: ads-model-3

traffic_percentageint32

The percentage of endpoint traffic to send to this route. It must be an integer between 0 and 100 inclusive.

Example: 100

served_entity_namestring

Example: ads-model-3

tagsarray of object

Tags to be attached to the serving endpoint and automatically propagated to billing logs.

Show child attributesHide child attributes

keystring

Key field for a serving endpoint tag.

Example: team

valuestring

Optional value field for a serving endpoint tag.

Example: data science

ai_gatewayobject

The AI Gateway configuration for the serving endpoint.

Show child attributesHide child attributes

usage_tracking_configobject

Configuration to enable usage tracking using system tables. These tables allow you to monitor operational usage on endpoints and their associated costs.

Show child attributesHide child attributes

enabledboolean

Whether to enable usage tracking.

Example: true

inference_table_configobject

Configuration for payload logging using inference tables. Use these tables to monitor and audit data being sent to and received from model APIs and to improve model quality.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the catalog name.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the schema name.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you have to disable inference table first in order to change the prefix name.

Example: my-prefix

enabledboolean

Indicates whether the inference table is enabled.

Example: true

rate_limitsarray of object

Configuration for rate limits which can be set to limit endpoint traffic.

Show child attributesHide child attributes

callsint64

Used to specify how many calls are allowed for a key within the renewal_period.

Example: 15

keystring

Key field for a rate limit. Currently, 'user', 'user_group, 'service_principal', and 'endpoint' are supported, with 'endpoint' being the default if not specified.

renewal_periodstring

Renewal period field for a rate limit. Currently, only 'minute' is supported.

principalstring

Principal field for a user, user group, or service principal to apply rate limiting to. Accepts a user email, group name, or service principal application ID.

Example: user@test.com

tokensint64

Used to specify how many tokens are allowed for a key within the renewal_period.

Example: 10000

guardrailsobject

Configuration for AI Guardrails to prevent unwanted data and unsafe data in requests and responses.

Show child attributesHide child attributes

inputobject

Configuration for input guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

outputobject

Configuration for output guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

fallback_configobject

Configuration for traffic fallback which auto fallbacks to other served entities if the request to a served entity fails with certain error codes, to increase availability.

Show child attributesHide child attributes

enabledboolean

Whether to enable traffic fallback. When a served entity in the serving endpoint returns specific error codes (e.g. 500), the request will automatically be round-robin attempted with other served entities in the same endpoint, following the order of served entity list, until a successful response is returned. If all attempts fail, return the last response with the error code.

Example: true

budget_policy_idstring

The budget policy associated with the endpoint.

email_notificationsobject

Email notification settings.

Show child attributesHide child attributes

on_update_successarray of string

A list of email addresses to be notified when an endpoint successfully updates its configuration or state.

Example: user.name@databricks.com

on_update_failurearray of string

A list of email addresses to be notified when an endpoint fails to update its configuration or state.

Example: user.name@databricks.com

Response

namestring

The name of the serving endpoint.

Example: feed-ads

creatorstring

The email of the user who created the serving endpoint.

Example: alice@company.com

creation_timestampint64

The timestamp when the endpoint was created in Unix time.

last_updated_timestampint64

The timestamp when the endpoint was last updated by a user in Unix time.

stateobject

Information corresponding to the state of the serving endpoint.

Show child attributesHide child attributes

readystring

The state of an endpoint, indicating whether or not the endpoint is queryable. An endpoint is READY if all of the served entities in its active configuration are ready. If any of the actively served entities are in a non-ready state, the endpoint state will be NOT_READY.

Values: READY_STATE_UNSPECIFIED, READY, NOT_READY

config_updatestring

The state of an endpoint's config update. This informs the user if the pending_config is in progress, if the update failed, or if there is no update in progress. Note that if the endpoint's config_update state value is IN_PROGRESS, another update can not be made until the update completes or fails.

Values: CONFIG_UPDATE_STATE_UNSPECIFIED, NOT_UPDATING, IN_PROGRESS, UPDATE_FAILED, UPDATE_CANCELED

configobject

The config that is currently being served by the endpoint.

Show child attributesHide child attributes

config_versionint64

The config version that the serving endpoint is currently serving.

served_entitiesarray of object

The list of served entities under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

served_modelsarray of object

(Deprecated, use served_entities instead) The list of served models under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

traffic_configobject

The traffic configuration associated with the serving endpoint config.

Show child attributesHide child attributes

routesarray of object

The list of routes that define traffic to each served entity.

Show child attributesHide child attributes

served_model_namestring

The name of the served model this route configures traffic for.

Example: ads-model-3

traffic_percentageint32

The percentage of endpoint traffic to send to this route. It must be an integer between 0 and 100 inclusive.

Example: 100

served_entity_namestring

Example: ads-model-3

auto_capture_configobject

Configuration for legacy Inference Tables which automatically log requests and responses to Unity Catalog. Deprecated: please use AI Gateway inference tables instead. See https://docs.databricks.com/aws/en/ai-gateway/inference-tables.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog name if the inference table is already enabled.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. NOTE: On update, you cannot change the schema name if the inference table is already enabled.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.

Example: my-prefix-

stateobject

Show child attributesHide child attributes

payload_tableobject

enabledboolean

Indicates whether the inference table is enabled.

Example: true

pending_configobject

The config that the endpoint is attempting to update to.

Show child attributesHide child attributes

served_entitiesarray of object

The list of served entities belonging to the last issued update to the serving endpoint.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

served_modelsarray of object

(Deprecated, use served_entities instead) The list of served models belonging to the last issued update to the serving endpoint.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

traffic_configobject

The traffic config defining how invocations to the serving endpoint should be routed.

Show child attributesHide child attributes

routesarray of object

The list of routes that define traffic to each served entity.

Show child attributesHide child attributes

served_model_namestring

The name of the served model this route configures traffic for.

Example: ads-model-3

traffic_percentageint32

The percentage of endpoint traffic to send to this route. It must be an integer between 0 and 100 inclusive.

Example: 100

served_entity_namestring

Example: ads-model-3

config_versionint32

The config version that the serving endpoint is currently serving.

start_timeint64

The timestamp when the update to the pending config started.

auto_capture_configobject

Configuration for legacy Inference Tables which automatically log requests and responses to Unity Catalog. Deprecated: please use AI Gateway inference tables instead. See https://docs.databricks.com/aws/en/ai-gateway/inference-tables.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog name if the inference table is already enabled.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. NOTE: On update, you cannot change the schema name if the inference table is already enabled.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.

Example: my-prefix-

stateobject

Show child attributesHide child attributes

payload_tableobject

enabledboolean

Indicates whether the inference table is enabled.

Example: true

idstring

System-generated ID of the endpoint. This is used to refer to the endpoint in the Permissions API

Example: 88fd3f75a0d24b0380ddc40484d7a31b

permission_levelstring

The permission level of the principal making the request.

Values: CAN_MANAGE, CAN_QUERY, CAN_VIEW

Example: CAN_MANAGE

tagsarray of object

Tags attached to the serving endpoint.

Show child attributesHide child attributes

keystring

Key field for a serving endpoint tag.

Example: team

valuestring

Optional value field for a serving endpoint tag.

Example: data science

taskstring

The task type of the serving endpoint.

Example: model-serving-task

route_optimizedboolean

Boolean representing if route optimization has been enabled for the endpoint

Example: true

endpoint_urlstring

Endpoint invocation url if route optimization is enabled for endpoint

data_plane_infoobject

Information required to query DataPlane APIs.

Show child attributesHide child attributes

query_infoobject

Information required to query DataPlane API 'query' endpoint.

Show child attributesHide child attributes

endpoint_urlstring

The URL of the endpoint for this operation in the dataplane.

authorization_detailsstring

Authorization details as a string.

ai_gatewayobject

The AI Gateway configuration for the serving endpoint. NOTE: External model, provisioned throughput, and pay-per-token endpoints are fully supported; agent endpoints currently only support inference tables.

Show child attributesHide child attributes

usage_tracking_configobject

Configuration to enable usage tracking using system tables. These tables allow you to monitor operational usage on endpoints and their associated costs.

Show child attributesHide child attributes

enabledboolean

Whether to enable usage tracking.

Example: true

inference_table_configobject

Configuration for payload logging using inference tables. Use these tables to monitor and audit data being sent to and received from model APIs and to improve model quality.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the catalog name.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the schema name.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you have to disable inference table first in order to change the prefix name.

Example: my-prefix

enabledboolean

Indicates whether the inference table is enabled.

Example: true

rate_limitsarray of object

Configuration for rate limits which can be set to limit endpoint traffic.

Show child attributesHide child attributes

callsint64

Used to specify how many calls are allowed for a key within the renewal_period.

Example: 15

keystring

Key field for a rate limit. Currently, 'user', 'user_group, 'service_principal', and 'endpoint' are supported, with 'endpoint' being the default if not specified.

renewal_periodstring

Renewal period field for a rate limit. Currently, only 'minute' is supported.

principalstring

Principal field for a user, user group, or service principal to apply rate limiting to. Accepts a user email, group name, or service principal application ID.

Example: user@test.com

tokensint64

Used to specify how many tokens are allowed for a key within the renewal_period.

Example: 10000

guardrailsobject

Configuration for AI Guardrails to prevent unwanted data and unsafe data in requests and responses.

Show child attributesHide child attributes

inputobject

Configuration for input guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

outputobject

Configuration for output guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

fallback_configobject

Configuration for traffic fallback which auto fallbacks to other served entities if the request to a served entity fails with certain error codes, to increase availability.

Show child attributesHide child attributes

enabledboolean

Whether to enable traffic fallback. When a served entity in the serving endpoint returns specific error codes (e.g. 500), the request will automatically be round-robin attempted with other served entities in the same endpoint, following the order of served entity list, until a successful response is returned. If all attempts fail, return the last response with the error code.

Example: true

budget_policy_idstring

The budget policy associated with the endpoint.

email_notificationsobject

Email notification settings.

Show child attributesHide child attributes

on_update_successarray of string

A list of email addresses to be notified when an endpoint successfully updates its configuration or state.

Example: user.name@databricks.com

on_update_failurearray of string

A list of email addresses to be notified when an endpoint fails to update its configuration or state.

Example: user.name@databricks.com

descriptionstring

Description of the serving model

telemetry_configobject

Telemetry configuration for the endpoint, including inference-table payload logging.

Show child attributesHide child attributes

telemetry_profile_idstringRequired

The ID of an existing telemetry profile to apply to this endpoint. Provide this to reuse a telemetry profile that has already been created, instead of specifying table_names.

table_namesobjectRequired

The Unity Catalog tables to which endpoint telemetry (logs, traces, and metrics) is exported. Provide this to create a new telemetry profile for the endpoint from the given tables.

Show child attributesHide child attributes

logs_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported logs.

metrics_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported metrics.

traces_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported traces (spans).

annotations_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported annotations.

inference_table_configobject

Configuration for inference table payload logging, including sampling.

Show child attributesHide child attributes

sampling_fractiondouble

Fraction of requests sampled for payload logging, in the range [0.0, 1.0], where 1.0 logs all requests.

namestring

The full name of the inference table created for this endpoint.

enabled_telemetry_featuresarray of string

The telemetry signals to enable for this endpoint. If empty or omitted, all signals are enabled; otherwise only the listed signals are enabled.

Values: TELEMETRY_FEATURE_LOGS, TELEMETRY_FEATURE_TRACES, TELEMETRY_FEATURE_METRICS, TELEMETRY_FEATURE_INFERENCE_TABLE

View as Markdown

Request

{
  "name": "feed-ads",
  "config": {
    "served_entities": [
      {}
    ],
    "traffic_config": {}
  },
  "tags": [
    {
      "key": "team",
      "value": "data science"
    }
  ],
  "ai_gateway": {
    "usage_tracking_config": {},
    "inference_table_config": {},
    "rate_limits": [
      {}
    ],
    "guardrails": {},
    "fallback_config": {}
  },
  "budget_policy_id": "string",
  "email_notifications": {
    "on_update_success": "user.name@databricks.com",
    "on_update_failure": "user.name@databricks.com"
  }
}

Response

{
  "name": "feed-ads",
  "creator": "alice@company.com",
  "creation_timestamp": 0,
  "last_updated_timestamp": 0,
  "state": {
    "ready": "string",
    "config_update": "string"
  },
  "config": {
    "config_version": 0,
    "served_entities": [
      {}
    ],
    "served_models": [
      {}
    ],
    "traffic_config": {},
    "auto_capture_config": {}
  },
  "pending_config": {
    "served_entities": [
      {}
    ],
    "served_models": [
      {}
    ],
    "traffic_config": {},
    "config_version": 0,
    "start_time": 0,
    "auto_capture_config": {}
  },
  "id": "88fd3f75a0d24b0380ddc40484d7a31b",
  "permission_level": "CAN_MANAGE",
  "tags": [
    {
      "key": "team",
      "value": "data science"
    }
  ],
  "task": "model-serving-task",
  "route_optimized": true,
  "endpoint_url": "string",
  "data_plane_info": {
    "query_info": {}
  },
  "ai_gateway": {
    "usage_tracking_config": {},
    "inference_table_config": {},
    "rate_limits": [
      {}
    ],
    "guardrails": {},
    "fallback_config": {}
  },
  "budget_policy_id": "string",
  "email_notifications": {
    "on_update_success": "user.name@databricks.com",
    "on_update_failure": "user.name@databricks.com"
  },
  "description": "string",
  "telemetry_config": {
    "telemetry_profile_id": "string",
    "table_names": {},
    "inference_table_config": {},
    "enabled_telemetry_features": [
      "string"
    ]
  }
}

Get Export Endpoint Metrics

GET /api/2.0/serving-endpoints/{name}/metrics

Retrieves the metrics associated with the provided serving endpoint in either Prometheus or OpenMetrics exposition format.

API scopes: model-serving

Parameters

namestringpath

The name of the serving endpoint to retrieve metrics for. This field is required.

Constraints: [ 1 .. 63 ] characters

Response

contentsstring

View as Markdown

Response

{
  "contents": "string"
}

Get Inference Endpoint Schema

GET /api/2.0/serving-endpoints/{name}/openapi

Get the query schema of the serving endpoint in OpenAPI format. The schema contains information for the supported paths, input and output format and datatypes.

API scopes: model-serving

Parameters

namestringpath

The name of the serving endpoint that the served model belongs to. This field is required.

Example: feed-ads

Constraints: [ 1 .. 63 ] characters

Response

contentsstring

View as Markdown

Response

{
  "contents": "string"
}

Get Served Model Build Logs

GET /api/2.0/serving-endpoints/{name}/served-models/{served_model_name}/build-logs

Retrieves the build logs associated with the provided served model.

API scopes: model-serving

Parameters

namestringpath

The name of the serving endpoint that the served model belongs to. This field is required.

Constraints: [ 1 .. 63 ] characters

served_model_namestringpath

The name of the served model that build logs will be retrieved for. This field is required.

Constraints: [ 1 .. 63 ] characters

Response

logsstring

The logs associated with building the served entity's environment.

View as Markdown

Response

{
  "logs": "- conda-forge\ndependencies:\n- python=3.9.5\n- pip\u003c=21.2.4\n- pip:\n- mlflow\n- cloudpickle==2.2.0\nname: mlflow-env\nCollecting package metadata (repodata.json): ...working... done\nSolving environment: ...working... done\nPreparing transaction: ...working... done\nVerifying transaction: ...working... done\nExecuting transaction: ...working... done\nInstalling pip dependencies: ...working... done\n"
}

Get Served Model Logs

GET /api/2.0/serving-endpoints/{name}/served-models/{served_model_name}/logs

Retrieves the service logs associated with the provided served model.

API scopes: model-serving

Parameters

namestringpath

The name of the serving endpoint that the served model belongs to. This field is required.

Constraints: [ 1 .. 63 ] characters

served_model_namestringpath

The name of the served model that logs will be retrieved for. This field is required.

Constraints: [ 1 .. 63 ] characters

Response

logsstring

The most recent log lines of the model server processing invocation requests.

View as Markdown

Response

{
  "logs": "[hc4q8] [2023-01-10 19:12:58 +0000] [2] [INFO] Starting gunicorn 20.1.0\n[hc4q8] [2023-01-10 19:12:58 +0000] [2] [INFO] Listening at: http://0.0.0.0:8080 (2)\n[hc4q8] [2023-01-10 19:12:58 +0000] [2] [INFO] Using worker: sync\n[hc4q8] [2023-01-10 19:12:58 +0000] [3] [INFO] Booting worker with pid: 3\n[hc4q8] [2023-01-10 19:12:58 +0000] [4] [INFO] Booting worker with pid: 4\n[hc4q8] [2023-01-10 19:12:58 +0000] [5] [INFO] Booting worker with pid: 5\n[hc4q8] [2023-01-10 19:12:58 +0000] [6] [INFO] Booting worker with pid: 6\n"
}

Patch Inference Endpoint Telemetry Config

PATCH /api/2.0/serving-endpoints/{name}/telemetry-config

Updates the telemetry configuration of a serving endpoint.

API scopes: model-serving

Parameters

namestringpath

The name of the serving endpoint whose telemetry configuration is being updated. This field is required.

Constraints: [ 1 .. 63 ] characters

Request body

telemetry_configobject

The telemetry configuration to be applied to the serving endpoint. Can specify either a telemetry_profile_id to use an existing profile, or table_names to create a new profile with the specified Unity Catalog tables. If not provided, the telemetry configuration will be removed from the endpoint.

Show child attributesHide child attributes

telemetry_profile_idstringRequired

The ID of an existing telemetry profile to apply to this endpoint. Provide this to reuse a telemetry profile that has already been created, instead of specifying table_names.

table_namesobjectRequired

The Unity Catalog tables to which endpoint telemetry (logs, traces, and metrics) is exported. Provide this to create a new telemetry profile for the endpoint from the given tables.

Show child attributesHide child attributes

logs_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported logs.

metrics_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported metrics.

traces_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported traces (spans).

annotations_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported annotations.

inference_table_configobject

Configuration for inference table payload logging, including sampling.

Show child attributesHide child attributes

sampling_fractiondouble

Fraction of requests sampled for payload logging, in the range [0.0, 1.0], where 1.0 logs all requests.

namestring

The full name of the inference table created for this endpoint.

enabled_telemetry_featuresarray of string

The telemetry signals to enable for this endpoint. If empty or omitted, all signals are enabled; otherwise only the listed signals are enabled.

Values: TELEMETRY_FEATURE_LOGS, TELEMETRY_FEATURE_TRACES, TELEMETRY_FEATURE_METRICS, TELEMETRY_FEATURE_INFERENCE_TABLE

Response

namestring

The name of the serving endpoint.

Example: feed-ads

creatorstring

The email of the user who created the serving endpoint.

Example: alice@company.com

creation_timestampint64

The timestamp when the endpoint was created in Unix time.

last_updated_timestampint64

The timestamp when the endpoint was last updated by a user in Unix time.

stateobject

Information corresponding to the state of the serving endpoint.

Show child attributesHide child attributes

readystring

The state of an endpoint, indicating whether or not the endpoint is queryable. An endpoint is READY if all of the served entities in its active configuration are ready. If any of the actively served entities are in a non-ready state, the endpoint state will be NOT_READY.

Values: READY_STATE_UNSPECIFIED, READY, NOT_READY

config_updatestring

The state of an endpoint's config update. This informs the user if the pending_config is in progress, if the update failed, or if there is no update in progress. Note that if the endpoint's config_update state value is IN_PROGRESS, another update can not be made until the update completes or fails.

Values: CONFIG_UPDATE_STATE_UNSPECIFIED, NOT_UPDATING, IN_PROGRESS, UPDATE_FAILED, UPDATE_CANCELED

configobject

The config that is currently being served by the endpoint.

Show child attributesHide child attributes

config_versionint64

The config version that the serving endpoint is currently serving.

served_entitiesarray of object

The list of served entities under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

served_modelsarray of object

(Deprecated, use served_entities instead) The list of served models under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

traffic_configobject

The traffic configuration associated with the serving endpoint config.

Show child attributesHide child attributes

routesarray of object

The list of routes that define traffic to each served entity.

Show child attributesHide child attributes

served_model_namestring

The name of the served model this route configures traffic for.

Example: ads-model-3

traffic_percentageint32

The percentage of endpoint traffic to send to this route. It must be an integer between 0 and 100 inclusive.

Example: 100

served_entity_namestring

Example: ads-model-3

auto_capture_configobject

Configuration for legacy Inference Tables which automatically log requests and responses to Unity Catalog. Deprecated: please use AI Gateway inference tables instead. See https://docs.databricks.com/aws/en/ai-gateway/inference-tables.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog name if the inference table is already enabled.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. NOTE: On update, you cannot change the schema name if the inference table is already enabled.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.

Example: my-prefix-

stateobject

Show child attributesHide child attributes

payload_tableobject

enabledboolean

Indicates whether the inference table is enabled.

Example: true

pending_configobject

The config that the endpoint is attempting to update to.

Show child attributesHide child attributes

served_entitiesarray of object

The list of served entities belonging to the last issued update to the serving endpoint.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

served_modelsarray of object

(Deprecated, use served_entities instead) The list of served models belonging to the last issued update to the serving endpoint.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

traffic_configobject

The traffic config defining how invocations to the serving endpoint should be routed.

Show child attributesHide child attributes

routesarray of object

The list of routes that define traffic to each served entity.

Show child attributesHide child attributes

served_model_namestring

The name of the served model this route configures traffic for.

Example: ads-model-3

traffic_percentageint32

The percentage of endpoint traffic to send to this route. It must be an integer between 0 and 100 inclusive.

Example: 100

served_entity_namestring

Example: ads-model-3

config_versionint32

The config version that the serving endpoint is currently serving.

start_timeint64

The timestamp when the update to the pending config started.

auto_capture_configobject

Configuration for legacy Inference Tables which automatically log requests and responses to Unity Catalog. Deprecated: please use AI Gateway inference tables instead. See https://docs.databricks.com/aws/en/ai-gateway/inference-tables.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog name if the inference table is already enabled.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. NOTE: On update, you cannot change the schema name if the inference table is already enabled.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.

Example: my-prefix-

stateobject

Show child attributesHide child attributes

payload_tableobject

enabledboolean

Indicates whether the inference table is enabled.

Example: true

idstring

System-generated ID of the endpoint. This is used to refer to the endpoint in the Permissions API

Example: 88fd3f75a0d24b0380ddc40484d7a31b

permission_levelstring

The permission level of the principal making the request.

Values: CAN_MANAGE, CAN_QUERY, CAN_VIEW

Example: CAN_MANAGE

tagsarray of object

Tags attached to the serving endpoint.

Show child attributesHide child attributes

keystring

Key field for a serving endpoint tag.

Example: team

valuestring

Optional value field for a serving endpoint tag.

Example: data science

taskstring

The task type of the serving endpoint.

Example: model-serving-task

route_optimizedboolean

Boolean representing if route optimization has been enabled for the endpoint

Example: true

endpoint_urlstring

Endpoint invocation url if route optimization is enabled for endpoint

data_plane_infoobject

Information required to query DataPlane APIs.

Show child attributesHide child attributes

query_infoobject

Information required to query DataPlane API 'query' endpoint.

Show child attributesHide child attributes

endpoint_urlstring

The URL of the endpoint for this operation in the dataplane.

authorization_detailsstring

Authorization details as a string.

ai_gatewayobject

The AI Gateway configuration for the serving endpoint. NOTE: External model, provisioned throughput, and pay-per-token endpoints are fully supported; agent endpoints currently only support inference tables.

Show child attributesHide child attributes

usage_tracking_configobject

Configuration to enable usage tracking using system tables. These tables allow you to monitor operational usage on endpoints and their associated costs.

Show child attributesHide child attributes

enabledboolean

Whether to enable usage tracking.

Example: true

inference_table_configobject

Configuration for payload logging using inference tables. Use these tables to monitor and audit data being sent to and received from model APIs and to improve model quality.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the catalog name.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the schema name.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you have to disable inference table first in order to change the prefix name.

Example: my-prefix

enabledboolean

Indicates whether the inference table is enabled.

Example: true

rate_limitsarray of object

Configuration for rate limits which can be set to limit endpoint traffic.

Show child attributesHide child attributes

callsint64

Used to specify how many calls are allowed for a key within the renewal_period.

Example: 15

keystring

Key field for a rate limit. Currently, 'user', 'user_group, 'service_principal', and 'endpoint' are supported, with 'endpoint' being the default if not specified.

renewal_periodstring

Renewal period field for a rate limit. Currently, only 'minute' is supported.

principalstring

Principal field for a user, user group, or service principal to apply rate limiting to. Accepts a user email, group name, or service principal application ID.

Example: user@test.com

tokensint64

Used to specify how many tokens are allowed for a key within the renewal_period.

Example: 10000

guardrailsobject

Configuration for AI Guardrails to prevent unwanted data and unsafe data in requests and responses.

Show child attributesHide child attributes

inputobject

Configuration for input guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

outputobject

Configuration for output guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

fallback_configobject

Configuration for traffic fallback which auto fallbacks to other served entities if the request to a served entity fails with certain error codes, to increase availability.

Show child attributesHide child attributes

enabledboolean

Whether to enable traffic fallback. When a served entity in the serving endpoint returns specific error codes (e.g. 500), the request will automatically be round-robin attempted with other served entities in the same endpoint, following the order of served entity list, until a successful response is returned. If all attempts fail, return the last response with the error code.

Example: true

budget_policy_idstring

The budget policy associated with the endpoint.

email_notificationsobject

Email notification settings.

Show child attributesHide child attributes

on_update_successarray of string

A list of email addresses to be notified when an endpoint successfully updates its configuration or state.

Example: user.name@databricks.com

on_update_failurearray of string

A list of email addresses to be notified when an endpoint fails to update its configuration or state.

Example: user.name@databricks.com

descriptionstring

Description of the serving model

telemetry_configobject

Telemetry configuration for the endpoint, including inference-table payload logging.

Show child attributesHide child attributes

telemetry_profile_idstringRequired

The ID of an existing telemetry profile to apply to this endpoint. Provide this to reuse a telemetry profile that has already been created, instead of specifying table_names.

table_namesobjectRequired

The Unity Catalog tables to which endpoint telemetry (logs, traces, and metrics) is exported. Provide this to create a new telemetry profile for the endpoint from the given tables.

Show child attributesHide child attributes

logs_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported logs.

metrics_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported metrics.

traces_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported traces (spans).

annotations_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported annotations.

inference_table_configobject

Configuration for inference table payload logging, including sampling.

Show child attributesHide child attributes

sampling_fractiondouble

Fraction of requests sampled for payload logging, in the range [0.0, 1.0], where 1.0 logs all requests.

namestring

The full name of the inference table created for this endpoint.

enabled_telemetry_featuresarray of string

The telemetry signals to enable for this endpoint. If empty or omitted, all signals are enabled; otherwise only the listed signals are enabled.

Values: TELEMETRY_FEATURE_LOGS, TELEMETRY_FEATURE_TRACES, TELEMETRY_FEATURE_METRICS, TELEMETRY_FEATURE_INFERENCE_TABLE

View as Markdown

Request

{
  "telemetry_config": {
    "telemetry_profile_id": "string",
    "table_names": {},
    "inference_table_config": {},
    "enabled_telemetry_features": [
      "string"
    ]
  }
}

Response

{
  "name": "feed-ads",
  "creator": "alice@company.com",
  "creation_timestamp": 0,
  "last_updated_timestamp": 0,
  "state": {
    "ready": "string",
    "config_update": "string"
  },
  "config": {
    "config_version": 0,
    "served_entities": [
      {}
    ],
    "served_models": [
      {}
    ],
    "traffic_config": {},
    "auto_capture_config": {}
  },
  "pending_config": {
    "served_entities": [
      {}
    ],
    "served_models": [
      {}
    ],
    "traffic_config": {},
    "config_version": 0,
    "start_time": 0,
    "auto_capture_config": {}
  },
  "id": "88fd3f75a0d24b0380ddc40484d7a31b",
  "permission_level": "CAN_MANAGE",
  "tags": [
    {
      "key": "team",
      "value": "data science"
    }
  ],
  "task": "model-serving-task",
  "route_optimized": true,
  "endpoint_url": "string",
  "data_plane_info": {
    "query_info": {}
  },
  "ai_gateway": {
    "usage_tracking_config": {},
    "inference_table_config": {},
    "rate_limits": [
      {}
    ],
    "guardrails": {},
    "fallback_config": {}
  },
  "budget_policy_id": "string",
  "email_notifications": {
    "on_update_success": "user.name@databricks.com",
    "on_update_failure": "user.name@databricks.com"
  },
  "description": "string",
  "telemetry_config": {
    "telemetry_profile_id": "string",
    "table_names": {},
    "inference_table_config": {},
    "enabled_telemetry_features": [
      "string"
    ]
  }
}

Put Inference Endpoint Rate Limits

PUT /api/2.0/serving-endpoints/{name}/rate-limits

Deprecated: Please use AI Gateway to manage rate limits instead.

API scopes: model-serving

Parameters

namestringpath

The name of the serving endpoint whose rate limits are being updated. This field is required.

Constraints: [ 1 .. 63 ] characters

Request body

rate_limitsarray of object

The list of endpoint rate limits.

Show child attributesHide child attributes

callsint64

Used to specify how many calls are allowed for a key within the renewal_period.

Example: 15

keystring

Key field for a serving endpoint rate limit. Currently, only 'user' and 'endpoint' are supported, with 'endpoint' being the default if not specified.

renewal_periodstring

Renewal period field for a serving endpoint rate limit. Currently, only 'minute' is supported.

Response

rate_limitsarray of object

The list of endpoint rate limits.

Show child attributesHide child attributes

callsint64

Used to specify how many calls are allowed for a key within the renewal_period.

Example: 15

keystring

Key field for a serving endpoint rate limit. Currently, only 'user' and 'endpoint' are supported, with 'endpoint' being the default if not specified.

renewal_periodstring

Renewal period field for a serving endpoint rate limit. Currently, only 'minute' is supported.

View as Markdown

Request

{
  "rate_limits": [
    {
      "calls": 15,
      "key": "string",
      "renewal_period": "string"
    }
  ]
}

Response

{
  "rate_limits": [
    {
      "calls": 15,
      "key": "string",
      "renewal_period": "string"
    }
  ]
}

Put Provisioned Throughput Inference Endpoint Config

PUT /api/2.0/serving-endpoints/pt/{name}/config

Updates any combination of the pt endpoint's served entities, the compute configuration of those served entities, and the endpoint's traffic config. Updates are instantaneous and endpoint should be updated instantly

API scopes: model-serving

Parameters

namestringpath

The name of the pt endpoint to update. This field is required.

Example: feed-ads

Constraints: [ 1 .. 63 ] characters

Request body

configobject

Show child attributesHide child attributes

served_entitiesarray of object

The list of served entities under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

provisioned_model_unitsint64

The number of model units to be provisioned.

Example: 1000

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

traffic_configobject

Show child attributesHide child attributes

routesarray of object

The list of routes that define traffic to each served entity.

Show child attributesHide child attributes

served_model_namestring

The name of the served model this route configures traffic for.

Example: ads-model-3

traffic_percentageint32

The percentage of endpoint traffic to send to this route. It must be an integer between 0 and 100 inclusive.

Example: 100

served_entity_namestring

Example: ads-model-3

Response

namestring

The name of the serving endpoint.

Example: feed-ads

creatorstring

The email of the user who created the serving endpoint.

Example: alice@company.com

creation_timestampint64

The timestamp when the endpoint was created in Unix time.

last_updated_timestampint64

The timestamp when the endpoint was last updated by a user in Unix time.

stateobject

Information corresponding to the state of the serving endpoint.

Show child attributesHide child attributes

readystring

The state of an endpoint, indicating whether or not the endpoint is queryable. An endpoint is READY if all of the served entities in its active configuration are ready. If any of the actively served entities are in a non-ready state, the endpoint state will be NOT_READY.

Values: READY_STATE_UNSPECIFIED, READY, NOT_READY

config_updatestring

The state of an endpoint's config update. This informs the user if the pending_config is in progress, if the update failed, or if there is no update in progress. Note that if the endpoint's config_update state value is IN_PROGRESS, another update can not be made until the update completes or fails.

Values: CONFIG_UPDATE_STATE_UNSPECIFIED, NOT_UPDATING, IN_PROGRESS, UPDATE_FAILED, UPDATE_CANCELED

configobject

The config that is currently being served by the endpoint.

Show child attributesHide child attributes

config_versionint64

The config version that the serving endpoint is currently serving.

served_entitiesarray of object

The list of served entities under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

served_modelsarray of object

(Deprecated, use served_entities instead) The list of served models under the serving endpoint config.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

traffic_configobject

The traffic configuration associated with the serving endpoint config.

Show child attributesHide child attributes

routesarray of object

The list of routes that define traffic to each served entity.

Show child attributesHide child attributes

served_model_namestring

The name of the served model this route configures traffic for.

Example: ads-model-3

traffic_percentageint32

The percentage of endpoint traffic to send to this route. It must be an integer between 0 and 100 inclusive.

Example: 100

served_entity_namestring

Example: ads-model-3

auto_capture_configobject

Configuration for legacy Inference Tables which automatically log requests and responses to Unity Catalog. Deprecated: please use AI Gateway inference tables instead. See https://docs.databricks.com/aws/en/ai-gateway/inference-tables.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog name if the inference table is already enabled.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. NOTE: On update, you cannot change the schema name if the inference table is already enabled.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.

Example: my-prefix-

stateobject

Show child attributesHide child attributes

payload_tableobject

enabledboolean

Indicates whether the inference table is enabled.

Example: true

pending_configobject

The config that the endpoint is attempting to update to.

Show child attributesHide child attributes

served_entitiesarray of object

The list of served entities belonging to the last issued update to the serving endpoint.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

served_modelsarray of object

(Deprecated, use served_entities instead) The list of served models belonging to the last issued update to the serving endpoint.

Show child attributesHide child attributes

namestring

The name of a served entity. It must be unique across an endpoint. A served entity name can consist of alphanumeric characters, dashes, and underscores. If not specified for an external model, this field defaults to external_model.name, with '.' and ':' replaced with '-', and if not specified for other entities, it defaults to entity_name-entity_version.

Example: ads-model-3

external_modelobject

The external model to be served. NOTE: Only one of external_model and (entity_name, entity_version, workload_size, workload_type, and scale_to_zero_enabled) can be specified with the latter set being used for custom model serving for a <Databricks> registered model. For an existing endpoint with external_model, it cannot be updated to an endpoint without external_model. If the endpoint is created without external_model, users cannot update it to add external_model later. The task type of all external models within an endpoint must be the same.

Show child attributesHide child attributes

providerstring

The name of the provider for the external model. Currently, the supported providers are 'ai21labs', 'anthropic', 'amazon-bedrock', 'cohere', 'databricks-model-serving', 'google-cloud-vertex-ai', 'openai', 'palm', and 'custom'.

namestring

The name of the external model.

Example: gpt-4

taskstring

The task type of the external model.

Example: llm/v1/chat

ai21labs_configobjectRequired

AI21Labs Config. Only required if the provider is 'ai21labs'.

anthropic_configobjectRequired

Anthropic Config. Only required if the provider is 'anthropic'.

amazon_bedrock_configobjectRequired

Amazon Bedrock Config. Only required if the provider is 'amazon-bedrock'.

cohere_configobjectRequired

Cohere Config. Only required if the provider is 'cohere'.

google_cloud_vertex_ai_configobjectRequired

Google Cloud Vertex AI Config. Only required if the provider is 'google-cloud-vertex-ai'.

databricks_model_serving_configobjectRequired

Databricks Model Serving Config. Only required if the provider is 'databricks-model-serving'.

openai_configobjectRequired

OpenAI Config. Only required if the provider is 'openai'.

palm_configobjectRequired

PaLM Config. Only required if the provider is 'palm'.

custom_provider_configobjectRequired

Custom Provider Config. Only required if the provider is 'custom'.

entity_namestring

The name of the entity to be served. The entity may be a model in the Databricks Model Registry, a model in the Unity Catalog (UC), or a function of type FEATURE_SPEC in the UC. If it is a UC object, the full name of the object should be given in the form of catalog_name.schema_name.model_name.

Example: ads-model

entity_versionstring

Example: 3

min_provisioned_throughputint32

The minimum tokens per second that the endpoint can scale down to.

Example: 970

max_provisioned_throughputint32

The maximum tokens per second that the endpoint can scale up to.

Example: 1960

min_provisioned_concurrencyint32

The minimum provisioned concurrency that the endpoint can scale down to. Do not use if workload_size is specified.

Example: 8

max_provisioned_concurrencyint32

The maximum provisioned concurrency that the endpoint can scale up to. Do not use if workload_size is specified.

Example: 32

workload_sizestring

The workload size of the served entity. The workload size corresponds to a range of provisioned concurrency that the compute autoscales between. A single unit of provisioned concurrency can process one request at a time. Valid workload sizes are "Small" (4 - 4 provisioned concurrency), "Medium" (8 - 16 provisioned concurrency), and "Large" (16 - 64 provisioned concurrency). Additional custom workload sizes can also be used when available in the workspace. If scale-to-zero is enabled, the lower bound of the provisioned concurrency for each workload size is 0. Do not use if min_provisioned_concurrency and max_provisioned_concurrency are specified.

provisioned_model_unitsint64

The number of model units provisioned.

Example: 100

burst_scaling_enabledboolean

Whether burst scaling is enabled. When enabled (default), the endpoint can automatically scale up beyond provisioned capacity to handle traffic spikes. When disabled, the endpoint maintains fixed capacity at provisioned_model_units.

Example: true

scale_to_zero_enabledboolean

Whether the compute resources for the served entity should scale down to zero.

Example: false

model_namestring

model_versionstring

environment_varsobject

An object containing a set of optional, user-specified environment variable key-value pairs used for serving this entity. Note: this is an experimental feature and subject to change. Example entity environment variables that refer to <Databricks> secrets: {"OPENAI_API_KEY": "{{secrets/my_scope/my_key}}", "DATABRICKS_TOKEN": "{{secrets/my_scope2/my_key2}}"}

instance_profile_arnstring

ARN of the instance profile that the served entity uses to access AWS resources.

foundation_modelobject

Show child attributesHide child attributes

namestring

display_namestring

docsstring

descriptionstring

stateobject

Show child attributesHide child attributes

deploymentstring

Values: DEPLOYMENT_UNKNOWN, DEPLOYMENT_CREATING, DEPLOYMENT_RECOVERING, DEPLOYMENT_READY, DEPLOYMENT_FAILED, DEPLOYMENT_ABORTED, DEPLOYMENT_STOPPED

deployment_state_messagestring

creatorstring

creation_timestampint64

traffic_configobject

The traffic config defining how invocations to the serving endpoint should be routed.

Show child attributesHide child attributes

routesarray of object

The list of routes that define traffic to each served entity.

Show child attributesHide child attributes

served_model_namestring

The name of the served model this route configures traffic for.

Example: ads-model-3

traffic_percentageint32

The percentage of endpoint traffic to send to this route. It must be an integer between 0 and 100 inclusive.

Example: 100

served_entity_namestring

Example: ads-model-3

config_versionint32

The config version that the serving endpoint is currently serving.

start_timeint64

The timestamp when the update to the pending config started.

auto_capture_configobject

Configuration for legacy Inference Tables which automatically log requests and responses to Unity Catalog. Deprecated: please use AI Gateway inference tables instead. See https://docs.databricks.com/aws/en/ai-gateway/inference-tables.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. NOTE: On update, you cannot change the catalog name if the inference table is already enabled.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. NOTE: On update, you cannot change the schema name if the inference table is already enabled.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you cannot change the prefix name if the inference table is already enabled.

Example: my-prefix-

stateobject

Show child attributesHide child attributes

payload_tableobject

enabledboolean

Indicates whether the inference table is enabled.

Example: true

idstring

System-generated ID of the endpoint. This is used to refer to the endpoint in the Permissions API

Example: 88fd3f75a0d24b0380ddc40484d7a31b

permission_levelstring

The permission level of the principal making the request.

Values: CAN_MANAGE, CAN_QUERY, CAN_VIEW

Example: CAN_MANAGE

tagsarray of object

Tags attached to the serving endpoint.

Show child attributesHide child attributes

keystring

Key field for a serving endpoint tag.

Example: team

valuestring

Optional value field for a serving endpoint tag.

Example: data science

taskstring

The task type of the serving endpoint.

Example: model-serving-task

route_optimizedboolean

Boolean representing if route optimization has been enabled for the endpoint

Example: true

endpoint_urlstring

Endpoint invocation url if route optimization is enabled for endpoint

data_plane_infoobject

Information required to query DataPlane APIs.

Show child attributesHide child attributes

query_infoobject

Information required to query DataPlane API 'query' endpoint.

Show child attributesHide child attributes

endpoint_urlstring

The URL of the endpoint for this operation in the dataplane.

authorization_detailsstring

Authorization details as a string.

ai_gatewayobject

The AI Gateway configuration for the serving endpoint. NOTE: External model, provisioned throughput, and pay-per-token endpoints are fully supported; agent endpoints currently only support inference tables.

Show child attributesHide child attributes

usage_tracking_configobject

Configuration to enable usage tracking using system tables. These tables allow you to monitor operational usage on endpoints and their associated costs.

Show child attributesHide child attributes

enabledboolean

Whether to enable usage tracking.

Example: true

inference_table_configobject

Configuration for payload logging using inference tables. Use these tables to monitor and audit data being sent to and received from model APIs and to improve model quality.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the catalog name.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the schema name.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you have to disable inference table first in order to change the prefix name.

Example: my-prefix

enabledboolean

Indicates whether the inference table is enabled.

Example: true

rate_limitsarray of object

Configuration for rate limits which can be set to limit endpoint traffic.

Show child attributesHide child attributes

callsint64

Used to specify how many calls are allowed for a key within the renewal_period.

Example: 15

keystring

Key field for a rate limit. Currently, 'user', 'user_group, 'service_principal', and 'endpoint' are supported, with 'endpoint' being the default if not specified.

renewal_periodstring

Renewal period field for a rate limit. Currently, only 'minute' is supported.

principalstring

Principal field for a user, user group, or service principal to apply rate limiting to. Accepts a user email, group name, or service principal application ID.

Example: user@test.com

tokensint64

Used to specify how many tokens are allowed for a key within the renewal_period.

Example: 10000

guardrailsobject

Configuration for AI Guardrails to prevent unwanted data and unsafe data in requests and responses.

Show child attributesHide child attributes

inputobject

Configuration for input guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

outputobject

Configuration for output guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

fallback_configobject

Configuration for traffic fallback which auto fallbacks to other served entities if the request to a served entity fails with certain error codes, to increase availability.

Show child attributesHide child attributes

enabledboolean

Whether to enable traffic fallback. When a served entity in the serving endpoint returns specific error codes (e.g. 500), the request will automatically be round-robin attempted with other served entities in the same endpoint, following the order of served entity list, until a successful response is returned. If all attempts fail, return the last response with the error code.

Example: true

budget_policy_idstring

The budget policy associated with the endpoint.

email_notificationsobject

Email notification settings.

Show child attributesHide child attributes

on_update_successarray of string

A list of email addresses to be notified when an endpoint successfully updates its configuration or state.

Example: user.name@databricks.com

on_update_failurearray of string

A list of email addresses to be notified when an endpoint fails to update its configuration or state.

Example: user.name@databricks.com

descriptionstring

Description of the serving model

telemetry_configobject

Telemetry configuration for the endpoint, including inference-table payload logging.

Show child attributesHide child attributes

telemetry_profile_idstringRequired

The ID of an existing telemetry profile to apply to this endpoint. Provide this to reuse a telemetry profile that has already been created, instead of specifying table_names.

table_namesobjectRequired

The Unity Catalog tables to which endpoint telemetry (logs, traces, and metrics) is exported. Provide this to create a new telemetry profile for the endpoint from the given tables.

Show child attributesHide child attributes

logs_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported logs.

metrics_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported metrics.

traces_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported traces (spans).

annotations_tablestring

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported annotations.

inference_table_configobject

Configuration for inference table payload logging, including sampling.

Show child attributesHide child attributes

sampling_fractiondouble

Fraction of requests sampled for payload logging, in the range [0.0, 1.0], where 1.0 logs all requests.

namestring

The full name of the inference table created for this endpoint.

enabled_telemetry_featuresarray of string

The telemetry signals to enable for this endpoint. If empty or omitted, all signals are enabled; otherwise only the listed signals are enabled.

Values: TELEMETRY_FEATURE_LOGS, TELEMETRY_FEATURE_TRACES, TELEMETRY_FEATURE_METRICS, TELEMETRY_FEATURE_INFERENCE_TABLE

View as Markdown

Request

{
  "config": {
    "served_entities": [
      {}
    ],
    "traffic_config": {}
  }
}

Response

{
  "name": "feed-ads",
  "creator": "alice@company.com",
  "creation_timestamp": 0,
  "last_updated_timestamp": 0,
  "state": {
    "ready": "string",
    "config_update": "string"
  },
  "config": {
    "config_version": 0,
    "served_entities": [
      {}
    ],
    "served_models": [
      {}
    ],
    "traffic_config": {},
    "auto_capture_config": {}
  },
  "pending_config": {
    "served_entities": [
      {}
    ],
    "served_models": [
      {}
    ],
    "traffic_config": {},
    "config_version": 0,
    "start_time": 0,
    "auto_capture_config": {}
  },
  "id": "88fd3f75a0d24b0380ddc40484d7a31b",
  "permission_level": "CAN_MANAGE",
  "tags": [
    {
      "key": "team",
      "value": "data science"
    }
  ],
  "task": "model-serving-task",
  "route_optimized": true,
  "endpoint_url": "string",
  "data_plane_info": {
    "query_info": {}
  },
  "ai_gateway": {
    "usage_tracking_config": {},
    "inference_table_config": {},
    "rate_limits": [
      {}
    ],
    "guardrails": {},
    "fallback_config": {}
  },
  "budget_policy_id": "string",
  "email_notifications": {
    "on_update_success": "user.name@databricks.com",
    "on_update_failure": "user.name@databricks.com"
  },
  "description": "string",
  "telemetry_config": {
    "telemetry_profile_id": "string",
    "table_names": {},
    "inference_table_config": {},
    "enabled_telemetry_features": [
      "string"
    ]
  }
}

Update Inference Endpoint Notifications

PATCH /api/2.0/serving-endpoints/{name}/notifications

Updates the email and webhook notification settings for an endpoint.

API scopes: model-serving

Parameters

namestringpath

The name of the serving endpoint whose notifications are being updated. This field is required.

Example: my-endpoint

Constraints: [ 1 .. 63 ] characters

Request body

email_notificationsobject

The email notification settings to update. Specify email addresses to notify when endpoint state changes occur.

Show child attributesHide child attributes

on_update_successarray of string

A list of email addresses to be notified when an endpoint successfully updates its configuration or state.

Example: user.name@databricks.com

on_update_failurearray of string

A list of email addresses to be notified when an endpoint fails to update its configuration or state.

Example: user.name@databricks.com

Response

namestring

email_notificationsobject

Show child attributesHide child attributes

on_update_successarray of string

A list of email addresses to be notified when an endpoint successfully updates its configuration or state.

Example: user.name@databricks.com

on_update_failurearray of string

A list of email addresses to be notified when an endpoint fails to update its configuration or state.

Example: user.name@databricks.com

View as Markdown

Request

{
  "email_notifications": {
    "on_update_success": "user.name@databricks.com",
    "on_update_failure": "user.name@databricks.com"
  }
}

Response

{
  "name": "string",
  "email_notifications": {
    "on_update_success": "user.name@databricks.com",
    "on_update_failure": "user.name@databricks.com"
  }
}

Put Inference Endpoint Ai Gateway

PUT /api/2.0/serving-endpoints/{name}/ai-gateway

Used to update the AI Gateway of a serving endpoint. NOTE: External model, provisioned throughput, and pay-per-token endpoints are fully supported; agent endpoints currently only support inference tables.

API scopes: model-serving

Parameters

namestringpath

The name of the serving endpoint whose AI Gateway is being updated. This field is required.

Example: feed-ads

Constraints: [ 1 .. 63 ] characters

Request body

usage_tracking_configobject

Configuration to enable usage tracking using system tables. These tables allow you to monitor operational usage on endpoints and their associated costs.

Show child attributesHide child attributes

enabledboolean

Whether to enable usage tracking.

Example: true

inference_table_configobject

Configuration for payload logging using inference tables. Use these tables to monitor and audit data being sent to and received from model APIs and to improve model quality.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the catalog name.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the schema name.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you have to disable inference table first in order to change the prefix name.

Example: my-prefix

enabledboolean

Indicates whether the inference table is enabled.

Example: true

rate_limitsarray of object

Configuration for rate limits which can be set to limit endpoint traffic.

Show child attributesHide child attributes

callsint64

Used to specify how many calls are allowed for a key within the renewal_period.

Example: 15

keystring

Key field for a rate limit. Currently, 'user', 'user_group, 'service_principal', and 'endpoint' are supported, with 'endpoint' being the default if not specified.

renewal_periodstring

Renewal period field for a rate limit. Currently, only 'minute' is supported.

principalstring

Principal field for a user, user group, or service principal to apply rate limiting to. Accepts a user email, group name, or service principal application ID.

Example: user@test.com

tokensint64

Used to specify how many tokens are allowed for a key within the renewal_period.

Example: 10000

guardrailsobject

Configuration for AI Guardrails to prevent unwanted data and unsafe data in requests and responses.

Show child attributesHide child attributes

inputobject

Configuration for input guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

Show child attributesHide child attributes

behaviorstring

Configuration for input guardrail filters.

Values: BEHAVIOR_UNSPECIFIED, NONE, BLOCK, MASK

Example: BLOCK

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

outputobject

Configuration for output guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

Show child attributesHide child attributes

behaviorstring

Configuration for input guardrail filters.

Values: BEHAVIOR_UNSPECIFIED, NONE, BLOCK, MASK

Example: BLOCK

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

fallback_configobject

Configuration for traffic fallback which auto fallbacks to other served entities if the request to a served entity fails with certain error codes, to increase availability.

Show child attributesHide child attributes

enabledboolean

Whether to enable traffic fallback. When a served entity in the serving endpoint returns specific error codes (e.g. 500), the request will automatically be round-robin attempted with other served entities in the same endpoint, following the order of served entity list, until a successful response is returned. If all attempts fail, return the last response with the error code.

Example: true

Response

usage_tracking_configobject

Configuration to enable usage tracking using system tables. These tables allow you to monitor operational usage on endpoints and their associated costs.

Show child attributesHide child attributes

enabledboolean

Whether to enable usage tracking.

Example: true

inference_table_configobject

Configuration for payload logging using inference tables. Use these tables to monitor and audit data being sent to and received from model APIs and to improve model quality.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the catalog name.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the schema name.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you have to disable inference table first in order to change the prefix name.

Example: my-prefix

enabledboolean

Indicates whether the inference table is enabled.

Example: true

rate_limitsarray of object

Configuration for rate limits which can be set to limit endpoint traffic.

Show child attributesHide child attributes

callsint64

Used to specify how many calls are allowed for a key within the renewal_period.

Example: 15

keystring

Key field for a rate limit. Currently, 'user', 'user_group, 'service_principal', and 'endpoint' are supported, with 'endpoint' being the default if not specified.

renewal_periodstring

Renewal period field for a rate limit. Currently, only 'minute' is supported.

principalstring

Principal field for a user, user group, or service principal to apply rate limiting to. Accepts a user email, group name, or service principal application ID.

Example: user@test.com

tokensint64

Used to specify how many tokens are allowed for a key within the renewal_period.

Example: 10000

guardrailsobject

Configuration for AI Guardrails to prevent unwanted data and unsafe data in requests and responses.

Show child attributesHide child attributes

inputobject

Configuration for input guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

Show child attributesHide child attributes

behaviorstring

Configuration for input guardrail filters.

Values: BEHAVIOR_UNSPECIFIED, NONE, BLOCK, MASK

Example: BLOCK

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

outputobject

Configuration for output guardrail filters.

Show child attributesHide child attributes

safetyboolean

Indicates whether the safety filter is enabled.

Example: true

piiobject

Configuration for guardrail PII filter.

Show child attributesHide child attributes

behaviorstring

Configuration for input guardrail filters.

Values: BEHAVIOR_UNSPECIFIED, NONE, BLOCK, MASK

Example: BLOCK

valid_topicsarray of string

The list of allowed topics. Given a chat request, this guardrail flags the request if its topic is not in the allowed topics.

invalid_keywordsarray of string

List of invalid keywords. AI guardrail uses keyword or string matching to decide if the keyword exists in the request or response content.

fallback_configobject

Configuration for traffic fallback which auto fallbacks to other served entities if the request to a served entity fails with certain error codes, to increase availability.

Show child attributesHide child attributes

enabledboolean

Whether to enable traffic fallback. When a served entity in the serving endpoint returns specific error codes (e.g. 500), the request will automatically be round-robin attempted with other served entities in the same endpoint, following the order of served entity list, until a successful response is returned. If all attempts fail, return the last response with the error code.

Example: true

View as Markdown

Request

{
  "guardrails": {
    "input": {
      "invalid_keywords": [
        "keyword1"
      ],
      "pii": {
        "behavior": "BLOCK"
      },
      "safety": true,
      "valid_topics": [
        "topic1"
      ]
    },
    "output": {
      "invalid_keywords": [
        "keyword1"
      ],
      "pii": {
        "behavior": "BLOCK"
      },
      "safety": true,
      "valid_topics": [
        "topic1"
      ]
    }
  },
  "inference_table_config": {
    "catalog_name": "my-catalog",
    "enabled": true,
    "schema_name": "my-schema",
    "table_name_prefix": "my-prefix"
  },
  "rate_limits": [
    {
      "calls": 15,
      "key": "user",
      "renewal_period": "minute"
    }
  ],
  "usage_tracking_config": {
    "enabled": true
  }
}

Response

{
  "guardrails": {
    "input": {
      "invalid_keywords": [
        "keyword1"
      ],
      "pii": {
        "behavior": "BLOCK"
      },
      "safety": true,
      "valid_topics": [
        "topic1"
      ]
    },
    "output": {
      "invalid_keywords": [
        "keyword1"
      ],
      "pii": {
        "behavior": "BLOCK"
      },
      "safety": true,
      "valid_topics": [
        "topic1"
      ]
    }
  },
  "inference_table_config": {
    "catalog_name": "my-catalog",
    "enabled": true,
    "schema_name": "my-schema",
    "table_name_prefix": "my-prefix"
  },
  "rate_limits": [
    {
      "calls": 15,
      "key": "user",
      "renewal_period": "minute"
    }
  ],
  "usage_tracking_config": {
    "enabled": true
  }
}

Patch Inference Endpoint Tags

PATCH /api/2.0/serving-endpoints/{name}/tags

Used to batch add and delete tags from a serving endpoint with a single API call.

API scopes: model-serving

Parameters

namestringpath

The name of the serving endpoint who's tags to patch. This field is required.

Example: feed-ads

Constraints: [ 1 .. 63 ] characters

Request body

add_tagsarray of object

List of endpoint tags to add

Show child attributesHide child attributes

keystring

Key field for a serving endpoint tag.

Example: team

valuestring

Optional value field for a serving endpoint tag.

Example: data science

delete_tagsarray of string

List of tag keys to delete

Response

tagsarray of object

Show child attributesHide child attributes

keystring

Key field for a serving endpoint tag.

Example: team

valuestring

Optional value field for a serving endpoint tag.

Example: data science

View as Markdown

Request

{
  "add_tags": [
    {
      "key": "team",
      "value": "data science"
    }
  ],
  "delete_tags": [
    "string"
  ]
}

Response

[
  {
    "key": "team",
    "value": "data science"
  }
]