Serving Endpoint Summary

View as Markdown

ListGET /api/2.0/serving-endpoints

List GA

GET /api/2.0/serving-endpoints

Get all serving endpoints.

API scopes: model-serving

Response

endpointsarray of object

The list of endpoints.

Show child attributesHide child attributes

namestring

The name of the serving endpoint.

Example: feed-ads

creatorstring

The email of the user who created the serving endpoint.

Example: alice@company.com

creation_timestampint64

The timestamp when the endpoint was created in Unix time.

last_updated_timestampint64

The timestamp when the endpoint was last updated by a user in Unix time.

stateobject

Information corresponding to the state of the serving endpoint.

Show child attributesHide child attributes

readystring

The state of an endpoint, indicating whether or not the endpoint is queryable. An endpoint is READY if all of the served entities in its active configuration are ready. If any of the actively served entities are in a non-ready state, the endpoint state will be NOT_READY.

Values: READY_STATE_UNSPECIFIED, READY, NOT_READY

config_updatestring

The state of an endpoint's config update. This informs the user if the pending_config is in progress, if the update failed, or if there is no update in progress. Note that if the endpoint's config_update state value is IN_PROGRESS, another update can not be made until the update completes or fails.

Values: CONFIG_UPDATE_STATE_UNSPECIFIED, NOT_UPDATING, IN_PROGRESS, UPDATE_FAILED, UPDATE_CANCELED

configobject

The config that is currently being served by the endpoint.

Show child attributesHide child attributes

served_entitiesarray of object

The list of served entities under the serving endpoint config.

Show child attributesHide child attributes

namestring

model_namestring

Only one of model_name and entity_name should be populated

entity_namestring

model_versionstring

Only one of model_version and entity_version should be populated

entity_versionstring

external_modelobject

foundation_modelobjectPublic Preview

served_modelsarray of object

(Deprecated, use served_entities instead) The list of served models under the serving endpoint config.

Show child attributesHide child attributes

namestring

model_namestring

Only one of model_name and entity_name should be populated

entity_namestring

model_versionstring

Only one of model_version and entity_version should be populated

entity_versionstring

external_modelobject

foundation_modelobjectPublic Preview

tagsarray of object

Tags attached to the serving endpoint.

Show child attributesHide child attributes

keystring

Key field for a serving endpoint tag.

Example: team

valuestring

Optional value field for a serving endpoint tag.

Example: data science

idstring

System-generated ID of the endpoint, included to be used by the Permissions API.

Example: 88fd3f75a0d24b0380ddc40484d7a31b

taskstring

The task type of the serving endpoint.

Example: model-serving-task

ai_gatewayobject

The AI Gateway configuration for the serving endpoint. NOTE: External model, provisioned throughput, and pay-per-token endpoints are fully supported; agent endpoints currently only support inference tables.

Show child attributesHide child attributes

usage_tracking_configobject

Configuration to enable usage tracking using system tables. These tables allow you to monitor operational usage on endpoints and their associated costs.

Show child attributesHide child attributes

enabledboolean

Whether to enable usage tracking.

Example: true

inference_table_configobject

Configuration for payload logging using inference tables. Use these tables to monitor and audit data being sent to and received from model APIs and to improve model quality.

Show child attributesHide child attributes

catalog_namestring

The name of the catalog in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the catalog name.

Example: my-catalog

schema_namestring

The name of the schema in Unity Catalog. Required when enabling inference tables. NOTE: On update, you have to disable inference table first in order to change the schema name.

Example: my-schema

table_name_prefixstring

The prefix of the table in Unity Catalog. NOTE: On update, you have to disable inference table first in order to change the prefix name.

Example: my-prefix

enabledboolean

Indicates whether the inference table is enabled.

Example: true

rate_limitsarray of object

Configuration for rate limits which can be set to limit endpoint traffic.

Show child attributesHide child attributes

callsint64

Used to specify how many calls are allowed for a key within the renewal_period.

Example: 15

keystring

Key field for a rate limit. Currently, 'user', 'user_group, 'service_principal', and 'endpoint' are supported, with 'endpoint' being the default if not specified.

renewal_periodstring

Renewal period field for a rate limit. Currently, only 'minute' is supported.

principalstring

Principal field for a user, user group, or service principal to apply rate limiting to. Accepts a user email, group name, or service principal application ID.

Example: user@test.com

tokensint64

Used to specify how many tokens are allowed for a key within the renewal_period.

Example: 10000

guardrailsobjectPublic Preview

Configuration for AI Guardrails to prevent unwanted data and unsafe data in requests and responses.

Show child attributesHide child attributes

inputobjectPublic Preview

Configuration for input guardrail filters.

outputobjectPublic Preview

Configuration for output guardrail filters.

fallback_configobject

Configuration for traffic fallback which auto fallbacks to other served entities if the request to a served entity fails with certain error codes, to increase availability.

Show child attributesHide child attributes

enabledboolean

Whether to enable traffic fallback. When a served entity in the serving endpoint returns specific error codes (e.g. 500), the request will automatically be round-robin attempted with other served entities in the same endpoint, following the order of served entity list, until a successful response is returned. If all attempts fail, return the last response with the error code.

Example: true

budget_policy_idstring

The budget policy associated with the endpoint.

descriptionstring

Description of the endpoint

usage_policy_idstring

The usage policy associated with serving endpoint.

telemetry_configobjectPublic Preview

Telemetry configuration for the endpoint, including inference-table payload logging.

Show child attributesHide child attributes

telemetry_profile_idstringRequiredPublic Preview

The ID of an existing telemetry profile to apply to this endpoint. Provide this to reuse a telemetry profile that has already been created, instead of specifying table_names.

table_namesobjectRequiredPublic Preview

The Unity Catalog tables to which endpoint telemetry (logs, traces, and metrics) is exported. Provide this to create a new telemetry profile for the endpoint from the given tables.

Show child attributesHide child attributes

logs_tablestringPublic Preview

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported logs.

metrics_tablestringPublic Preview

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported metrics.

traces_tablestringPublic Preview

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported traces (spans).

annotations_tablestringPublic Preview

The full three-level Unity Catalog name (catalog.schema.table) of the table that receives exported annotations.

inference_table_configobjectPublic Preview

Configuration for inference table payload logging, including sampling.

Show child attributesHide child attributes

sampling_fractiondoublePublic Preview

Fraction of requests sampled for payload logging, in the range [0.0, 1.0], where 1.0 logs all requests.

namestringPublic Preview

The full name of the inference table created for this endpoint.

enabled_telemetry_featuresarray of stringPublic Preview

The telemetry signals to enable for this endpoint. If empty or omitted, all signals are enabled; otherwise only the listed signals are enabled.

Values: TELEMETRY_FEATURE_LOGS, TELEMETRY_FEATURE_TRACES, TELEMETRY_FEATURE_METRICS, TELEMETRY_FEATURE_INFERENCE_TABLE

View as Markdown

Response

{
  "endpoints": [
    {
      "ai_gateway": {
        "guardrails": {
          "input": {
            "invalid_keywords": [
              "keyword1"
            ],
            "pii": {
              "behavior": "BLOCK"
            },
            "safety": true,
            "valid_topics": [
              "topic1"
            ]
          },
          "output": {
            "invalid_keywords": [
              "keyword1"
            ],
            "pii": {
              "behavior": "BLOCK"
            },
            "safety": true,
            "valid_topics": [
              "topic1"
            ]
          }
        },
        "inference_table_config": {
          "catalog_name": "my-catalog",
          "enabled": true,
          "schema_name": "my-schema",
          "table_name_prefix": "my-prefix"
        },
        "rate_limits": [
          {
            "calls": 15,
            "key": "user",
            "renewal_period": "minute"
          }
        ],
        "usage_tracking_config": {
          "enabled": true
        }
      },
      "config": {
        "served_entities": [
          {
            "entity_name": "ads-model",
            "entity_version": "3",
            "external_model": {
              "ai21labs_config": {
                "ai21labs_api_key": "{{secrets/my_scope/my_ai21labs_api_key}}",
                "ai21labs_api_key_plaintext": "Your API Key"
              },
              "amazon_bedrock_config": {
                "aws_access_key_id": "{{secrets/my_scope/my_aws_access_key_id}}",
                "aws_access_key_id_plaintext": "Your AWS Access Key ID",
                "aws_region": "myAwsRegion",
                "aws_secret_access_key": "{{secrets/my_scope/my_aws_secret_access_key}}",
                "aws_secret_access_key_plaintext": "Your AWS Secret Access Key",
                "bedrock_provider": "anthropic"
              },
              "anthropic_config": {
                "anthropic_api_key": "{{secrets/my_scope/my_anthropic_api_key}}",
                "anthropic_api_key_plaintext": "Your API Key"
              },
              "cohere_config": {
                "cohere_api_base": "https://api.cohere.ai/v1",
                "cohere_api_key": "{{secrets/my_scope/my_cohere_api_key}}",
                "cohere_api_key_plaintext": "Your API Key"
              },
              "databricks_model_serving_config": {
                "databricks_api_token": "{{secrets/my_scope/my_databricks_api_token}}",
                "databricks_api_token_plaintext": "Your Databricks API Token",
                "databricks_workspace_url": "https://my-databricks-workspace.com"
              },
              "google_cloud_vertex_ai_config": {
                "private_key": "{{secrets/my_scope/my_google_cloud_vertex_ai_api_key}}",
                "private_key_plaintext": "Your API Key",
                "project_id": "your-project-id",
                "region": "us-central1"
              },
              "name": "gpt-4",
              "openai_config": {
                "microsoft_entra_client_id": "12345678-abcd-1234-5678-12345678abcd",
                "microsoft_entra_client_secret": "{{secrets/my_scope/my_microsoft_entra_client_secret}}",
                "microsoft_entra_client_secret_plaintext": "Your Microsoft Entra Client Secret",
                "microsoft_entra_tenant_id": "12345678-abcd-1234-5678-12345678abcd",
                "openai_api_base": "https://api.openai.com/v1",
                "openai_api_key": "{{secrets/my_scope/my_openai_api_key}}",
                "openai_api_key_plaintext": "Your API Key",
                "openai_api_type": "azure",
                "openai_api_version": "2023-11-01",
                "openai_deployment_name": "my_deployment_resource",
                "openai_organization": "Databricks"
              },
              "palm_config": {
                "palm_api_key": "{{secrets/my_scope/my_palm_api_key}}",
                "palm_api_key_plaintext": "Your API Key"
              },
              "provider": "openai",
              "task": "llm/v1/chat"
            },
            "foundation_model": {
              "description": "Llama 3.1 is a state-of-the-art 70B parameter dense language model trained and released by Meta. The model supports a context length of 128K tokens. The model is optimized for multilingual dialogue use cases and aligned with human preferences for helpfulness and safety. It is not intended for use in languages other than English. Meta Llama 3.1 is licensed under the Meta Llama 3.1 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. Customers are responsible for ensuring compliance with applicable model licenses.",
              "display_name": "Meta Llama 3.1 70B Instruct",
              "docs": "https://docs.databricks.com/machine-learning/foundation-models/supported-models.html#meta-llama-31-70b-instruct",
              "name": "databricks-meta-llama-3-1-70b-instruct"
            },
            "name": "ads-model-3"
          }
        ],
        "served_models": [
          {
            "model_name": "ads-model",
            "model_version": "3",
            "name": "ads-model-3"
          }
        ]
      },
      "creation_timestamp": 0,
      "creator": "alice@company.com",
      "id": "88fd3f75a0d24b0380ddc40484d7a31b",
      "last_updated_timestamp": 0,
      "name": "feed-ads",
      "state": {
        "config_update": "NOT_UPDATING",
        "ready": "READY"
      },
      "tags": [
        {
          "key": "team",
          "value": "data science"
        }
      ],
      "task": "model-serving-task"
    }
  ]
}