Send HTTP requests to external systems

Preview

This feature is in Private Preview. To try it, reach out to your Databricks contact.

This article describes how to create an HTTP connection so you can send HTTP requests to external services to create external tools for agents. To learn more about the Mosaic AI Agent Framework and tool calling, see Create an AI agent.

Before you begin

Source requirements:

• A bearer token from the external system.

Workspace requirements:

• Workspace enabled for Unity Catalog.

Compute requirements:

• Network connectivity from a Databricks compute resource to the external system. See Networking recommendations for Lakehouse Federation.

• Databricks compute must use Databricks Runtime 15.4 or above and shared or single user access mode.

• SQL warehouses must be pro or serverless.

Permissions required:

• To create a connection, you must be a metastore admin or a user with the CREATE CONNECTION privilege on the Unity Catalog metastore attached to the workspace.

Additional permission requirements are specified in each task-based section that follows.

Create a connection

A connection specifies a path and credentials for accessing an external database system. To create a connection, you can use Catalog Explorer or the CREATE CONNECTION SQL command in a Databricks notebook or the Databricks SQL query editor.

Note

You can also use the Databricks REST API or the Databricks CLI to create a connection. See POST /api/2.1/unity-catalog/connections and Unity Catalog commands.

Permissions required: Metastore admin or user with the CREATE CONNECTION privilege.

1. In your Databricks workspace, click Catalog.

2. In the left pane, expand the External Data menu and select Connections.

3. Click Create connection.

4. Enter a user-friendly Connection name.

5. Select a Connection type of HTTP.

6. Enter the following connection properties for the HTTP connection:

   • Host: For example, https://databricks.com

   • Port: For example, 443

   • Base Path: For example, /api/

   • Bearer Token: For example, bearer-token

7. (Optional) Add a comment.

8. Click Create.

Run the following command in a notebook or the Databricks SQL query editor:

CREATE CONNECTION <connection-name> TYPE HTTP
OPTIONS (
  host '<hostname>',
  port '<port>',
  base_path '<base-path>',
  bearer_token '<bearer-token>'
);

Databricks recommends that you use secrets instead of plaintext strings for sensitive values like credentials. For example:

CREATE CONNECTION <connection-name> TYPE HTTP
OPTIONS (
  host '<hostname>',
  port '<port>',
  base_path '<base-path>',
  bearer_token secret ('<secret-scope>','<secret-key-password>')
)

For information about setting up secrets, see Secret management.

Send an HTTP request to the external system

You can now send HTTP requests to the external data source using the http_request built-in function.

Permissions required: USE CONNECTION on the connection object.

Run the following SQL command in a notebook or the Databricks SQL editor. Items in brackets are optional. Replace the placeholder values:

• <connection-name>: The connection object that specifies the host, port, base_path, and access credentials.

• <http-method>: The HTTP request method used to make the call. For example: GET, POST, PUT, DELETE

• <path>: The path to concatenate after the base_path to invoke the service resource.

• json: The JSON body to send with the request.

• headers: A map to specify the request headers.

SELECT http_request(
  conn => <connection-name>,
  method => <http-method>,
  path => <path>,
  json => to_json(named_struct(
    'text', text
  )),
  headers => map(
    'Accept', "application/vnd.github+json"
  )
);

Limitations

• Bearer token authentication is the only supported authentication method.