SQL endpoints

Preview

This feature is in Public Preview. Contact your Databricks representative to request access.

A Databricks SQL endpoint is a computation resource that lets you run SQL commands on data objects within the Databricks environment.

SQL endpoints appear in query history and record the user that ran the query.

SQL endpoints support the SQL commands in SQL reference for SQL Analytics.

The other type of data source is an external data source.

This section describes how to work with SQL endpoints using the UI. To work with SQL endpoints using the API, see SQL Endpoint APIs.

Queries through SQL endpoints are managed by the Query Watchdog with the defaults as described in that article.

Requirements

  • To create SQL endpoints you must have cluster create permission in Databricks Workspace.
  • To manage a SQL endpoint you must have Can Manage permission in Databricks SQL Analytics for the endpoint.

View SQL endpoints

Click the Endpoints Icon icon in the sidebar.

By default, endpoints are sorted in alphabetical order. You can reorder the list by clicking the column headings.

To filter the list of endpoints, enter text in the search box:

Filter endpoints

Create a SQL endpoint

  1. Click the Endpoints Icon icon in the sidebar.

  2. Click + New SQL Endpoint.

    Create endpoint
  3. Enter a name for the endpoint. Accept or edit the endpoint properties.

  4. Click Create.

Start, stop, or delete a SQL endpoint

  1. Click the Endpoints Icon icon in the sidebar.
  2. In the Actions column, click the vertical ellipsis Vertical Ellipsis and select Start, Stop, or Delete.

Edit a SQL endpoint

  1. Click the Endpoints Icon icon in the sidebar.

  2. Click an endpoint.

  3. Click Edit.

  4. Edit endpoint properties.

    • Cluster Size: the number of cluster workers and size of the coordinator. The default is X-Large. To reduce the latency of queries, increase the size. A larger size has a larger coordinator and doubles the number of cluster workers. For details, see Cluster size.

    • Auto Stop: whether the endpoint stops if it’s idle for the specified number of minutes. The default is Off, which means the endpoint never stops.

    • Multi-cluster Load Balancing: the minimum and maximum number of clusters over which queries sent to the endpoint are distributed. The default is Off and 1 cluster if On. To handle more concurrent users for a given query, enable load balancing and increase the cluster count.

    • Photon: whether queries are executed on a native vectorized engine that speeds up query execution. The default is Off. To enable, select On, read the disclaimer, and click Enable Photon.

      Preview

      Multi-cluster Load Balancing and Photon are in Preview. Contact your Databricks representative to request access.

    • Spot policy: whether all on-demand instances or spot instances are used for workers and on-demand for the driver. The default is Cost Optimized. Reliability Optimized uses all on-demand instances. Cost Optimized uses mostly spot instances and one on-demand instance.
    • Permissions: users and groups that can manage and use the endpoint.
    • Tags: key-value pair annotations that can identify who uses the endpoint.
  5. Click Save or Save and restart.

Add an endpoint tag

Tags allow you to easily monitor the cost of cloud resources used by various groups in your organization. You can specify tags as key-value pairs when you create an endpoint, and Databricks applies these tags to cloud resources.

To add an endpoint tag:

  1. Click the Endpoints Icon icon in the sidebar.

  2. Click an endpoint.

  3. Click Edit.

  4. Select More Options.

  5. Click the Tags tab.

  6. Enter a tag key and value.

    Add tag
  7. Click Save and restart.

Monitor a SQL endpoint

You can examine the number of queries handled by the endpoint and the number of clusters allocated to the endpoint.

  1. Click the Endpoints Icon icon in the sidebar.

  2. Click an endpoint.

  3. Click Monitoring.

    A chart showing the number of queries handled by the endpoint and the number of clusters allocated to the endpoint over the last 6 hours displays.

    Click a timescale button at the top right of the chart to change the displayed period. For example, the following screenshot shows these statistics over 7 days:

    Monitor endpoint

    Note

    The Cluster Count can be greater than one only if multi-cluster load balancing is enabled and configured.

Cluster size

The table in this section maps SQL endpoint cluster sizes to Databricks cluster driver size and worker counts.

Cluster size Driver size Worker count
2X-Small i3.xlarge 1
X-Small i3.2xlarge 2
Small i3.4xlarge 4
Medium i3.8xlarge 8
Large i3.8xlarge 16
X-Large i3.16xlarge 32
2X-Large i3.16xlarge 64
3X-Large i3.16xlarge 128
4X-Large i3.16xlarge 256

The instance size of all workers is i3.2xlarge.