Enforce AWS Instance Metadata Service v2 on a workspace

Preview

This feature is in Public Preview.

Instance metadata service (IMDS) is a service that runs locally on compute instances in AWS and is used to retrieve instance metadata. Crucially for security, instance metadata also includes credentials for the role associated with the instance. See Instance metadata and user data.

In response to security concerns around IMDS, AWS created IMDSv2 (version 2) which reduces risk from a common attack pattern and replaces the request-and-response flow with a session-oriented flow. For details of the improvements, see this AWS blog article.

Important

Databricks recommends that you configure your workspace to enforce IMDSv2.

You can enforce the use of IMDSv2 on clusters by enabling a workspace setting that is available as Public Preview.

Requirements

How to migrate and enforce IMDSv2 for all clusters

Warning

Enforcing IMDSv2 causes any existing workloads to fail if they use IMDSv1 to fetch instance metadata.

To enforce IMDSv2 on new clusters:

  1. IMDSv2 enforcement does not support use of an isolated AWS Glue catalog. To use Glue catalog, add one Spark conf line to your clusters to disable the isolation mode:

    spark.databricks.hive.metastore.glueCatalog.isolation.enabled false
    
  2. Upgrade your code to use IMDSv2.

    1. Upgrade any existing AWS CLIs and SDKs that your workloads use. Note that Databricks has already upgraded the SDK that is installed by default in the Databricks Runtime. Databricks recommends that you follow AWS’s upgrade guide to ensure a safe transition.

      Modify all notebooks in the workspace to remove any existing IMDSv1 usage and replace with IMDSv2 usage.

      For example, the following is IMDSv1 API client code:

      curl http://169.254.169.254/latest/meta-data/
      

      For that example, change it to IMDSv2 API client code:

      TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" \
      -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` && \
      \
      curl -H "X-aws-ec2-metadata-token: $TOKEN" \
        -v http://169.254.169.254/latest/meta-data/
      

      For more guidance and examples, see the the AWS article Retrieve instance metadata.

    2. Test your modified code to ensure it works correctly with IMDSv2.

  3. Enable enforcement of IMDSv2 for the workspace.

    1. As a workspace admin, go to the admin console.

    2. Click the Workspace settings tab.

    3. Click Enforce AWS Instance Metadata Service V2 for all clusters.

      IMDSv2 setting
    4. Refresh the page to ensure that the setting took effect.

  4. Restart any running clusters to ensure that all EC2 instances have IMDSv2 enforced. If clusters are attached to a fleet instance pool, create a new fleet instance pool and recreate the clusters using the new fleet instance pool.

  5. Monitor the CloudWatch metric MetadataNoToken to ensure that your workspace is not making any active IMDSv1 calls.