Enforce AWS Instance Metadata Service v2 on a workspace

Important

Because serverless compute resources automatically enforce IMDSv2, this setting does not apply to serverless compute resources.

Instance metadata service (IMDS) is a service that runs locally on compute instances in AWS and is used to retrieve instance metadata. Crucially for security, instance metadata also includes credentials for the role associated with the instance. See Instance metadata and user data.

In response to security concerns around IMDS, AWS created IMDSv2 (version 2) which reduces risk from a common attack pattern and replaces the request-and-response flow with a session-oriented flow. For details of the improvements, see this AWS blog article.

As a workspace admin, you can enforce the use of IMDSv2 on clusters by enabling Enforce AWS instance metadata v2s in the Compute tab of the admin settings page. Databricks recommends that you configure your workspace to enforce IMDSv2. If your workspace was created after October 1, 2022, your workspace has this admin setting enabled by default.

Requirements

IMDSv2 enforcement does not support use of an isolated AWS Glue catalog. To disable isolation, see How to migrate and enforce IMDSv2 for all clusters.
IMDSv2 enforcement requires use of a supported Databricks Runtime version as listed on Databricks Runtime release notes versions and compatibility, however the Light 2.4 Extended Support version is unsupported.

How to migrate and enforce IMDSv2 for all clusters

Warning

Enforcing IMDSv2 causes any existing workloads to fail if they use IMDSv1 to fetch instance metadata.

To enforce IMDSv2 on new, non-serverless clusters:

IMDSv2 enforcement does not support use of an isolated AWS Glue catalog. To use Glue catalog, add one Spark conf line to your clusters to disable the isolation mode:
```
spark.databricks.hive.metastore.glueCatalog.isolation.enabled false
```
Upgrade your code to use IMDSv2.
1. Upgrade any existing AWS CLIs and SDKs that your workloads use. Note that Databricks has already upgraded the SDK that is installed by default in the Databricks Runtime. Databricks recommends that you follow AWS’s upgrade guide to ensure a safe transition.
  
  Modify all notebooks in the workspace to remove any existing IMDSv1 usage and replace with IMDSv2 usage.
  
  For example, the following is IMDSv1 API client code:
```
curl http://169.254.169.254/latest/meta-data/
```
  For that example, change it to IMDSv2 API client code:
```
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" \
-H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` && \
\
curl -H "X-aws-ec2-metadata-token: $TOKEN" \
  -v http://169.254.169.254/latest/meta-data/
```
  For more guidance and examples, see the the AWS article Retrieve instance metadata.
2. Test your modified code to ensure it works correctly with IMDSv2.
Enable enforcement of IMDSv2 for the workspace.
1. As a workspace admin, Go to the settings page.
2. Click the Compute tab.
3. Click Enforce AWS instance metadata v2.
4. Refresh the page to ensure that the setting took effect.
Restart any running clusters to ensure that all EC2 instances have IMDSv2 enforced. If clusters are attached to a fleet instance pool, create a new fleet instance pool and recreate the clusters using the new fleet instance pool.
Monitor the CloudWatch metric MetadataNoToken to ensure that your workspace is not making any active IMDSv1 calls.