What files can I reference in an init script?
The support for referencing other files in an init script depends on where the referenced files are stored. This article outlines this behavior and provides recommendations.
Databricks recommends managing all init scripts as cluster-scoped init scripts.
What identity is used to run init scripts?
In dedicated access mode (formerly single user), the identity of the assigned principal (the user or group) is used.
In standard access mode (formerly shared) or no-isolation shared access mode, init scripts use the identity of the cluster owner.
Not all locations for storing init scripts are supported on all Databricks Runtime versions and access modes. See Where can init scripts be installed?.
Can I reference files in Unity Catalog volumes from init scripts?
You can reference libraries and init scripts stored in Unity Catalog volumes from init scripts stored in Unity Catalog volumes.
Credentials required to access other files stored in Unity Catalog volumes are only made available within init scripts stored in Unity Catalog volumes. You cannot reference any files in Unity Catalog volumes from init scripts configured from other locations.
For clusters with standard access mode, only the configured init script needs to be added to the allowlist. Access to other files referenced in the init script is governed by Unity Catalog.
Can I reference workspace files from init scripts?
In Databricks Runtime 11.3 LTS and above, you can reference other workspace files such as libraries, configuration files, or shell scripts from init scripts stored with workspace files.
Can I reference files in cloud object storage from init scripts?
You can reference libraries and init scripts stored in cloud object storage from init scripts.
For clusters with standard access mode, only the configured init script needs to be added to the allowlist. Access to other files referenced in the init script is determined by access configured to cloud object storage.
Databricks recommends using instance profiles to manage access to libraries and init scripts stored in S3. Use the following documentation in the cross-reference link to complete this setup:
- Create a IAM role with read and list permissions on your desired buckets. See Tutorial: Configure S3 access with an instance profile.
- Launch a cluster with the instance profile. See Instance profiles.