Skip to main content
Last updated on

Opt out of pre-installed Python libraries for Databricks apps

This page describes how to opt out of pre-installed Python libraries on a horizontally scaled Databricks app, manage your own dependencies, and turn the pre-installed libraries back on later. Opting out is available only on horizontally scaled apps that were converted from a standard app. See Convert a standard app to use horizontal scaling. Horizontal scaling is in Beta.

When you convert a standard app to horizontally scaled, Databricks pins the converted app to the pre-installed Python libraries so your app's existing imports keep working. Opt out to run the app on a clean base OS image and declare every library yourself.

Opt out of pre-installed libraries​

Opting out tells Databricks to build your next deployment against a clean base OS image. The pre-installed libraries are no longer available unless you declare them in requirements.txt or pyproject.toml.

To opt out:

  1. On the app details page, click Edit.
  2. In the Configure step, clear the Use preinstalled Python libraries checkbox.
  3. Click Save. The currently running deployment is not affected.
  4. Deploy your app again from the Deployments tab. This new deployment builds against the clean base OS image.

Your dependencies might include pre-installed libraries transitively, so it isn't always clear which libraries your app requires directly. Find missing libraries on a test app before opting out in production:

  1. Create a new horizontally scaled app. New horizontally scaled apps already run on the clean base OS image, so this matches your post-opt-out target.
  2. Deploy your production app's source to the test app. Update requirements.txt (or pyproject.toml) and redeploy until the test app starts, serves requests, and passes your tests.
  3. Copy the final dependency declarations into your production app's source.
  4. On the production app, follow the steps in Opt out of pre-installed libraries and deploy the updated source.

Production keeps running on the image with the pre-installed libraries until your new deployment passes health checks, so this approach avoids downtime while you identify missing dependencies on the test app.

Resolve errors after opting out​

The most common failure after opting out is a ModuleNotFoundError at app startup, caused by a framework or library your code imported implicitly from the pre-installed set. To resolve it:

  1. Open the Logs tab on the app details page.
  2. Find the missing module name in the build or runtime logs.
  3. Add the corresponding package to requirements.txt (or pyproject.toml) and redeploy.

A new deployment only replaces the running deployment if the app process stays running for the first few seconds after startup. An opt-out deployment that crashes immediately on a missing import never takes over, and traffic continues on the previous working deployment while you fix the dependency list. The startup check is shallow, so test that your app actually serves requests after opting out rather than relying on the check to catch broken deployments.

Turn pre-installed libraries back on​

To return to the pre-installed libraries after opting out:

  1. On the app details page, click Edit.
  2. In the Configure step, select the Use preinstalled Python libraries checkbox.
  3. Click Save.
  4. Deploy your app again from the Deployments tab. This new deployment builds against the base image with the pre-installed libraries.

Your requirements.txt and pyproject.toml continue to apply, so any explicit declarations you added while opted out are preserved.

note

Converting a horizontally scaled app back to a standard app also restores the pre-installed libraries: the standard app is rebuilt with them, regardless of whether you had opted out. If you later convert the app to horizontally scaled again, it includes the pre-installed libraries until you opt out again.

On this page