Best Practices for Hiding Secrets in Databricks

Hiding Secrets

There are keys tricks for storing secrets in Databricks.

  1. It is a best practice to keep all your secrets in Databricks in one location. We recommend inside of your user folder. If you enable the Databricks Operational Security Package, you can use user Managing Access Control so that you and only you will have access to this notebook.
  2. Leverage the function scoping functionality of both Scala and Python to hide true values.
  3. Take advantage of Databricks’ %run notebook functionality to effectively run a notebook from another location. For more information on %run see Running a Notebook from Another Notebook.

The below notebook has an example of how this functionality works!

Note that when used this way, your secret will not show up in output. Additionally, when you change your keys. It’s trivial to change them in one location without having to think about every notebook that uses them individually.