cluster-encryption-init-script(Python)

Init Script: Configure Encrypted Traffic Between Worker Nodes

This init script creates the Spark configuration to enable encrypted traffic between worker nodes in a cluster. The first time you run it, it copies the keystore file to DBFS.

1. Copy <init-script-folder> with the location to put the init script and <keystore_directory> with the target location of keystore in DBFS.
2. Run this notebook to create the script enable-encryption.sh.
3. Configure a cluster with the enable-encryption.sh cluster-scoped init script using the UI, Databricks CLI, or by invoking the Clusters API.

You may also configure the script as global init script type in the Admin console by copying the init script content in command 5 with path changes made. Restart all the clusters to have changes take in effect.

#!/bin/bash

set -euo pipefail

keystore_dbfs_file="/dbfs/<keystore_directory>/jetty_ssl_driver_keystore.jks"

## Wait till keystore file is available via Fuse

max_attempts=30
while [ ! -f ${keystore_dbfs_file} ];
do
  if [ "$max_attempts" == 0 ]; then
    echo "ERROR: Unable to find the file : $keystore_dbfs_file .Failing the script."
    exit 1
  fi
  sleep 2s
  ((max_attempts--))
done
 ## Derive shared internode encryption secret from the hash of the keystore file
sasl_secret=$(sha256sum $keystore_dbfs_file | cut -d' ' -f1)


if [ -z "${sasl_secret}" ]; then
  echo "ERROR: Unable to derive the secret.Failing the script."
  exit 1
fi

# The JKS keystore file used for enabling SSL/HTTPS
local_keystore_file="$DB_HOME/keys/jetty_ssl_driver_keystore.jks"
# Password of the JKS keystore file. This jks password is hardcoded and is not intended to protect the confidentiality
# of the keystore. Do not assume the keystore file itself is protected.
local_keystore_password="gb1gQqZ9ZIHS"

## Updating spark-branch.conf is only needed for driver

if [[ $DB_IS_DRIVER = "TRUE" ]]; then
  driver_conf=${DB_HOME}/driver/conf/spark-branch.conf
  echo "Configuring driver conf at $driver_conf"
  
  if [ ! -e $driver_conf ] ; then
    touch $driver_conf
  fi

cat << EOF >>  $driver_conf
  [driver] {
    // Configure inter-node authentication
   "spark.authenticate" = true
   "spark.authenticate.secret" = "$sasl_secret"
   // Configure AES encryption
   "spark.network.crypto.enabled" = true
   "spark.network.crypto.saslFallback" = false
   // Configure SSL
   "spark.ssl.enabled" = true
   "spark.ssl.keyPassword" = "$local_keystore_password"
   "spark.ssl.keyStore" = "$local_keystore_file"
   "spark.ssl.keyStorePassword" = "$local_keystore_password"
   "spark.ssl.protocol" ="TLSv1.3"
   "spark.ssl.standalone.enabled" = true
   "spark.ssl.ui.enabled" = true
   }
EOF
  echo "Successfully configured driver conf at $driver_conf"
fi

# Setting configs in spark-defaults.conf for the spark master and worker

spark_defaults_conf="$DB_HOME/spark/conf/spark-defaults.conf"
echo "Configuring spark defaults conf at $spark_defaults_conf"
if [ ! -e $spark_defaults_conf ] ; then
  touch $spark_defaults_conf
fi

cat << EOF >>  $spark_defaults_conf
spark.authenticate true
spark.authenticate.secret $sasl_secret
spark.network.crypto.enabled true
spark.network.crypto.saslFallback false

spark.ssl.enabled true
spark.ssl.keyPassword $local_keystore_password
spark.ssl.keyStore $local_keystore_file
spark.ssl.keyStorePassword $local_keystore_password
spark.ssl.protocol TLSv1.3
spark.ssl.standalone.enabled true
spark.ssl.ui.enabled true
EOF

echo "Successfully configured spark defaults conf at $spark_defaults_conf"