TensorFlow  is an open-source framework for Machine Learning intelligence created by Google. It supports deep-learning and general numerical computations on CPUs, GPUs and clusters of GPUs. It is subject to the terms and conditions of the Apache 2.0 License.
In the sections below, we provide guidance on installing TensorFlow on Databricks and give an example of running TensorFlow programs. See Integrating Deep Learning Libraries with Apache Spark for an example of integrating a deep learning library with Spark.
This guide is not a comprehensive guide on TensorFlow. Please also refer to the TensorFlow website.
Previous versions of TensorFlow required installation using an Init Script. This is no longer necessary!
Setting up TensorBoard¶
TensorBoard  is TensorFlow’s suite of visualization tools for debugging, optimizing and understanding TensorFlow programs. If you’d like to run TensorBoard on your Databricks cluster to monitor TensorFlow, you will first need to update the Databricks security group in your AWS account to give ingress access to incoming TensorBoard connections. You will need to specify which IP addresses will be allowed to connect to TensorBoard. You can give access to an individual IP address or provide a range that represents your entire office IP range. You or your admin only need to complete this step once.
Follow the below steps in AWS console to set it up:
- From your AWS console, find the Databricks security group. It will have a
label similar to
- Edit the security group, and add an inbound TCP rule to allow port
6006to worker machines. It can be a single IP address of your machine or a range.
- Make sure your laptop and office allows sending TCP traffic on port
Running TensorBoard is not currently supported on Community Edition accounts.
New in version 2.1.1-db5.
Once you’ve enabled incoming connections to TensorBoard, you can start TensorBoard directly from your notebook using a single command.
log_dir = "/tmp/tensorflow_log_dir" dbutils.tensorboard.start(log_dir)
This command will display a link that, when clicked, opens TensorBoard in a new tab. Make sure
to use the same log directory when you start TensorBoard and when you run your TensorFlow program.
We recommend you use a local directory on the driver, for example
store your log files for the best performance (and copy to persistent storage as needed)”.
TensorBoard will continue to run until you either. TensorBoard will continue to run until you either
dbutils.tensorboard.stop() to stop it or until you shut down your cluster.
Use TensorFlow on a single node¶
To test and migrate single-machine TensorFlow workflows, you can start with a driver-only cluster on Databricks by setting the number of workers to zero. Though Apache Spark is not functional under this setting, it is a cost-effective way to run single-machine TensorFlow workflows. This example shows how you can run TensorFlow, with TensorBoard monitoring on a driver-only cluster.
|||(1, 2) TensorFlow and TensorBoard are trademarks of Google, Inc.|
Spark-Tensorflow Data Conversion¶
spark-tensorflow-connector is a library within the Tensorflow ecosystem that enables conversion between Spark DataFrames and TFRecords (a popular format for storing data for Tensorflow). With spark-tensorflow-connector, you can use Spark DataFrame APIs to read TFRecords files into DataFrames and write DataFrames as TFRecords.
To use spark-tensorflow-connector on Databricks, you’ll need to build the project JAR locally, upload it to Databricks, and attach it to your cluster as a Library:
First, ensure you have SBT 0.13.13 in your PATH (on
brew install sbt to install sbt). Verify your installation by running
$ sbt sbt-version.
The default maven build of spark-tensorflow-connector (described in the project README) is not recommended, as the resulting JAR does not include all of spark-tensorflow-connector’s dependencies.
Clone the spark-tensorflow-connector repo and build the project with SBT:
git clone https://github.com/tensorflow/ecosystem/tree/master/spark/spark-tensorflow-connector cd spark-tensorflow-connector sbt clean assembly
sbt clean assembly command prints the path of the assembly JAR, e.g.:
[info] Packaging /Users/yourusername/code/tf-ecosystem/spark/spark-tensorflow-connector/target/scala-2.11/spark-tensorflow-connector-assembly-1.0.0.jar ...
Upload this JAR to Databricks as a Library and attach it to your cluster. You should now be able to run the example notebook below (adapted from the spark-tensorflow-connector usage examples):