Advanced usage of Databricks Connect for Scala

Note

This article covers Databricks Connect for Databricks Runtime 14.3 and above.

This article describes topics that go beyond the basic set up of Databricks Connect.

Logging and debug logs

Databricks Connect for Scala uses SLF4J logging, and does not ship with any SLF4J providers.

Applications using Databricks Connect are expected to include an SLF4J provider that is suitable to them.

A suitable logging provider needs to be added, and in some cases, configured to print the log messages. The simplest option is to include the slf4j-simple provider which will print log messages at the INFO level and higher to the standard error stream (stderr).

A more configurable alternative is to use the slf4j-reload4j provider which picks up configuration from a log4j.properties file in the classpath.

The following code snippet shows a simple log4j.properties file.

log4j.rootLogger=INFO,stderr

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%p\t%d{ISO8601}\t%r\t%c\t[%t]\t%m%n

In the preceding example, debug logs will be printed if the root logger or a specific logger is configured at the DEBUG level, that is, by changing the line:

log4j.rootLogger=DEBUG,stderr

Certificates

If your cluster relies on a custom SSL/TLS certificate to resolve a Databricks workspace fully qualified domain name (FQDN), you must set the environment variable GRPC_DEFAULT_SSL_ROOTS_FILE_PATH on your local development machine. This environment variable must be set to the full path to the installed certificate on the cluster.

For example, you set this environment variable in Scala code as follows:

import scala.sys.process._

val envVar = "GRPC_DEFAULT_SSL_ROOTS_FILE_PATH"
val envValue = "/etc/ssl/certs/ca-bundle.crt"

// Setting the environment variable
val process = Process(Seq("bash", "-c", s"export $envVar=$envValue"))
process.!

For other ways to set environment variables, see your operating system’s documentation.