Advanced usage of Databricks Connect for Scala

Note

This article covers Databricks Connect for Databricks Runtime 14.3 and above.

This article describes topics that go beyond the basic set up of Databricks Connect.

Logging and debug logs

Databricks Connect for Scala uses SLF4J logging, and does not ship with any SLF4J providers.

Applications using Databricks Connect are expected to include an SLF4J provider that is suitable to them.

A suitable logging provider needs to be added, and in some cases, configured to print the log messages. The simplest option is to include the slf4j-simple provider which will print log messages at the INFO level and higher to the standard error stream (stderr).

A more configurable alternative is to use the slf4j-reload4j provider which picks up configuration from a log4j.properties file in the classpath.

The following code snippet shows a simple log4j.properties file.

log4j.rootLogger=INFO,stderr

log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%p\t%d{ISO8601}\t%r\t%c\t[%t]\t%m%n

In the preceding example, debug logs will be printed if the root logger or a specific logger is configured at the DEBUG level, that is, by changing the line:

log4j.rootLogger=DEBUG,stderr