External Metastore Hive 2.1.1(Scala)
Import Notebook

Instructions for Replacing datanucleus-rdbms

In this notebook, you download JARs jars required for accessing the Hive 2.1.1 metastore. Then, you copy all of the jars to a folder in DBFS and replace the datanucleus-rdbms 4.1.7 with datanucleus-rdbms 4.1.16. Finally, you create an init script that sets up clusters to use those JARs in DBFS.

Note: Instructions in this notebook also apply to Hive 2.0 to 2.2 metastores backed by SQL Server.

Step 1: Set up a cluster with spark.sql.hive.metastore.version set to 2.1.1 and spark.sql.hive.metastore.jars set to maven

Follow the instructions in https://docs.azuredatabricks.net/user-guide/advanced/external-hive-metastore.html. Besides spark.sql.hive.metastore.version and spark.sql.hive.metastore.jars, you must also set spark.hadoop.javax.jdo.option.ConnectionURL, spark.hadoop.javax.jdo.option.ConnectionUserName, spark.hadoop.javax.jdo.option.ConnectionPassword, and spark.hadoop.javax.jdo.option.ConnectionDriverName (to com.microsoft.sqlserver.jdbc.SQLServerDriver).

Step 2: Find the location of downloaded metastore JARs.

%sh grep "Downloaded metastore jars to" /databricks/driver/logs/log4j-active.log
18/01/21 04:52:39 INFO IsolatedClientLoader: Downloaded metastore jars to /tmp/hive-v2_1-9e1bf2ea-f62f-4d72-87e1-6b7c27c2fa82

In this example, those metastore JARs are downloaded to /tmp/hive-v2_1-9e1bf2ea-f62f-4d72-87e1-6b7c27c2fa82 in the driver node.

%sh ls /tmp/hive-v2_1-9e1bf2ea-f62f-4d72-87e1-6b7c27c2fa82
antlr_antlr-2.7.7.jar aopalliance_aopalliance-1.0.jar asm_asm-3.2.jar asm_asm-commons-3.1.jar asm_asm-tree-3.1.jar co.cask.tephra_tephra-api-0.6.0.jar co.cask.tephra_tephra-core-0.6.0.jar co.cask.tephra_tephra-hbase-compat-1.0-0.6.0.jar com.fasterxml.jackson.core_jackson-annotations-2.4.0.jar com.fasterxml.jackson.core_jackson-core-2.4.2.jar com.fasterxml.jackson.core_jackson-databind-2.4.2.jar com.github.joshelser_dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar com.github.stephenc.findbugs_findbugs-annotations-1.3.9-1.jar com.google.code.findbugs_jsr305-3.0.0.jar com.google.code.gson_gson-2.2.4.jar com.google.guava_guava-14.0.1.jar com.google.inject.extensions_guice-assistedinject-3.0.jar com.google.inject.extensions_guice-servlet-3.0.jar com.google.inject_guice-3.0.jar com.google.protobuf_protobuf-java-2.5.0.jar com.jamesmurty.utils_java-xmlbuilder-0.4.jar com.jcraft_jsch-0.1.42.jar com.jolbox_bonecp-0.8.0.RELEASE.jar commons-beanutils_commons-beanutils-1.7.0.jar commons-beanutils_commons-beanutils-core-1.8.0.jar commons-cli_commons-cli-1.2.jar commons-codec_commons-codec-1.4.jar commons-collections_commons-collections-3.2.2.jar commons-configuration_commons-configuration-1.6.jar commons-daemon_commons-daemon-1.0.13.jar commons-dbcp_commons-dbcp-1.4.jar commons-digester_commons-digester-1.8.jar commons-el_commons-el-1.0.jar commons-httpclient_commons-httpclient-3.1.jar commons-io_commons-io-2.4.jar commons-lang_commons-lang-2.6.jar commons-logging_commons-logging-1.2.jar commons-net_commons-net-3.1.jar commons-pool_commons-pool-1.5.4.jar com.sun.jersey.contribs_jersey-guice-1.9.jar com.sun.jersey_jersey-client-1.9.jar com.sun.jersey_jersey-core-1.14.jar com.sun.jersey_jersey-json-1.14.jar com.sun.jersey_jersey-server-1.14.jar com.sun.xml.bind_jaxb-impl-2.2.3-1.jar com.thoughtworks.paranamer_paranamer-2.3.jar io.dropwizard.metrics_metrics-core-3.1.0.jar io.dropwizard.metrics_metrics-json-3.1.0.jar io.dropwizard.metrics_metrics-jvm-3.1.0.jar io.netty_netty-3.7.0.Final.jar io.netty_netty-all-4.0.23.Final.jar it.unimi.dsi_fastutil-6.5.6.jar javax.activation_activation-1.1.jar javax.inject_javax.inject-1.jar javax.jdo_jdo-api-3.0.1.jar javax.mail_mail-1.4.1.jar javax.servlet.jsp_jsp-api-2.1.jar javax.servlet_servlet-api-2.5.jar javax.transaction_jta-1.1.jar javax.transaction_transaction-api-1.1.jar javax.xml.bind_jaxb-api-2.2.2.jar javax.xml.stream_stax-api-1.0-2.jar javolution_javolution-5.5.1.jar jline_jline-2.12.jar joda-time_joda-time-2.5.jar junit_junit-4.11.jar log4j_log4j-1.2.17.jar net.hydromatic_eigenbase-properties-1.1.5.jar net.java.dev.jets3t_jets3t-0.9.0.jar net.sf.opencsv_opencsv-2.3.jar org.antlr_antlr-runtime-3.4.jar org.antlr_ST4-4.0.4.jar org.antlr_stringtemplate-3.2.1.jar org.apache.ant_ant-1.9.1.jar org.apache.ant_ant-launcher-1.9.1.jar org.apache.avro_avro-1.7.7.jar org.apache.calcite_calcite-avatica-1.6.0.jar org.apache.calcite_calcite-core-1.6.0.jar org.apache.calcite_calcite-linq4j-1.6.0.jar org.apache.commons_commons-compress-1.9.jar org.apache.commons_commons-lang3-3.1.jar org.apache.commons_commons-math3-3.1.1.jar org.apache.derby_derby-10.10.2.0.jar org.apache.directory.api_api-asn1-api-1.0.0-M20.jar org.apache.directory.api_api-util-1.0.0-M20.jar org.apache.directory.server_apacheds-i18n-2.0.0-M15.jar org.apache.directory.server_apacheds-kerberos-codec-2.0.0-M15.jar org.apache.geronimo.specs_geronimo-annotation_1.0_spec-1.1.1.jar org.apache.geronimo.specs_geronimo-jaspic_1.0_spec-1.0.jar org.apache.geronimo.specs_geronimo-jta_1.1_spec-1.1.1.jar org.apache.hadoop_hadoop-annotations-2.7.3.jar org.apache.hadoop_hadoop-auth-2.7.3.jar org.apache.hadoop_hadoop-client-2.7.3.jar org.apache.hadoop_hadoop-common-2.7.3.jar org.apache.hadoop_hadoop-hdfs-2.7.3.jar org.apache.hadoop_hadoop-mapreduce-client-app-2.7.3.jar org.apache.hadoop_hadoop-mapreduce-client-common-2.7.3.jar org.apache.hadoop_hadoop-mapreduce-client-core-2.7.3.jar org.apache.hadoop_hadoop-mapreduce-client-jobclient-2.7.3.jar org.apache.hadoop_hadoop-mapreduce-client-shuffle-2.7.3.jar org.apache.hadoop_hadoop-yarn-api-2.7.3.jar org.apache.hadoop_hadoop-yarn-client-2.7.3.jar org.apache.hadoop_hadoop-yarn-common-2.7.3.jar org.apache.hadoop_hadoop-yarn-server-applicationhistoryservice-2.6.1.jar org.apache.hadoop_hadoop-yarn-server-common-2.7.3.jar org.apache.hadoop_hadoop-yarn-server-resourcemanager-2.6.1.jar org.apache.hadoop_hadoop-yarn-server-web-proxy-2.6.1.jar org.apache.hbase_hbase-annotations-1.1.1.jar org.apache.hbase_hbase-client-1.1.1.jar org.apache.hbase_hbase-protocol-1.1.1.jar org.apache.hive_hive-ant-2.1.1.jar org.apache.hive_hive-common-2.1.1.jar org.apache.hive_hive-exec-2.1.1.jar org.apache.hive_hive-llap-client-2.1.1.jar org.apache.hive_hive-llap-common-2.1.1.jar org.apache.hive_hive-llap-tez-2.1.1.jar org.apache.hive_hive-metastore-2.1.1.jar org.apache.hive_hive-orc-2.1.1.jar org.apache.hive_hive-serde-2.1.1.jar org.apache.hive_hive-service-rpc-2.1.1.jar org.apache.hive_hive-shims-2.1.1.jar org.apache.hive_hive-storage-api-2.1.1.jar org.apache.hive.shims_hive-shims-0.23-2.1.1.jar org.apache.hive.shims_hive-shims-common-2.1.1.jar org.apache.hive.shims_hive-shims-scheduler-2.1.1.jar org.apache.htrace_htrace-core-3.1.0-incubating.jar org.apache.httpcomponents_httpclient-4.4.jar org.apache.httpcomponents_httpcore-4.4.jar org.apache.ivy_ivy-2.4.0.jar org.apache.logging.log4j_log4j-1.2-api-2.4.1.jar org.apache.logging.log4j_log4j-slf4j-impl-2.4.1.jar org.apache.logging.log4j_log4j-web-2.4.1.jar org.apache.parquet_parquet-hadoop-bundle-1.8.1.jar org.apache.thrift_libfb303-0.9.3.jar org.apache.thrift_libthrift-0.9.3.jar org.apache.twill_twill-api-0.6.0-incubating.jar org.apache.twill_twill-common-0.6.0-incubating.jar org.apache.twill_twill-core-0.6.0-incubating.jar org.apache.twill_twill-discovery-api-0.6.0-incubating.jar org.apache.twill_twill-discovery-core-0.6.0-incubating.jar org.apache.twill_twill-zookeeper-0.6.0-incubating.jar org.apache.velocity_velocity-1.5.jar org.apache.zookeeper_zookeeper-3.4.6.jar org.codehaus.groovy_groovy-all-2.4.4.jar org.codehaus.jackson_jackson-core-asl-1.9.13.jar org.codehaus.jackson_jackson-jaxrs-1.9.13.jar org.codehaus.jackson_jackson-mapper-asl-1.9.13.jar org.codehaus.jackson_jackson-xc-1.9.13.jar org.codehaus.janino_commons-compiler-2.7.6.jar org.codehaus.janino_janino-2.7.6.jar org.codehaus.jettison_jettison-1.1.jar org.datanucleus_datanucleus-api-jdo-4.2.1.jar org.datanucleus_datanucleus-core-4.1.6.jar org.datanucleus_datanucleus-rdbms-4.1.7.jar org.datanucleus_javax.jdo-3.2.0-m3.jar org.eclipse.jetty.aggregate_jetty-all-7.6.0.v20120127.jar org.eclipse.jetty.orbit_javax.servlet-3.0.0.v201112011016.jar org.fusesource.leveldbjni_leveldbjni-all-1.8.jar org.hamcrest_hamcrest-core-1.3.jar org.iq80.snappy_snappy-0.2.jar org.jruby.jcodings_jcodings-1.0.8.jar org.jruby.joni_joni-2.1.2.jar org.json_json-20090211.jar org.mortbay.jetty_jetty-6.1.26.jar org.mortbay.jetty_jetty-util-6.1.26.jar org.slf4j_slf4j-api-1.7.10.jar org.slf4j_slf4j-log4j12-1.7.10.jar org.sonatype.sisu.inject_cglib-2.2.1-v20090111.jar org.xerial.snappy_snappy-java-1.0.5.jar oro_oro-2.0.8.jar stax_stax-api-1.0.1.jar tomcat_jasper-compiler-5.5.23.jar tomcat_jasper-runtime-5.5.23.jar xerces_xercesImpl-2.9.1.jar xml-apis_xml-apis-1.3.04.jar xmlenc_xmlenc-0.52.jar

Step 3: Copy downloaded JARs to DBFS using the DBFS mount point in the local file system of the driver node. You can use those JARs for clusters launched in future.

%sh cp -r /tmp/hive-v2_1-9e1bf2ea-f62f-4d72-87e1-6b7c27c2fa82 /dbfs/hive_2_1_1_metastore_jars

%sh ls /dbfs/hive_2_1_1_metastore_jars
antlr_antlr-2.7.7.jar aopalliance_aopalliance-1.0.jar asm_asm-3.2.jar asm_asm-commons-3.1.jar asm_asm-tree-3.1.jar co.cask.tephra_tephra-api-0.6.0.jar co.cask.tephra_tephra-core-0.6.0.jar co.cask.tephra_tephra-hbase-compat-1.0-0.6.0.jar com.fasterxml.jackson.core_jackson-annotations-2.4.0.jar com.fasterxml.jackson.core_jackson-core-2.4.2.jar com.fasterxml.jackson.core_jackson-databind-2.4.2.jar com.github.joshelser_dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar com.github.stephenc.findbugs_findbugs-annotations-1.3.9-1.jar com.google.code.findbugs_jsr305-3.0.0.jar com.google.code.gson_gson-2.2.4.jar com.google.guava_guava-14.0.1.jar com.google.inject.extensions_guice-assistedinject-3.0.jar com.google.inject.extensions_guice-servlet-3.0.jar com.google.inject_guice-3.0.jar com.google.protobuf_protobuf-java-2.5.0.jar com.jamesmurty.utils_java-xmlbuilder-0.4.jar com.jcraft_jsch-0.1.42.jar com.jolbox_bonecp-0.8.0.RELEASE.jar commons-beanutils_commons-beanutils-1.7.0.jar commons-beanutils_commons-beanutils-core-1.8.0.jar commons-cli_commons-cli-1.2.jar commons-codec_commons-codec-1.4.jar commons-collections_commons-collections-3.2.2.jar commons-configuration_commons-configuration-1.6.jar commons-daemon_commons-daemon-1.0.13.jar commons-dbcp_commons-dbcp-1.4.jar commons-digester_commons-digester-1.8.jar commons-el_commons-el-1.0.jar commons-httpclient_commons-httpclient-3.1.jar commons-io_commons-io-2.4.jar commons-lang_commons-lang-2.6.jar commons-logging_commons-logging-1.2.jar commons-net_commons-net-3.1.jar commons-pool_commons-pool-1.5.4.jar com.sun.jersey.contribs_jersey-guice-1.9.jar com.sun.jersey_jersey-client-1.9.jar com.sun.jersey_jersey-core-1.14.jar com.sun.jersey_jersey-json-1.14.jar com.sun.jersey_jersey-server-1.14.jar com.sun.xml.bind_jaxb-impl-2.2.3-1.jar com.thoughtworks.paranamer_paranamer-2.3.jar io.dropwizard.metrics_metrics-core-3.1.0.jar io.dropwizard.metrics_metrics-json-3.1.0.jar io.dropwizard.metrics_metrics-jvm-3.1.0.jar io.netty_netty-3.7.0.Final.jar io.netty_netty-all-4.0.23.Final.jar it.unimi.dsi_fastutil-6.5.6.jar javax.activation_activation-1.1.jar javax.inject_javax.inject-1.jar javax.jdo_jdo-api-3.0.1.jar javax.mail_mail-1.4.1.jar javax.servlet.jsp_jsp-api-2.1.jar javax.servlet_servlet-api-2.5.jar javax.transaction_jta-1.1.jar javax.transaction_transaction-api-1.1.jar javax.xml.bind_jaxb-api-2.2.2.jar javax.xml.stream_stax-api-1.0-2.jar javolution_javolution-5.5.1.jar jline_jline-2.12.jar joda-time_joda-time-2.5.jar junit_junit-4.11.jar log4j_log4j-1.2.17.jar net.hydromatic_eigenbase-properties-1.1.5.jar net.java.dev.jets3t_jets3t-0.9.0.jar net.sf.opencsv_opencsv-2.3.jar org.antlr_antlr-runtime-3.4.jar org.antlr_ST4-4.0.4.jar org.antlr_stringtemplate-3.2.1.jar org.apache.ant_ant-1.9.1.jar org.apache.ant_ant-launcher-1.9.1.jar org.apache.avro_avro-1.7.7.jar org.apache.calcite_calcite-avatica-1.6.0.jar org.apache.calcite_calcite-core-1.6.0.jar org.apache.calcite_calcite-linq4j-1.6.0.jar org.apache.commons_commons-compress-1.9.jar org.apache.commons_commons-lang3-3.1.jar org.apache.commons_commons-math3-3.1.1.jar org.apache.derby_derby-10.10.2.0.jar org.apache.directory.api_api-asn1-api-1.0.0-M20.jar org.apache.directory.api_api-util-1.0.0-M20.jar org.apache.directory.server_apacheds-i18n-2.0.0-M15.jar org.apache.directory.server_apacheds-kerberos-codec-2.0.0-M15.jar org.apache.geronimo.specs_geronimo-annotation_1.0_spec-1.1.1.jar org.apache.geronimo.specs_geronimo-jaspic_1.0_spec-1.0.jar org.apache.geronimo.specs_geronimo-jta_1.1_spec-1.1.1.jar org.apache.hadoop_hadoop-annotations-2.7.3.jar org.apache.hadoop_hadoop-auth-2.7.3.jar org.apache.hadoop_hadoop-client-2.7.3.jar org.apache.hadoop_hadoop-common-2.7.3.jar org.apache.hadoop_hadoop-hdfs-2.7.3.jar org.apache.hadoop_hadoop-mapreduce-client-app-2.7.3.jar org.apache.hadoop_hadoop-mapreduce-client-common-2.7.3.jar org.apache.hadoop_hadoop-mapreduce-client-core-2.7.3.jar org.apache.hadoop_hadoop-mapreduce-client-jobclient-2.7.3.jar org.apache.hadoop_hadoop-mapreduce-client-shuffle-2.7.3.jar org.apache.hadoop_hadoop-yarn-api-2.7.3.jar org.apache.hadoop_hadoop-yarn-client-2.7.3.jar org.apache.hadoop_hadoop-yarn-common-2.7.3.jar org.apache.hadoop_hadoop-yarn-server-applicationhistoryservice-2.6.1.jar org.apache.hadoop_hadoop-yarn-server-common-2.7.3.jar org.apache.hadoop_hadoop-yarn-server-resourcemanager-2.6.1.jar org.apache.hadoop_hadoop-yarn-server-web-proxy-2.6.1.jar org.apache.hbase_hbase-annotations-1.1.1.jar org.apache.hbase_hbase-client-1.1.1.jar org.apache.hbase_hbase-protocol-1.1.1.jar org.apache.hive_hive-ant-2.1.1.jar org.apache.hive_hive-common-2.1.1.jar org.apache.hive_hive-exec-2.1.1.jar org.apache.hive_hive-llap-client-2.1.1.jar org.apache.hive_hive-llap-common-2.1.1.jar org.apache.hive_hive-llap-tez-2.1.1.jar org.apache.hive_hive-metastore-2.1.1.jar org.apache.hive_hive-orc-2.1.1.jar org.apache.hive_hive-serde-2.1.1.jar org.apache.hive_hive-service-rpc-2.1.1.jar org.apache.hive_hive-shims-2.1.1.jar org.apache.hive_hive-storage-api-2.1.1.jar org.apache.hive.shims_hive-shims-0.23-2.1.1.jar org.apache.hive.shims_hive-shims-common-2.1.1.jar org.apache.hive.shims_hive-shims-scheduler-2.1.1.jar org.apache.htrace_htrace-core-3.1.0-incubating.jar org.apache.httpcomponents_httpclient-4.4.jar org.apache.httpcomponents_httpcore-4.4.jar org.apache.ivy_ivy-2.4.0.jar org.apache.logging.log4j_log4j-1.2-api-2.4.1.jar org.apache.logging.log4j_log4j-slf4j-impl-2.4.1.jar org.apache.logging.log4j_log4j-web-2.4.1.jar org.apache.parquet_parquet-hadoop-bundle-1.8.1.jar org.apache.thrift_libfb303-0.9.3.jar org.apache.thrift_libthrift-0.9.3.jar org.apache.twill_twill-api-0.6.0-incubating.jar org.apache.twill_twill-common-0.6.0-incubating.jar org.apache.twill_twill-core-0.6.0-incubating.jar org.apache.twill_twill-discovery-api-0.6.0-incubating.jar org.apache.twill_twill-discovery-core-0.6.0-incubating.jar org.apache.twill_twill-zookeeper-0.6.0-incubating.jar org.apache.velocity_velocity-1.5.jar org.apache.zookeeper_zookeeper-3.4.6.jar org.codehaus.groovy_groovy-all-2.4.4.jar org.codehaus.jackson_jackson-core-asl-1.9.13.jar org.codehaus.jackson_jackson-jaxrs-1.9.13.jar org.codehaus.jackson_jackson-mapper-asl-1.9.13.jar org.codehaus.jackson_jackson-xc-1.9.13.jar org.codehaus.janino_commons-compiler-2.7.6.jar org.codehaus.janino_janino-2.7.6.jar org.codehaus.jettison_jettison-1.1.jar org.datanucleus_datanucleus-api-jdo-4.2.1.jar org.datanucleus_datanucleus-core-4.1.6.jar org.datanucleus_datanucleus-rdbms-4.1.7.jar org.datanucleus_javax.jdo-3.2.0-m3.jar org.eclipse.jetty.aggregate_jetty-all-7.6.0.v20120127.jar org.eclipse.jetty.orbit_javax.servlet-3.0.0.v201112011016.jar org.fusesource.leveldbjni_leveldbjni-all-1.8.jar org.hamcrest_hamcrest-core-1.3.jar org.iq80.snappy_snappy-0.2.jar org.jruby.jcodings_jcodings-1.0.8.jar org.jruby.joni_joni-2.1.2.jar org.json_json-20090211.jar org.mortbay.jetty_jetty-6.1.26.jar org.mortbay.jetty_jetty-util-6.1.26.jar org.slf4j_slf4j-api-1.7.10.jar org.slf4j_slf4j-log4j12-1.7.10.jar org.sonatype.sisu.inject_cglib-2.2.1-v20090111.jar org.xerial.snappy_snappy-java-1.0.5.jar oro_oro-2.0.8.jar stax_stax-api-1.0.1.jar tomcat_jasper-compiler-5.5.23.jar tomcat_jasper-runtime-5.5.23.jar xerces_xercesImpl-2.9.1.jar xml-apis_xml-apis-1.3.04.jar xmlenc_xmlenc-0.52.jar 
// All JARs are copied to the folder called "hive_2_1_1_metastore_jars" in DBFS.
display(dbutils.fs.ls("dbfs:/hive_2_1_1_metastore_jars"))
dbfs:/hive_2_1_1_metastore_jars/antlr_antlr-2.7.7.jarantlr_antlr-2.7.7.jar445288
dbfs:/hive_2_1_1_metastore_jars/aopalliance_aopalliance-1.0.jaraopalliance_aopalliance-1.0.jar4467
dbfs:/hive_2_1_1_metastore_jars/asm_asm-3.2.jarasm_asm-3.2.jar43398
dbfs:/hive_2_1_1_metastore_jars/asm_asm-commons-3.1.jarasm_asm-commons-3.1.jar32693
dbfs:/hive_2_1_1_metastore_jars/asm_asm-tree-3.1.jarasm_asm-tree-3.1.jar21879
dbfs:/hive_2_1_1_metastore_jars/co.cask.tephra_tephra-api-0.6.0.jarco.cask.tephra_tephra-api-0.6.0.jar8836
dbfs:/hive_2_1_1_metastore_jars/co.cask.tephra_tephra-core-0.6.0.jarco.cask.tephra_tephra-core-0.6.0.jar635211
dbfs:/hive_2_1_1_metastore_jars/co.cask.tephra_tephra-hbase-compat-1.0-0.6.0.jarco.cask.tephra_tephra-hbase-compat-1.0-0.6.0.jar28216
dbfs:/hive_2_1_1_metastore_jars/com.fasterxml.jackson.core_jackson-annotations-2.4.0.jarcom.fasterxml.jackson.core_jackson-annotations-2.4.0.jar38605
dbfs:/hive_2_1_1_metastore_jars/com.fasterxml.jackson.core_jackson-core-2.4.2.jarcom.fasterxml.jackson.core_jackson-core-2.4.2.jar225316
dbfs:/hive_2_1_1_metastore_jars/com.fasterxml.jackson.core_jackson-databind-2.4.2.jarcom.fasterxml.jackson.core_jackson-databind-2.4.2.jar1075759
dbfs:/hive_2_1_1_metastore_jars/com.github.joshelser_dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jarcom.github.joshelser_dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar15935
dbfs:/hive_2_1_1_metastore_jars/com.github.stephenc.findbugs_findbugs-annotations-1.3.9-1.jarcom.github.stephenc.findbugs_findbugs-annotations-1.3.9-1.jar15322
dbfs:/hive_2_1_1_metastore_jars/com.google.code.findbugs_jsr305-3.0.0.jarcom.google.code.findbugs_jsr305-3.0.0.jar33031
dbfs:/hive_2_1_1_metastore_jars/com.google.code.gson_gson-2.2.4.jarcom.google.code.gson_gson-2.2.4.jar190432
dbfs:/hive_2_1_1_metastore_jars/com.google.guava_guava-14.0.1.jarcom.google.guava_guava-14.0.1.jar2189117
dbfs:/hive_2_1_1_metastore_jars/com.google.inject.extensions_guice-assistedinject-3.0.jarcom.google.inject.extensions_guice-assistedinject-3.0.jar36998
dbfs:/hive_2_1_1_metastore_jars/com.google.inject.extensions_guice-servlet-3.0.jarcom.google.inject.extensions_guice-servlet-3.0.jar65012
dbfs:/hive_2_1_1_metastore_jars/com.google.inject_guice-3.0.jarcom.google.inject_guice-3.0.jar710492
dbfs:/hive_2_1_1_metastore_jars/com.google.protobuf_protobuf-java-2.5.0.jarcom.google.protobuf_protobuf-java-2.5.0.jar533455
dbfs:/hive_2_1_1_metastore_jars/com.jamesmurty.utils_java-xmlbuilder-0.4.jarcom.jamesmurty.utils_java-xmlbuilder-0.4.jar18490
dbfs:/hive_2_1_1_metastore_jars/com.jcraft_jsch-0.1.42.jarcom.jcraft_jsch-0.1.42.jar185746
dbfs:/hive_2_1_1_metastore_jars/com.jolbox_bonecp-0.8.0.RELEASE.jarcom.jolbox_bonecp-0.8.0.RELEASE.jar110600
dbfs:/hive_2_1_1_metastore_jars/com.sun.jersey.contribs_jersey-guice-1.9.jarcom.sun.jersey.contribs_jersey-guice-1.9.jar14786
dbfs:/hive_2_1_1_metastore_jars/com.sun.jersey_jersey-client-1.9.jarcom.sun.jersey_jersey-client-1.9.jar130458
dbfs:/hive_2_1_1_metastore_jars/com.sun.jersey_jersey-core-1.14.jarcom.sun.jersey_jersey-core-1.14.jar466989
dbfs:/hive_2_1_1_metastore_jars/com.sun.jersey_jersey-json-1.14.jarcom.sun.jersey_jersey-json-1.14.jar163559
dbfs:/hive_2_1_1_metastore_jars/com.sun.jersey_jersey-server-1.14.jarcom.sun.jersey_jersey-server-1.14.jar600146
dbfs:/hive_2_1_1_metastore_jars/com.sun.xml.bind_jaxb-impl-2.2.3-1.jarcom.sun.xml.bind_jaxb-impl-2.2.3-1.jar890168
dbfs:/hive_2_1_1_metastore_jars/com.thoughtworks.paranamer_paranamer-2.3.jarcom.thoughtworks.paranamer_paranamer-2.3.jar29555
dbfs:/hive_2_1_1_metastore_jars/commons-beanutils_commons-beanutils-1.7.0.jarcommons-beanutils_commons-beanutils-1.7.0.jar188671
dbfs:/hive_2_1_1_metastore_jars/commons-beanutils_commons-beanutils-core-1.8.0.jarcommons-beanutils_commons-beanutils-core-1.8.0.jar206035
dbfs:/hive_2_1_1_metastore_jars/commons-cli_commons-cli-1.2.jarcommons-cli_commons-cli-1.2.jar41123
dbfs:/hive_2_1_1_metastore_jars/commons-codec_commons-codec-1.4.jarcommons-codec_commons-codec-1.4.jar58160
dbfs:/hive_2_1_1_metastore_jars/commons-collections_commons-collections-3.2.2.jarcommons-collections_commons-collections-3.2.2.jar588337
dbfs:/hive_2_1_1_metastore_jars/commons-configuration_commons-configuration-1.6.jarcommons-configuration_commons-configuration-1.6.jar298829
dbfs:/hive_2_1_1_metastore_jars/commons-daemon_commons-daemon-1.0.13.jarcommons-daemon_commons-daemon-1.0.13.jar24239
dbfs:/hive_2_1_1_metastore_jars/commons-dbcp_commons-dbcp-1.4.jarcommons-dbcp_commons-dbcp-1.4.jar160519
dbfs:/hive_2_1_1_metastore_jars/commons-digester_commons-digester-1.8.jarcommons-digester_commons-digester-1.8.jar143602
dbfs:/hive_2_1_1_metastore_jars/commons-el_commons-el-1.0.jarcommons-el_commons-el-1.0.jar112341
dbfs:/hive_2_1_1_metastore_jars/commons-httpclient_commons-httpclient-3.1.jarcommons-httpclient_commons-httpclient-3.1.jar305001
dbfs:/hive_2_1_1_metastore_jars/commons-io_commons-io-2.4.jarcommons-io_commons-io-2.4.jar185140
dbfs:/hive_2_1_1_metastore_jars/commons-lang_commons-lang-2.6.jarcommons-lang_commons-lang-2.6.jar284220
dbfs:/hive_2_1_1_metastore_jars/commons-logging_commons-logging-1.2.jarcommons-logging_commons-logging-1.2.jar61829
dbfs:/hive_2_1_1_metastore_jars/commons-net_commons-net-3.1.jarcommons-net_commons-net-3.1.jar273370
dbfs:/hive_2_1_1_metastore_jars/commons-pool_commons-pool-1.5.4.jarcommons-pool_commons-pool-1.5.4.jar96221
dbfs:/hive_2_1_1_metastore_jars/io.dropwizard.metrics_metrics-core-3.1.0.jario.dropwizard.metrics_metrics-core-3.1.0.jar111908
dbfs:/hive_2_1_1_metastore_jars/io.dropwizard.metrics_metrics-json-3.1.0.jario.dropwizard.metrics_metrics-json-3.1.0.jar15823
dbfs:/hive_2_1_1_metastore_jars/io.dropwizard.metrics_metrics-jvm-3.1.0.jario.dropwizard.metrics_metrics-jvm-3.1.0.jar35907
dbfs:/hive_2_1_1_metastore_jars/io.netty_netty-3.7.0.Final.jario.netty_netty-3.7.0.Final.jar1208356
dbfs:/hive_2_1_1_metastore_jars/io.netty_netty-all-4.0.23.Final.jario.netty_netty-all-4.0.23.Final.jar1779991
dbfs:/hive_2_1_1_metastore_jars/it.unimi.dsi_fastutil-6.5.6.jarit.unimi.dsi_fastutil-6.5.6.jar16898038
dbfs:/hive_2_1_1_metastore_jars/javax.activation_activation-1.1.jarjavax.activation_activation-1.1.jar62983
dbfs:/hive_2_1_1_metastore_jars/javax.inject_javax.inject-1.jarjavax.inject_javax.inject-1.jar2497
dbfs:/hive_2_1_1_metastore_jars/javax.jdo_jdo-api-3.0.1.jarjavax.jdo_jdo-api-3.0.1.jar201124
dbfs:/hive_2_1_1_metastore_jars/javax.mail_mail-1.4.1.jarjavax.mail_mail-1.4.1.jar447676
dbfs:/hive_2_1_1_metastore_jars/javax.servlet.jsp_jsp-api-2.1.jarjavax.servlet.jsp_jsp-api-2.1.jar100636
dbfs:/hive_2_1_1_metastore_jars/javax.servlet_servlet-api-2.5.jarjavax.servlet_servlet-api-2.5.jar105112
dbfs:/hive_2_1_1_metastore_jars/javax.transaction_jta-1.1.jarjavax.transaction_jta-1.1.jar15071
dbfs:/hive_2_1_1_metastore_jars/javax.transaction_transaction-api-1.1.jarjavax.transaction_transaction-api-1.1.jar15071
dbfs:/hive_2_1_1_metastore_jars/javax.xml.bind_jaxb-api-2.2.2.jarjavax.xml.bind_jaxb-api-2.2.2.jar105134
dbfs:/hive_2_1_1_metastore_jars/javax.xml.stream_stax-api-1.0-2.jarjavax.xml.stream_stax-api-1.0-2.jar23346
dbfs:/hive_2_1_1_metastore_jars/javolution_javolution-5.5.1.jarjavolution_javolution-5.5.1.jar395195
dbfs:/hive_2_1_1_metastore_jars/jline_jline-2.12.jarjline_jline-2.12.jar213854
dbfs:/hive_2_1_1_metastore_jars/joda-time_joda-time-2.5.jarjoda-time_joda-time-2.5.jar588001
dbfs:/hive_2_1_1_metastore_jars/junit_junit-4.11.jarjunit_junit-4.11.jar245039
dbfs:/hive_2_1_1_metastore_jars/log4j_log4j-1.2.17.jarlog4j_log4j-1.2.17.jar489884
dbfs:/hive_2_1_1_metastore_jars/net.hydromatic_eigenbase-properties-1.1.5.jarnet.hydromatic_eigenbase-properties-1.1.5.jar18482
dbfs:/hive_2_1_1_metastore_jars/net.java.dev.jets3t_jets3t-0.9.0.jarnet.java.dev.jets3t_jets3t-0.9.0.jar539735
dbfs:/hive_2_1_1_metastore_jars/net.sf.opencsv_opencsv-2.3.jarnet.sf.opencsv_opencsv-2.3.jar19827
dbfs:/hive_2_1_1_metastore_jars/org.antlr_ST4-4.0.4.jarorg.antlr_ST4-4.0.4.jar236660
dbfs:/hive_2_1_1_metastore_jars/org.antlr_antlr-runtime-3.4.jarorg.antlr_antlr-runtime-3.4.jar164368
dbfs:/hive_2_1_1_metastore_jars/org.antlr_stringtemplate-3.2.1.jarorg.antlr_stringtemplate-3.2.1.jar148627
dbfs:/hive_2_1_1_metastore_jars/org.apache.ant_ant-1.9.1.jarorg.apache.ant_ant-1.9.1.jar1997485
dbfs:/hive_2_1_1_metastore_jars/org.apache.ant_ant-launcher-1.9.1.jarorg.apache.ant_ant-launcher-1.9.1.jar18336
dbfs:/hive_2_1_1_metastore_jars/org.apache.avro_avro-1.7.7.jarorg.apache.avro_avro-1.7.7.jar436303
dbfs:/hive_2_1_1_metastore_jars/org.apache.calcite_calcite-avatica-1.6.0.jarorg.apache.calcite_calcite-avatica-1.6.0.jar3306158
dbfs:/hive_2_1_1_metastore_jars/org.apache.calcite_calcite-core-1.6.0.jarorg.apache.calcite_calcite-core-1.6.0.jar3839798
dbfs:/hive_2_1_1_metastore_jars/org.apache.calcite_calcite-linq4j-1.6.0.jarorg.apache.calcite_calcite-linq4j-1.6.0.jar474924
dbfs:/hive_2_1_1_metastore_jars/org.apache.commons_commons-compress-1.9.jarorg.apache.commons_commons-compress-1.9.jar378217
dbfs:/hive_2_1_1_metastore_jars/org.apache.commons_commons-lang3-3.1.jarorg.apache.commons_commons-lang3-3.1.jar315805
dbfs:/hive_2_1_1_metastore_jars/org.apache.commons_commons-math3-3.1.1.jarorg.apache.commons_commons-math3-3.1.1.jar1599627
dbfs:/hive_2_1_1_metastore_jars/org.apache.derby_derby-10.10.2.0.jarorg.apache.derby_derby-10.10.2.0.jar2838580
dbfs:/hive_2_1_1_metastore_jars/org.apache.directory.api_api-asn1-api-1.0.0-M20.jarorg.apache.directory.api_api-asn1-api-1.0.0-M20.jar16560
dbfs:/hive_2_1_1_metastore_jars/org.apache.directory.api_api-util-1.0.0-M20.jarorg.apache.directory.api_api-util-1.0.0-M20.jar79912
dbfs:/hive_2_1_1_metastore_jars/org.apache.directory.server_apacheds-i18n-2.0.0-M15.jarorg.apache.directory.server_apacheds-i18n-2.0.0-M15.jar44925
dbfs:/hive_2_1_1_metastore_jars/org.apache.directory.server_apacheds-kerberos-codec-2.0.0-M15.jarorg.apache.directory.server_apacheds-kerberos-codec-2.0.0-M15.jar691479
dbfs:/hive_2_1_1_metastore_jars/org.apache.geronimo.specs_geronimo-annotation_1.0_spec-1.1.1.jarorg.apache.geronimo.specs_geronimo-annotation_1.0_spec-1.1.1.jar12452
dbfs:/hive_2_1_1_metastore_jars/org.apache.geronimo.specs_geronimo-jaspic_1.0_spec-1.0.jarorg.apache.geronimo.specs_geronimo-jaspic_1.0_spec-1.0.jar30548
dbfs:/hive_2_1_1_metastore_jars/org.apache.geronimo.specs_geronimo-jta_1.1_spec-1.1.1.jarorg.apache.geronimo.specs_geronimo-jta_1.1_spec-1.1.1.jar16030
dbfs:/hive_2_1_1_metastore_jars/org.apache.hadoop_hadoop-annotations-2.7.3.jarorg.apache.hadoop_hadoop-annotations-2.7.3.jar40863
dbfs:/hive_2_1_1_metastore_jars/org.apache.hadoop_hadoop-auth-2.7.3.jarorg.apache.hadoop_hadoop-auth-2.7.3.jar94150
dbfs:/hive_2_1_1_metastore_jars/org.apache.hadoop_hadoop-client-2.7.3.jarorg.apache.hadoop_hadoop-client-2.7.3.jar26012
dbfs:/hive_2_1_1_metastore_jars/org.apache.hadoop_hadoop-common-2.7.3.jarorg.apache.hadoop_hadoop-common-2.7.3.jar3479293
dbfs:/hive_2_1_1_metastore_jars/org.apache.hadoop_hadoop-hdfs-2.7.3.jarorg.apache.hadoop_hadoop-hdfs-2.7.3.jar8316190
dbfs:/hive_2_1_1_metastore_jars/org.apache.hadoop_hadoop-mapreduce-client-app-2.7.3.jarorg.apache.hadoop_hadoop-mapreduce-client-app-2.7.3.jar542869
dbfs:/hive_2_1_1_metastore_jars/org.apache.hadoop_hadoop-mapreduce-client-common-2.7.3.jarorg.apache.hadoop_hadoop-mapreduce-client-common-2.7.3.jar776634
dbfs:/hive_2_1_1_metastore_jars/org.apache.hadoop_hadoop-mapreduce-client-core-2.7.3.jarorg.apache.hadoop_hadoop-mapreduce-client-core-2.7.3.jar1556539
dbfs:/hive_2_1_1_metastore_jars/org.apache.hadoop_hadoop-mapreduce-client-jobclient-2.7.3.jarorg.apache.hadoop_hadoop-mapreduce-client-jobclient-2.7.3.jar62304
dbfs:/hive_2_1_1_metastore_jars/org.apache.hadoop_hadoop-mapreduce-client-shuffle-2.7.3.jarorg.apache.hadoop_hadoop-mapreduce-client-shuffle-2.7.3.jar71737

Step 4: Replace datanucleus-rdbms 4.1.7 with datanucleus-rdbms 4.1.16, which is the version of datanucleus-rdbms that has the fix for SQL Server-based metastores.

Delete the datanucleus-rdbms 4.1.7 JAR.

%sh ls /dbfs/hive_2_1_1_metastore_jars | grep datanucleus-rdbms
org.datanucleus_datanucleus-rdbms-4.1.7.jar 
%sh rm /dbfs/hive_2_1_1_metastore_jars/org.datanucleus_datanucleus-rdbms-4.1.7.jar

Download the datanucleus-rdbms 4.1.16 JAR.

%sh wget -O /dbfs/hive_2_1_1_metastore_jars/datanucleus-rdbms-4.1.16.jar https://search.maven.org/remotecontent?filepath=org/datanucleus/datanucleus-rdbms/4.1.16/datanucleus-rdbms-4.1.16.jar
--2018-01-21 05:07:45-- https://search.maven.org/remotecontent?filepath=org/datanucleus/datanucleus-rdbms/4.1.16/datanucleus-rdbms-4.1.16.jar Resolving search.maven.org (search.maven.org)... 34.238.86.232, 52.55.93.33, 54.84.74.32 Connecting to search.maven.org (search.maven.org)|34.238.86.232|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 1906399 (1.8M) [application/java-archive] Saving to: ‘/dbfs/hive_2_1_1_metastore_jars/datanucleus-rdbms-4.1.16.jar’ 0K .......... .......... .......... .......... .......... 2% 686K 3s 50K .......... .......... .......... .......... .......... 5% 751K 2s 100K .......... .......... .......... .......... .......... 8% 4.39M 2s 150K .......... .......... .......... .......... .......... 10% 3.91M 1s 200K .......... .......... .......... .......... .......... 13% 910K 1s 250K .......... .......... .......... .......... .......... 16% 4.24M 1s 300K .......... .......... .......... .......... .......... 18% 4.50M 1s 350K .......... .......... .......... .......... .......... 21% 1.08M 1s 400K .......... .......... .......... .......... .......... 24% 3.16M 1s 450K .......... .......... .......... .......... .......... 26% 2.58M 1s 500K .......... .......... .......... .......... .......... 29% 4.36M 1s 550K .......... .......... .......... .......... .......... 32% 4.15M 1s 600K .......... .......... .......... .......... .......... 34% 4.76M 1s 650K .......... .......... .......... .......... .......... 37% 4.95M 1s 700K .......... .......... .......... .......... .......... 40% 5.43M 1s 750K .......... .......... .......... .......... .......... 42% 4.25M 1s 800K .......... .......... .......... .......... .......... 45% 5.92M 0s 850K .......... .......... .......... .......... .......... 48% 5.14M 0s 900K .......... .......... .......... .......... .......... 51% 4.97M 0s 950K .......... .......... .......... .......... .......... 53% 4.38M 0s 1000K .......... .......... .......... .......... .......... 56% 4.53M 0s 1050K .......... .......... .......... .......... .......... 59% 5.85M 0s 1100K .......... .......... .......... .......... .......... 61% 5.23M 0s 1150K .......... .......... .......... .......... .......... 64% 4.33M 0s 1200K .......... .......... .......... .......... .......... 67% 5.64M 0s 1250K .......... .......... .......... .......... .......... 69% 5.16M 0s 1300K .......... .......... .......... .......... .......... 72% 4.68M 0s 1350K .......... .......... .......... .......... .......... 75% 4.46M 0s 1400K .......... .......... .......... .......... .......... 77% 5.45M 0s 1450K .......... .......... .......... .......... .......... 80% 5.16M 0s 1500K .......... .......... .......... .......... .......... 83% 4.42M 0s 1550K .......... .......... .......... .......... .......... 85% 2.62M 0s 1600K .......... .......... .......... .......... .......... 88% 2.68M 0s 1650K .......... .......... .......... .......... .......... 91% 3.71M 0s 1700K .......... .......... .......... .......... .......... 93% 3.87M 0s 1750K .......... .......... .......... .......... .......... 96% 5.45M 0s 1800K .......... .......... .......... .......... .......... 99% 3.78M 0s 1850K .......... . 100% 8.29M=0.6s 2018-01-21 05:07:46 (2.95 MB/s) - ‘/dbfs/hive_2_1_1_metastore_jars/datanucleus-rdbms-4.1.16.jar’ saved [1906399/1906399] 
// Make sure the file has actually been downloaded to the right place
display(dbutils.fs.ls("dbfs:/hive_2_1_1_metastore_jars/datanucleus-rdbms-4.1.16.jar"))
dbfs:/hive_2_1_1_metastore_jars/datanucleus-rdbms-4.1.16.jardatanucleus-rdbms-4.1.16.jar1906399

Step 5: Create an init script to use metastore JARs stored in DBFS

  1. Create the base directory you want to store the init script in if it does not exist. The following example uses dbfs:/databricks/scripts.
  2. Run the following command to create the init script.
// This example is for an init script named `external-metastore.sh`.
dbutils.fs.put(
    "/databricks/scripts/external-metastore.sh",
    """#!/bin/sh
      |# A temporary workaround to make sure /dbfs is available.
      |sleep 10
      |# Copy metastore jars from DBFS to the local FileSystem of every node.
      |cp -r /dbfs/hive_2_1_1_metastore_jars /databricks/hive_2_1_1_metastore_jars
      |# Loads environment variables to determine the correct JDBC driver to use.
      |source /etc/environment
      |# Quoting the label (i.e. EOF) with single quotes to disable variable interpolation.
      |cat << 'EOF' > /databricks/driver/conf/00-custom-spark.conf
      |[driver] {
      |    # Hive specific configuration options.
      |    # spark.hadoop prefix is added to make sure these Hive specific options will propagate to the metastore client.
      |    # JDBC connect string for a JDBC metastore
      |    "spark.hadoop.javax.jdo.option.ConnectionURL" = "${mssql-connection-string}"
      |
      |    # Username to use against metastore database
      |    "spark.hadoop.javax.jdo.option.ConnectionUserName" = "${mssql-username}"
      |
      |    # Password to use against metastore database
      |    "spark.hadoop.javax.jdo.option.ConnectionPassword" = "${mssql-password}"
      |
      |    # Driver class name for a JDBC metastore
      |    "spark.hadoop.javax.jdo.option.ConnectionDriverName" = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
      |
      |    # Spark specific configuration options
      |    "spark.sql.hive.metastore.version" = "2.1.1"
      |    # Skip this one if ${hive-version} is 0.13.x.
      |    "spark.sql.hive.metastore.jars" = "/databricks/hive_2_1_1_metastore_jars/*"
      |}
      |EOF
      |""".stripMargin,
    overwrite = true
)
  1. Configure a cluster with the init script.
  2. Restart the cluster.

Step 6: Check that the JARs and configurations are correct

Make sure metastore JARs have been copied from DBFS to the local file system of the driver node. You should see datanucleus-rdbms-4.1.16.jar in the output.

%sh 
ls /databricks/hive_2_1_1_metastore_jars
antlr_antlr-2.7.7.jar aopalliance_aopalliance-1.0.jar asm_asm-3.2.jar asm_asm-commons-3.1.jar asm_asm-tree-3.1.jar co.cask.tephra_tephra-api-0.6.0.jar co.cask.tephra_tephra-core-0.6.0.jar co.cask.tephra_tephra-hbase-compat-1.0-0.6.0.jar com.fasterxml.jackson.core_jackson-annotations-2.4.0.jar com.fasterxml.jackson.core_jackson-core-2.4.2.jar com.fasterxml.jackson.core_jackson-databind-2.4.2.jar com.github.joshelser_dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar com.github.stephenc.findbugs_findbugs-annotations-1.3.9-1.jar com.google.code.findbugs_jsr305-3.0.0.jar com.google.code.gson_gson-2.2.4.jar com.google.guava_guava-14.0.1.jar com.google.inject.extensions_guice-assistedinject-3.0.jar com.google.inject.extensions_guice-servlet-3.0.jar com.google.inject_guice-3.0.jar com.google.protobuf_protobuf-java-2.5.0.jar com.jamesmurty.utils_java-xmlbuilder-0.4.jar com.jcraft_jsch-0.1.42.jar com.jolbox_bonecp-0.8.0.RELEASE.jar commons-beanutils_commons-beanutils-1.7.0.jar commons-beanutils_commons-beanutils-core-1.8.0.jar commons-cli_commons-cli-1.2.jar commons-codec_commons-codec-1.4.jar commons-collections_commons-collections-3.2.2.jar commons-configuration_commons-configuration-1.6.jar commons-daemon_commons-daemon-1.0.13.jar commons-dbcp_commons-dbcp-1.4.jar commons-digester_commons-digester-1.8.jar commons-el_commons-el-1.0.jar commons-httpclient_commons-httpclient-3.1.jar commons-io_commons-io-2.4.jar commons-lang_commons-lang-2.6.jar commons-logging_commons-logging-1.2.jar commons-net_commons-net-3.1.jar commons-pool_commons-pool-1.5.4.jar com.sun.jersey.contribs_jersey-guice-1.9.jar com.sun.jersey_jersey-client-1.9.jar com.sun.jersey_jersey-core-1.14.jar com.sun.jersey_jersey-json-1.14.jar com.sun.jersey_jersey-server-1.14.jar com.sun.xml.bind_jaxb-impl-2.2.3-1.jar com.thoughtworks.paranamer_paranamer-2.3.jar datanucleus-rdbms-4.1.16.jar io.dropwizard.metrics_metrics-core-3.1.0.jar io.dropwizard.metrics_metrics-json-3.1.0.jar io.dropwizard.metrics_metrics-jvm-3.1.0.jar io.netty_netty-3.7.0.Final.jar io.netty_netty-all-4.0.23.Final.jar it.unimi.dsi_fastutil-6.5.6.jar javax.activation_activation-1.1.jar javax.inject_javax.inject-1.jar javax.jdo_jdo-api-3.0.1.jar javax.mail_mail-1.4.1.jar javax.servlet.jsp_jsp-api-2.1.jar javax.servlet_servlet-api-2.5.jar javax.transaction_jta-1.1.jar javax.transaction_transaction-api-1.1.jar javax.xml.bind_jaxb-api-2.2.2.jar javax.xml.stream_stax-api-1.0-2.jar javolution_javolution-5.5.1.jar jline_jline-2.12.jar joda-time_joda-time-2.5.jar junit_junit-4.11.jar log4j_log4j-1.2.17.jar net.hydromatic_eigenbase-properties-1.1.5.jar net.java.dev.jets3t_jets3t-0.9.0.jar net.sf.opencsv_opencsv-2.3.jar org.antlr_antlr-runtime-3.4.jar org.antlr_ST4-4.0.4.jar org.antlr_stringtemplate-3.2.1.jar org.apache.ant_ant-1.9.1.jar org.apache.ant_ant-launcher-1.9.1.jar org.apache.avro_avro-1.7.7.jar org.apache.calcite_calcite-avatica-1.6.0.jar org.apache.calcite_calcite-core-1.6.0.jar org.apache.calcite_calcite-linq4j-1.6.0.jar org.apache.commons_commons-compress-1.9.jar org.apache.commons_commons-lang3-3.1.jar org.apache.commons_commons-math3-3.1.1.jar org.apache.derby_derby-10.10.2.0.jar org.apache.directory.api_api-asn1-api-1.0.0-M20.jar org.apache.directory.api_api-util-1.0.0-M20.jar org.apache.directory.server_apacheds-i18n-2.0.0-M15.jar org.apache.directory.server_apacheds-kerberos-codec-2.0.0-M15.jar org.apache.geronimo.specs_geronimo-annotation_1.0_spec-1.1.1.jar org.apache.geronimo.specs_geronimo-jaspic_1.0_spec-1.0.jar org.apache.geronimo.specs_geronimo-jta_1.1_spec-1.1.1.jar org.apache.hadoop_hadoop-annotations-2.7.3.jar org.apache.hadoop_hadoop-auth-2.7.3.jar org.apache.hadoop_hadoop-client-2.7.3.jar org.apache.hadoop_hadoop-common-2.7.3.jar org.apache.hadoop_hadoop-hdfs-2.7.3.jar org.apache.hadoop_hadoop-mapreduce-client-app-2.7.3.jar org.apache.hadoop_hadoop-mapreduce-client-common-2.7.3.jar org.apache.hadoop_hadoop-mapreduce-client-core-2.7.3.jar org.apache.hadoop_hadoop-mapreduce-client-jobclient-2.7.3.jar org.apache.hadoop_hadoop-mapreduce-client-shuffle-2.7.3.jar org.apache.hadoop_hadoop-yarn-api-2.7.3.jar org.apache.hadoop_hadoop-yarn-client-2.7.3.jar org.apache.hadoop_hadoop-yarn-common-2.7.3.jar org.apache.hadoop_hadoop-yarn-server-applicationhistoryservice-2.6.1.jar org.apache.hadoop_hadoop-yarn-server-common-2.7.3.jar org.apache.hadoop_hadoop-yarn-server-resourcemanager-2.6.1.jar org.apache.hadoop_hadoop-yarn-server-web-proxy-2.6.1.jar org.apache.hbase_hbase-annotations-1.1.1.jar org.apache.hbase_hbase-client-1.1.1.jar org.apache.hbase_hbase-protocol-1.1.1.jar org.apache.hive_hive-ant-2.1.1.jar org.apache.hive_hive-common-2.1.1.jar org.apache.hive_hive-exec-2.1.1.jar org.apache.hive_hive-llap-client-2.1.1.jar org.apache.hive_hive-llap-common-2.1.1.jar org.apache.hive_hive-llap-tez-2.1.1.jar org.apache.hive_hive-metastore-2.1.1.jar org.apache.hive_hive-orc-2.1.1.jar org.apache.hive_hive-serde-2.1.1.jar org.apache.hive_hive-service-rpc-2.1.1.jar org.apache.hive_hive-shims-2.1.1.jar org.apache.hive_hive-storage-api-2.1.1.jar org.apache.hive.shims_hive-shims-0.23-2.1.1.jar org.apache.hive.shims_hive-shims-common-2.1.1.jar org.apache.hive.shims_hive-shims-scheduler-2.1.1.jar org.apache.htrace_htrace-core-3.1.0-incubating.jar org.apache.httpcomponents_httpclient-4.4.jar org.apache.httpcomponents_httpcore-4.4.jar org.apache.ivy_ivy-2.4.0.jar org.apache.logging.log4j_log4j-1.2-api-2.4.1.jar org.apache.logging.log4j_log4j-slf4j-impl-2.4.1.jar org.apache.logging.log4j_log4j-web-2.4.1.jar org.apache.parquet_parquet-hadoop-bundle-1.8.1.jar org.apache.thrift_libfb303-0.9.3.jar org.apache.thrift_libthrift-0.9.3.jar org.apache.twill_twill-api-0.6.0-incubating.jar org.apache.twill_twill-common-0.6.0-incubating.jar org.apache.twill_twill-core-0.6.0-incubating.jar org.apache.twill_twill-discovery-api-0.6.0-incubating.jar org.apache.twill_twill-discovery-core-0.6.0-incubating.jar org.apache.twill_twill-zookeeper-0.6.0-incubating.jar org.apache.velocity_velocity-1.5.jar org.apache.zookeeper_zookeeper-3.4.6.jar org.codehaus.groovy_groovy-all-2.4.4.jar org.codehaus.jackson_jackson-core-asl-1.9.13.jar org.codehaus.jackson_jackson-jaxrs-1.9.13.jar org.codehaus.jackson_jackson-mapper-asl-1.9.13.jar org.codehaus.jackson_jackson-xc-1.9.13.jar org.codehaus.janino_commons-compiler-2.7.6.jar org.codehaus.janino_janino-2.7.6.jar org.codehaus.jettison_jettison-1.1.jar org.datanucleus_datanucleus-api-jdo-4.2.1.jar org.datanucleus_datanucleus-core-4.1.6.jar org.datanucleus_javax.jdo-3.2.0-m3.jar org.eclipse.jetty.aggregate_jetty-all-7.6.0.v20120127.jar org.eclipse.jetty.orbit_javax.servlet-3.0.0.v201112011016.jar org.fusesource.leveldbjni_leveldbjni-all-1.8.jar org.hamcrest_hamcrest-core-1.3.jar org.iq80.snappy_snappy-0.2.jar org.jruby.jcodings_jcodings-1.0.8.jar org.jruby.joni_joni-2.1.2.jar org.json_json-20090211.jar org.mortbay.jetty_jetty-6.1.26.jar org.mortbay.jetty_jetty-util-6.1.26.jar org.slf4j_slf4j-api-1.7.10.jar org.slf4j_slf4j-log4j12-1.7.10.jar org.sonatype.sisu.inject_cglib-2.2.1-v20090111.jar org.xerial.snappy_snappy-java-1.0.5.jar oro_oro-2.0.8.jar stax_stax-api-1.0.1.jar tomcat_jasper-compiler-5.5.23.jar tomcat_jasper-runtime-5.5.23.jar xerces_xercesImpl-2.9.1.jar xml-apis_xml-apis-1.3.04.jar xmlenc_xmlenc-0.52.jar

Check the logs to make sure spark.sql.hive.metastore.jars and spark.sql.hive.metastore.version are set correctly.

%sh grep "spark\.sql\.hive\.metastore" /databricks/driver/logs/log4j-active.log
18/01/21 05:38:54 INFO SparkConfUtils$: Set spark config: spark.sql.hive.metastore.sharedPrefixes -> org.mariadb.jdbc,com.mysql.jdbc,org.postgresql,com.microsoft.sqlserver,microsoft.sql.DateTimeOffset,microsoft.sql.Types,com.databricks,com.codahale,com.fasterxml.jackson,shaded.databricks.org.apache.hadoop.fs.azure 18/01/21 05:38:55 INFO SparkConfUtils$: Set spark config: spark.sql.hive.metastore.sharedPrefixes -> org.mariadb.jdbc,com.mysql.jdbc,org.postgresql,com.microsoft.sqlserver,microsoft.sql.DateTimeOffset,microsoft.sql.Types,com.databricks,com.codahale,com.fasterxml.jackson,shaded.databricks.org.apache.hadoop.fs.azure spark.sql.hive.metastore.jars=/databricks/hive_2_1_1_metastore_jars/* spark.sql.hive.metastore.sharedPrefixes=org.mariadb.jdbc,com.mysql.jdbc,org.postgresql,com.microsoft.sqlserver,microsoft.sql.DateTimeOffset,microsoft.sql.Types,com.databricks,com.codahale,com.fasterxml.jackson,shaded.databricks.org.apache.hadoop.fs.azure spark.sql.hive.metastore.version=2.1.1 18/01/21 05:39:10 INFO SparkConfUtils$: Set spark config: spark.sql.hive.metastore.sharedPrefixes -> org.mariadb.jdbc,com.mysql.jdbc,org.postgresql,com.microsoft.sqlserver,microsoft.sql.DateTimeOffset,microsoft.sql.Types,com.databricks,com.codahale,com.fasterxml.jackson,shaded.databricks.org.apache.hadoop.fs.azure

Step 7: Test dropping the table (drop table will fail if the version of datanucleus-rdbms is 4.1.7).

// Write a table
spark.range(100).write.format("json").saveAsTable("test_metastore")

%sql select * from test_metastore
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
// Drop the table
spark.sql("drop table test_metastore")
res7: org.apache.spark.sql.DataFrame = [] 
%sql show tables
OK