2.0.2-db2 cluster image

Databricks released this image in December, 2016.

Important

This release has been deprecated. For more information about the Databricks Runtime deprecation policy and schedule, see Databricks support lifecycles.

The following release notes provide information about the Spark 2.0.2-db2 cluster image powered by Apache Spark.

Apache Spark

2.0.2-db2 cluster image includes Apache Spark 2.0.2. Apache Spark 2.0.2 contains stability fixes, Apache Kafka support for Structured Streaming, and improved metrics for Structured Streaming. For more information, please see Apache Spark 2.0.2 release notes. 2.0.2-db2 cluster image also includes the following extra bug fixes and improvements:

Changes and Improvements

2.0.2-db2 image adds the fix of SPARK-18553 on top of 2.0.2-db1.

System Environment

  • Operating System: Ubuntu 16.04.1 LTS

  • Java: 1.8.0_111

  • Scala: 2.10.6 (Scala 2.10 cluster version)/2.11.8 (Scala 2.11 cluster version)

  • Python: 2.7.12

  • R: R version 3.2.3 (2015-12-10)

Pre-installed Python Libraries

Library

Version

Library

Version

Library

Version

ansi2html

1.1.1

argparse

1.2.1

boto

2.42.0

boto3

1.4.1

botocore

1.4.70

brewer2mpl

1.4.1

certifi

2016.2.28

cffi

1.7.0

chardet

2.3.0

colorama

0.3.7

configobj

5.0.6

cryptography

1.5

cycler

0.10.0

Cython

0.24.1

decorator

4.0.10

docutils

0.12

enum34

1.1.6

et-xmlfile

1.0.1

freetype-py

1.0.2

funcsigs

1.0.2

fusepy

2.0.4

futures

3.0.5

ggplot

0.6.8

html5lib

0.999

idna

2.1

ipaddress

1.0.16

ipython

2.2.0

ipython-genutils

0.1.0

jdcal

1.2

Jinja2

2.8

jmespath

0.9.0

llvmlite

0.13.0

lxml

3.6.4

MarkupSafe

0.23

matplotlib

1.5.3

mpld3

0.2

msgpack-python

0.4.7

ndg-httpsclient

0.3.3

numba

0.28.1

numpy

1.11.1

openpyxl

2.3.2

pandas

0.18.1

pathlib2

2.1.0

patsy

0.4.1

pexpect

4.0.1

pickleshare

0.7.4

Pillow

3.3.1

pip

8.1.2

pkg_resources

0.0.0

ply

3.9

prompt-toolkit

1.0.7

psycopg2

2.6.2

ptyprocess

0.5.1

py4j

0.10.3

pyasn1

0.1.9

pycparser

2.14

Pygments

2.1.3

PyGObject

3.20.0

pyOpenSSL

16.0.0

pyparsing

2.1.4

pypng

0.0.18

Python

2.7.12

python-dateutil

2.5.3

python-geohash

0.8.5

pytz

2016.6.1

requests

2.11.1

s3transfer

0.1.9

scikit-learn

0.17.1

scipy

0.18.1

scour

0.32

seaborn

0.7.1

setuptools

28.6.0

simplejson

3.8.2

simples3

1.0

singledispatch

3.4.0.3

six

1.10.0

statsmodels

0.6.1

traitlets

4.3.0

urllib3

1.19.1

virtualenv

15.0.1

wcwidth

0.1.7

wheel

0.30.0a0

wsgiref

0.1.2

Pre-installed R Libraries

Library

Version

Library

Version

Library

Version

abind

1.4-3

assertthat

0.1

base

3.2.3

BH

1.60.0-2

bitops

1.0-6

boot

1.3-17

brew

1.0-6

car

2.1-3

caret

6.0-71

chron

2.3-47

class

7.3-14

cluster

2.0.5

codetools

0.2-14

colorspace

1.2-4

compiler

3.2.3

crayon

1.3.1

curl

2.2

data.table

1.9.6

datasets

3.2.3

DBI

0.5-1

devtools

1.12.0

dichromat

2.0-0

digest

0.6.9

doMC

1.3.4

dplyr

0.5.0

foreach

1.4.3

foreign

0.8-66

gbm

2.1.1

ggplot2

2.1.0

git2r

0.15.0

glmnet

2.0-5

graphics

3.2.3

grDevices

3.2.3

grid

3.2.3

gsubfn

0.6-6

gtable

0.1.2

h2o

3.10.0.8

httr

1.2.1

hwriter

1.3.2

hwriterPlus

1.0-3

iterators

1.0.8

jsonlite

1.1

KernSmooth

2.23-15

labeling

0.3

lattice

0.20-34

lazyeval

0.2.0

littler

0.3.0

lme4

1.1-12

lubridate

1.6.0

magrittr

1.5

mapproj

1.2-4

maps

3.0.2

MASS

7.3-45

Matrix

1.2-7.1

MatrixModels

0.4-1

memoise

1.0.0

methods

3.2.3

mgcv

1.8-11

mime

0.5

minqa

1.2.4

multicore

0.2

munsell

0.4.2

mvtnorm

1.0-5

nlme

3.1-124

nloptr

1.0.4

nnet

7.3-12

openssl

0.9.4

parallel

3.2.3

pbkrtest

0.4-6

pkgKitten

0.1.3

plyr

1.8.4

praise

1.0.0

pROC

1.8

proto

0.3-10

quantreg

5.29

R.methodsS3

1.7.1

R.oo

1.20.0

R.utils

2.4.0

R6

2.2.0

randomForest

4.6-12

RColorBrewer

1.1-2

Rcpp

0.12.7

RcppEigen

0.3.2.9.0

RCurl

1.95-4.8

reshape2

1.4.2

RODBC

1.3-12

roxygen2

5.0.1

rpart

4.1-10

Rserve

1.7-3

RSQLite

1.0.0

rstudioapi

0.6

scales

0.3.0

sp

1.0-15

SparkR

2.0.2

SparseM

1.72

spatial

7.3-11

splines

3.2.3

sqldf

0.4-10

statmod

1.4.26

stats

3.2.3

stats4

3.2.3

stringi

1.0-1

stringr

1.0.0

survival

2.38-3

tcltk

3.2.3

TeachingDemos

2.10

testthat

1.0.2

tibble

1.2

tools

3.2.3

utils

3.2.3

whisker

0.3-2

withr

1.0.2