Photon runtime

Photon is the native vectorized query engine on Databricks, written to be directly compatible with Apache Spark APIs so it works with your existing code. It is developed in C++ to take advantage of modern hardware, and uses the latest techniques in vectorized query processing to capitalize on data- and instruction-level parallelism in CPUs, enhancing performance on real-world data and applications-—all natively on your data lake. Photon is part of a high-performance runtime that runs your existing SQL and DataFrame API calls faster and reduces your total cost per workload. Photon is used by default in Databricks SQL warehouses.

Databricks clusters

Photon is available for clusters running Databricks Runtime 9.1 LTS and above.

To enable Photon acceleration, select the Use Photon Acceleration checkbox when you create the cluster. If you create the cluster using the clusters API, set runtime_engine to PHOTON.

Photon supports a number of instance types on the driver and worker nodes. Photon instance types consume DBUs at a different rate than the same instance type running the non-Photon runtime. For more information about Photon instances and DBU consumption, see the Databricks pricing page.

Photon advantages

  • Supports SQL and equivalent DataFrame operations against Delta and Parquet tables.

  • Accelerates queries that process a significant amount of data (100GB+) and include aggregations and joins.

  • Faster performance when data is accessed repeatedly from the disk cache.

  • More robust scan performance on tables with many columns and many small files.

  • Faster Delta and Parquet writing using UPDATE, DELETE, MERGE INTO, INSERT, and CREATE TABLE AS SELECT, especially for wide tables (hundreds to thousands of columns).

  • Replaces sort-merge joins with hash-joins.

Photon coverage

Operators

  • Scan, Filter, Project

  • Hash Aggregate/Join/Shuffle

  • Nested-Loop Join

  • Null-Aware Anti Join

  • Union, Expand, ScalarSubquery

  • Delta/Parquet Write Sink

  • Sort

  • Window Function

Expressions

  • Comparison / Logic

  • Arithmetic / Math (most)

  • Conditional (IF, CASE, etc.)

  • String (common ones)

  • Casts

  • Aggregates(most common ones)

  • Date/Timestamp

Data types

  • Byte/Short/Int/Long

  • Boolean

  • String/Binary

  • Decimal

  • Float/Double

  • Date/Timestamp

  • Struct

  • Array

  • Map

The following table lists supported Databricks expressions and the minimum Databricks Runtime release version that supports it.

Name

Release

Abs

Databricks Runtime 8.3

Acos

Databricks Runtime 10.4 LTS

Add

Databricks Runtime 8.3

AddMonths

Databricks Runtime 8.3

AesDecrypt

Databricks Runtime 10.4 LTS

AesEncrypt

Databricks Runtime 10.4 LTS

And

Databricks Runtime 8.3

ArrayContains

Databricks Runtime 8.3

ArrayDistinct

Databricks Runtime 10.0

ArrayExcept

Databricks Runtime 10.1

ArrayExists

Databricks Runtime 10.4 LTS

ArrayFilter

Databricks Runtime 10.4 LTS

ArrayForAll

Databricks Runtime 10.4 LTS

ArrayIntersect

Databricks Runtime 10.1

ArrayJoin

Databricks Runtime 10.4 LTS

ArraySize

Databricks Runtime 10.4 LTS

ArrayTransform

Databricks Runtime 10.4 LTS

ArrayUnion

Databricks Runtime 10.1

Atan

Databricks Runtime 9.1 LTS

Atan2

Databricks Runtime 9.1 LTS

Average

Databricks Runtime 8.3

Base64

Databricks Runtime 9.1 LTS

Bin

Databricks Runtime 10.0

BitAndAgg

Databricks Runtime 8.3

BitLength

Databricks Runtime 11.3 LTS

BitOrAgg

Databricks Runtime 8.3

BitwiseAnd

Databricks Runtime 8.3

BitwiseNot

Databricks Runtime 8.3

BitwiseOr

Databricks Runtime 8.3

BitwiseReverse

Databricks Runtime 8.3

BitwiseXor

Databricks Runtime 8.3

BitXorAgg

Databricks Runtime 8.3

BoundaryAsGeojson

Databricks Runtime 11.3 LTS

BoundaryAsWkb

Databricks Runtime 11.3 LTS

BoundaryAsWkt

Databricks Runtime 11.3 LTS

Cast

Databricks Runtime 8.3

Cbrt

Databricks Runtime 8.4

CeilExpressionBuilder

Databricks Runtime 8.3

CenterAsGeojson

Databricks Runtime 11.3 LTS

CenterAsWkb

Databricks Runtime 11.3 LTS

CenterAsWkt

Databricks Runtime 11.3 LTS

Chr

Databricks Runtime 10.1

Coalesce

Databricks Runtime 8.3

CollectList

Databricks Runtime 9.0

Concat

Databricks Runtime 8.3

ConcatWs

Databricks Runtime 8.3

Conv

Databricks Runtime 8.3

Cos

Databricks Runtime 10.4 LTS

Count

Databricks Runtime 8.3

CreateArray

Databricks Runtime 8.3

CreateMap

Databricks Runtime 8.4

CreateNamedStruct

Databricks Runtime 8.3

CreateStruct

Databricks Runtime 8.3

CurrentCatalog

Databricks Runtime 8.3

CurrentDatabase

Databricks Runtime 8.3

CurrentDate

Databricks Runtime 8.3

CurrentTimestamp

Databricks Runtime 8.3

CurrentTimeZone

Databricks Runtime 8.3

CurrentUser

Databricks Runtime 8.3

DateAdd

Databricks Runtime 8.3

DateDiff

Databricks Runtime 8.3

DateFormatClass

Databricks Runtime 8.3

DateFromUnixDate

Databricks Runtime 8.3

DateSub

Databricks Runtime 8.3

DayOfMonth

Databricks Runtime 8.3

DayOfWeek

Databricks Runtime 8.3

DayOfYear

Databricks Runtime 8.3

Decode

Databricks Runtime 8.3

DenseRank

Databricks Runtime 10.4 LTS

Divide

Databricks Runtime 8.3

ElementAt

Databricks Runtime 8.3

EqualNullSafe

Databricks Runtime 8.3

EqualTo

Databricks Runtime 8.3

Exp

Databricks Runtime 8.4

Explode

Databricks Runtime 8.4

Extract

Databricks Runtime 8.3

First

Databricks Runtime 8.3

FloorExpressionBuilder

Databricks Runtime 8.3

FromUnixTime

Databricks Runtime 8.3

FromUTCTimestamp

Databricks Runtime 8.3

Get

Databricks Runtime 11.3 LTS

GetJsonObject

Databricks Runtime 11.2

GreaterThan

Databricks Runtime 8.3

GreaterThanOrEqual

Databricks Runtime 8.3

Greatest

Databricks Runtime 8.3

GridDistance

Databricks Runtime 11.3 LTS

H3ToString

Databricks Runtime 11.3 LTS

Hex

Databricks Runtime 9.1 LTS

Hour

Databricks Runtime 8.3

If

Databricks Runtime 8.3

In

Databricks Runtime 8.3

InitCap

Databricks Runtime 11.3 LTS

InputFileBlockLength

Databricks Runtime 8.3

InputFileBlockStart

Databricks Runtime 8.3

InputFileName

Databricks Runtime 8.3

InSet

Databricks Runtime 8.3

IntegralDivide

Databricks Runtime 8.3

IsChildOf

Databricks Runtime 11.3 LTS

IsNaN

Databricks Runtime 8.3

IsNotNull

Databricks Runtime 8.3

IsNull

Databricks Runtime 8.3

IsPentagon

Databricks Runtime 11.3 LTS

IsValid

Databricks Runtime 11.3 LTS

JsonToStructs

Databricks Runtime 11.2

Lag

Databricks Runtime 10.4 LTS

Last

Databricks Runtime 10.4 LTS

LastDay

Databricks Runtime 8.3

Lead

Databricks Runtime 10.4 LTS

Least

Databricks Runtime 8.3

Length

Databricks Runtime 8.3

LengthOfJsonArray

Databricks Runtime 11.1

LessThan

Databricks Runtime 8.3

Levenshtein

Databricks Runtime 10.1

Like

Databricks Runtime 8.3

Log

Databricks Runtime 8.3

Log2

Databricks Runtime 8.4

LongLatAsH3

Databricks Runtime 11.3 LTS

LongLatAsH3String

Databricks Runtime 11.3 LTS

Lower

Databricks Runtime 8.3

LPadExpressionBuilder

Databricks Runtime 8.3

MakeDate

Databricks Runtime 8.3

MakeTimestamp

Databricks Runtime 8.3

Max

Databricks Runtime 8.3

MaxChild

Databricks Runtime 11.3 LTS

Md5

Databricks Runtime 10.4 LTS

MicrosToTimestamp

Databricks Runtime 8.3

MillisToTimestamp

Databricks Runtime 8.3

Min

Databricks Runtime 8.3

MinChild

Databricks Runtime 11.3 LTS

Minute

Databricks Runtime 8.3

MonotonicallyIncreasingID

Databricks Runtime 8.3

Month

Databricks Runtime 8.3

MonthsBetween

Databricks Runtime 8.3

Multiply

Databricks Runtime 8.3

Murmur3Hash

Databricks Runtime 8.3

NaNvl

Databricks Runtime 8.3

NextDay

Databricks Runtime 8.3

Not

Databricks Runtime 8.3

Now

Databricks Runtime 8.3

NthValue

Databricks Runtime 10.4 LTS

NTile

Databricks Runtime 10.4 LTS

NullIf

Databricks Runtime 8.3

Nvl

Databricks Runtime 8.3

Nvl2

Databricks Runtime 8.3

OctetLength

Databricks Runtime 8.3

ParseToDate

Databricks Runtime 8.3

ParseToTimestamp

Databricks Runtime 8.3

Percentile

Databricks Runtime 10.4 LTS

PercentRank

Databricks Runtime 10.4 LTS

Pi

Databricks Runtime 8.3

Pmod

Databricks Runtime 8.3

PosExplode

Databricks Runtime 9.1 LTS

Pow

Databricks Runtime 8.3

Quarter

Databricks Runtime 8.3

Rand

Databricks Runtime 8.3

Rank

Databricks Runtime 10.4 LTS

RegExpExtract

Databricks Runtime 8.3

RegExpExtractAll

Databricks Runtime 11.1

RegExpReplace

Databricks Runtime 9.1 LTS

RegrAvgX

Databricks Runtime 10.5

RegrAvgY

Databricks Runtime 10.5

Remainder

Databricks Runtime 8.3

Resolution

Databricks Runtime 11.3 LTS

Reverse

Databricks Runtime 8.3

Reverse

Databricks Runtime 8.3

RLike

Databricks Runtime 8.3

Round

Databricks Runtime 8.3

RowNumber

Databricks Runtime 10.4 LTS

RPadExpressionBuilder

Databricks Runtime 8.3

Second

Databricks Runtime 8.3

SecondsToTimestamp

Databricks Runtime 8.3

Sha1

Databricks Runtime 10.4 LTS

Sha2

Databricks Runtime 10.4 LTS

ShiftLeft

Databricks Runtime 8.3

ShiftRight

Databricks Runtime 8.3

ShiftRightUnsigned

Databricks Runtime 8.3

Sin

Databricks Runtime 10.4 LTS

Size

Databricks Runtime 8.3

Slice

Databricks Runtime 8.3

SoundEx

Databricks Runtime 10.1

SparkVersion

Databricks Runtime 8.3

Sqrt

Databricks Runtime 8.4

StddevPop

Databricks Runtime 8.3

StddevSamp

Databricks Runtime 8.3

StringInstr

Databricks Runtime 8.3

StringLocate

Databricks Runtime 8.3

StringRepeat

Databricks Runtime 11.2

StringSpace

Databricks Runtime 8.3

StringSplit

Databricks Runtime 8.3

StringToH3

Databricks Runtime 11.3 LTS

StringTranslate

Databricks Runtime 10.4 LTS

StringTrim

Databricks Runtime 8.3

StringTrimBoth

Databricks Runtime 8.3

StringTrimLeft

Databricks Runtime 8.3

StringTrimRight

Databricks Runtime 8.3

StructsToJson

Databricks Runtime 11.1

Substring

Databricks Runtime 8.3

Subtract

Databricks Runtime 8.3

Sum

Databricks Runtime 8.3

Tan

Databricks Runtime 9.1 LTS

ToChildren

Databricks Runtime 11.3 LTS

ToParent

Databricks Runtime 11.3 LTS

ToRadians

Databricks Runtime 10.1

ToUnixTimestamp

Databricks Runtime 8.3

ToUTCTimestamp

Databricks Runtime 8.3

TruncDate

Databricks Runtime 8.3

TruncTimestamp

Databricks Runtime 8.3

TryElementAt

Databricks Runtime 10.0

TryValidate

Databricks Runtime 11.3 LTS

UnaryMinus

Databricks Runtime 8.3

UnBase64

Databricks Runtime 9.1 LTS

Unhex

Databricks Runtime 9.1 LTS

UnixDate

Databricks Runtime 8.3

UnixMicros

Databricks Runtime 8.3

UnixMillis

Databricks Runtime 8.3

UnixSeconds

Databricks Runtime 8.3

UnixTimestamp

Databricks Runtime 8.3

Upper

Databricks Runtime 8.3

Uuid

Databricks Runtime 8.3

Validate

Databricks Runtime 11.3 LTS

VarianceSamp

Databricks Runtime 10.1

WeekDay

Databricks Runtime 8.3

WeekOfYear

Databricks Runtime 8.3

XxHash64

Databricks Runtime 10.0

Year

Databricks Runtime 8.3

Limitations

  • Structured Streaming: Photon currently supports stateless streaming with Delta, Parquet, and CSV. Kafka and Kinesis support is in Public Preview

  • Does not support UDFs.

  • Does not support RDD APIs.

  • Not expected to improve short-running queries (<2 seconds), for example, queries against small amounts of data.

Features not supported by Photon run the same way they would with Databricks Runtime; there is no performance advantage for those features.