Welcome to Databricks

This documentation site provides how-to guidance and reference information for Databricks and Apache Spark.

Getting Started

This section shows how to get started with Databricks.

• Getting Started
  • Try Databricks
  • Databricks Quickstart
  • Databricks Concepts
  • Databricks Datasets
  • Apache Spark Getting Started
  • Training and FAQ
  • Supported Browsers

Databricks Runtimes

This section provides an overview of the variety of Databricks runtimes.

• Databricks Runtimes
  • Databricks Runtime
  • Databricks Runtime with Conda
  • Databricks Runtime for Machine Learning
  • Databricks Runtime for Genomics
  • Databricks Light

Workspace

This section shows how to use a Databricks Workspace.

• Workspace
  • Workspace Assets
  • Working with Workspace Objects
  • Get Workspace, Cluster, Notebook, and Job Identifiers

Clusters

This section shows how to create and manage Databricks clusters.

• Clusters
  • Create a Cluster
  • Manage Clusters
  • Cluster Configurations
  • Cluster Node Initialization Scripts
  • GPU-enabled Clusters
  • Pools

Notebooks

This section shows how to use Databricks notebooks.

• Notebooks
  • Manage Notebooks
  • Use Notebooks
  • Visualizations
  • Dashboards
  • Widgets
  • Notebook Workflows
  • Package Cells

Jobs

This section shows how to use Databricks jobs.

• Jobs
  • View jobs
  • Create a job
  • View job details
  • Run a job
  • Run a job with different parameters
  • Notebook job tips
  • JAR job tips
  • View job run details
  • Export job run results
  • Edit a job
  • Delete a job
  • Library dependencies
  • Job options
  • Control access to jobs

Libraries

This section shows how to use Databricks libraries.

• Libraries
  • Library modes
  • Library lifecycles
  • Workspace libraries
  • Install a library on a cluster
  • Uninstall a library from a cluster
  • View the libraries installed on a cluster
  • Update a cluster-installed library

Data

This section shows how to work with data in Databricks.

• Data
  • Databricks File System
  • Access Data
  • Databases and Tables
  • Metastores
  • Data Sources
  • The FileStore

Business Intelligence Tools

This section shows how to connect business intelligence (BI) tools to Databricks clusters to query data in tables.

• Business Intelligence Tools
  • Connecting BI Tools
  • Tableau
  • Power BI
  • Alteryx
  • Looker
  • SQL Workbench/J

Delta Lake

This section shows how to use Delta Lake on Databricks.

• Delta Lake
  • Introduction to Delta Lake
  • Introductory Notebooks
  • Table Batch Reads and Writes
  • Table Streaming Reads and Writes
  • Table Deletes, Updates, and Merges
  • Table Utility Commands
  • Delta Lake API Reference
  • Concurrency Control
  • Migrate Workloads to Delta Lake on Databricks
  • Optimizations
  • Presto and Athena Compatibility Support for Delta Lake
  • Frequently Asked Questions (FAQ)
  • Additional Resources

Apache Spark

These sections provide Apache Spark information using Databricks examples.

• DataFrames and Datasets
  • Introduction to DataFrames - Python
  • Introduction to DataFrames - Scala
  • Introduction to Datasets
  • Complex and Nested Data
  • Aggregators
• Structured Streaming
  • Introductory Notebooks
  • Streaming Data Sources and Sinks
  • Structured Streaming in Production
  • Examples
  • Spark Streaming (Legacy)
• SQL
  • SQL Language Manual
  • Spark SQL Examples
  • Compatibility with Apache Hive
• R
  • SparkR
  • sparklyr
  • RStudio
  • Tips

Machine Learning

These sections provide information about machine learning features supported by Databricks.

• Machine Learning
  • Apache Spark MLlib
  • AutoML
  • Exporting and Importing ML Models
  • Third-Party Machine Learning Integrations
• Deep Learning
  • Data Preparation
  • Single Node Training
  • Distributed Training
  • Model Inference
• MLflow
  • Quick Start
  • Tracking Machine Learning Training Runs
  • Saving, Loading, and Deploying Models
  • Reproducible Runs with MLflow Projects
• Graph Analysis
  • GraphFrames Documentation
  • Graph Analysis Tutorial with GraphX (Legacy)

Genomics

This section provides information about genomics application support in Databricks.

• Genomics
  • Secondary Analysis
  • ETL
  • Tertiary Analysis

Developer Tools

Developer tools help you develop Databricks applications using the Databricks REST API, Databricks Utilities, Databricks CLI, or tools outside the Databricks environment.

• Developer Tools
  • Databricks REST API
  • Databricks Utilities
  • Databricks CLI
  • Databricks Connect
  • sbt Plugin
  • Managing Dependencies in Data Pipelines

Migration

This section provides information about migrating workloads to Databricks.

• Migration
  • Migrate Production Workloads to Databricks
  • Migrate Single Node Workloads to Databricks
  • Migrate Workloads to Delta Lake on Databricks

Security and Privacy

These topics provide information about securing your Databricks infrastructure and data and ensuring that privacy requirements are met.

• Security and Privacy
  • Secrets
  • Credential Redaction
  • HIPAA-Compliant Deployment

Administration

This guide shows how to manage your Databricks account.

• Administration
  • Manage Your Databricks Account
  • Access the Admin Console
  • Manage Users and Groups
  • Manage Access Control
  • Manage Workspace Storage
  • Manage AWS Infrastructure
  • Plan Capacity and Control Cost

Release Notes

This section provides information about new Databricks functionality.

• Release Notes
  • Platform Release Notes
  • Databricks Runtime Release Notes
  • Databricks Preview Releases

Support

This guide provides information about submitting and managing support tickets, as well as information about managing your Databricks Support contract.

• Support
  • Requirements
  • Log in to the Databricks Help Center
  • Create a support case
  • Update or respond to a support case
  • Close a support case
  • Update or respond to support cases opened by others
  • Escalate a support case
  • Update your profile
  • The admin console
  • Enable and create contacts (admin only)
  • Set read and write access (admin only)
  • Update your organization’s preferred timezone (admin only)

Ideas Portal

This guide provides information about submitting feature requests and other feedback using the Ideas Portal.

• Ideas Portal
  • Get started
  • Create an idea
  • Vote and comment
  • How does the product team use ideas?
  • What does it take for an idea to be prioritized in the Databricks roadmap?
  • SLA for product management response
  • What happened to the requests submitted in the previous feedback portal?
  • Ideas Portal etiquette