Welcome to Databricks

This documentation site provides how-to guidance and reference information for Databricks and Apache Spark.

Getting started

This section shows how to get started with Databricks.

• Getting started
  • Sign up for a free Databricks trial
  • Get started as a Databricks user
  • Get started as a Databricks administrator
  • Get Databricks training
  • Set up and deploy your Databricks account
  • Databricks overview
  • Databricks concepts
  • Introduction to Apache Spark
  • Supported browsers

Languages

This section provides information about the languages supported in Databricks notebooks and jobs: Python, R, Scala, and SQL.

• Language guides
  • Python
  • R
  • Scala
  • SQL

Databricks runtimes

This section provides an overview of the variety of Databricks runtimes.

• Databricks runtimes
  • Databricks Runtime
  • Databricks Runtime for Machine Learning
  • Databricks Runtime for Genomics
  • Databricks Light

Workspace

This section shows how to use a Databricks Workspace.

• Workspace
  • Workspace assets
  • Work with workspace objects
  • Get workspace, cluster, notebook, model, and job identifiers

Clusters

This section shows how to create and manage Databricks clusters.

• Clusters
  • Create a cluster
  • Manage clusters
  • Configure clusters
  • Task preemption
  • Customize containers with Databricks Container Services
  • Cluster node initialization scripts
  • GPU-enabled clusters
  • Single Node clusters
  • Pools
  • Web terminal

Notebooks

This section shows how to use Databricks notebooks.

• Notebooks
  • Manage notebooks
  • Use notebooks
  • Visualizations
  • Dashboards
  • Widgets
  • Notebook workflows
  • Package cells

Jobs

This section shows how to use Databricks jobs.

• Jobs
  • View jobs
  • Create a job
  • View job details
  • Run a job
  • Run a job with different parameters
  • Notebook job tips
  • JAR job tips
  • View job run details
  • Export job run results
  • Edit a job
  • Delete a job
  • Library dependencies
  • Advanced job options
  • Control access to jobs

Libraries

This section shows how to make third-party or custom libraries available to notebooks and jobs running on your clusters.

• Libraries
  • Workspace libraries
  • Cluster libraries
  • Notebook-scoped Python libraries

Data

This section shows how to work with data in Databricks.

• Data
  • Data overview
  • Databases and tables
  • Metastores
  • Data sources
  • Databricks File System (DBFS)
  • Databricks datasets
  • FileStore

Integrations

This section shows how to connect third-party tools, such as business intelligence (BI) tools and partner data sources, to Databricks.

• Integrations
  • Business intelligence tools
  • Partner data integrations

Delta Lake

This section shows how to use Delta Lake on Databricks.

• Delta Lake
  • Introduction
  • Delta Lake quickstart
  • Introductory notebooks
  • Ingest data into Delta Lake
  • Table batch reads and writes
  • Table streaming reads and writes
  • Table deletes, updates, and merges
  • Table utility commands
  • Table versioning
  • API reference
  • Concurrency control
  • Integrations
  • Migration Guide
  • Best practices
  • Frequently asked questions (FAQ)
  • Resources

Delta Engine

This section shows how Delta Engine optimizations make Delta Lake operations highly performant.

• Delta Engine
  • Optimize performance with file management
  • Auto Optimize
  • Optimize performance with caching
  • Dynamic file pruning
  • Isolation levels
  • Bloom filter indexes
  • Optimize join performance
  • Optimized data transformation

Apache Spark

These sections provide Apache Spark information using Databricks examples.

• DataFrames and Datasets
  • Introduction to DataFrames - Python
  • Introduction to DataFrames - Scala
  • Introduction to Datasets
  • Complex and nested data
  • Aggregators
  • Dates and timestamps
• Structured Streaming
  • Demo notebooks
  • Streaming data sources and sinks
  • Structured Streaming in production
  • Structured Streaming examples
  • Spark Streaming (Legacy)

Machine learning

These sections provide information about machine learning features supported by Databricks.

• Machine learning and deep learning
  • 10-minute tutorial: machine learning on Databricks
  • Load data
  • Preprocess data
  • Train models
  • Hyperparameter tuning and AutoML
  • Model inference
  • Deploy models
  • Export and import models
  • Reference solutions
• MLflow
  • Quick start
  • Track machine learning training runs
  • Log, load, and deploy MLflow Models
  • Run MLflow Projects on Databricks
  • Manage the Lifecycle of MLflow Models in MLflow Model Registry
  • MLflow Model Serving on Databricks
  • End-to-end example of building machine learning models on Databricks
• Graph analysis
  • GraphFrames
  • Graph analysis tutorial with GraphX (Legacy)

Genomics

This section provides information about genomics application support in Databricks.

• Genomics
  • Secondary analysis
  • Tertiary analysis

Developer tools

Developer tools help you develop Databricks applications using the Databricks REST API, Databricks Utilities, Databricks CLI, or tools outside the Databricks environment.

• APIs and developer tools
  • Databricks REST API
  • Databricks Utilities
  • Databricks CLI
  • Databricks Connect
  • Continuous integration and delivery
  • Managing dependencies in data pipelines

Migration

This section provides information about migrating workloads to Databricks.

• Migration
  • Migrate production workloads to Databricks
  • Migrate single node workloads to Databricks

Security and privacy

These articles provide information about securing your Databricks infrastructure and data and ensuring that privacy requirements are met.

• Security and privacy
  • Access control
  • Secret management
  • Credential passthrough
  • Credential redaction
  • Secure cluster connectivity
  • Encrypt traffic between cluster worker nodes
  • IP access lists
  • Configure domain name firewall rules
  • Best practices: GDPR and CCPA compliance using Delta Lake
  • HIPAA-compliant deployment
  • Best practices: Data governance on Databricks

Administration

This guide shows how to manage your Databricks account.

• Administration
  • Manage your Databricks account
  • Create a new workspace using the Account API
  • Using automation templates to create a new workspace using the Account API
  • Access the Admin Console
  • Manage users and groups
  • Enable access control
  • Manage workspace objects and behavior
  • Manage cluster configuration options
  • Manage AWS Infrastructure
  • Plan capacity and control cost

Release notes

This section provides information about new Databricks functionality.

• Release notes
  • Platform release notes
  • Databricks runtime release notes
  • Databricks preview releases

Support

This guide provides information about submitting and managing support tickets, as well as information about managing your Databricks Support contract.

• Support
  • Requirements
  • Log in to the Databricks Help Center
  • Create a support case
  • Update or respond to a support case
  • Close a support case
  • Update or respond to support cases opened by others
  • Escalate a support case
  • Update your profile
  • The admin console
  • Enable and create contacts (admin only)
  • Set read and write access (admin only)
  • Update your organization’s preferred timezone (admin only)

Ideas Portal

This guide provides information about submitting feature requests and other feedback using the Ideas Portal.

• Ideas Portal
  • Get started
  • Create an idea
  • Vote and comment
  • How does the product team use ideas?
  • What does it take for an idea to be prioritized in the Databricks roadmap?
  • SLA for product management response
  • What happened to the requests submitted in the previous feedback portal?
  • Ideas Portal etiquette

Status

This section provides information about using the Databricks Status Page to view service status and subscribe to alerts.

• Status
  • Service icons
  • Incident page
  • Subscribe