This article introduces the set of fundamental concepts you need to understand in order to use Databricks effectively.
In Databricks, a workspace is a Databricks deployment in the cloud that functions as an environment for your team to access Databricks assets. Your organization can choose to have either multiple workspaces or just one, depending on its needs.
A Databricks account represents a single entity that can include multiple workspaces. Accounts enabled for Unity Catalog can be used to manage users and their access to data centrally across all of the workspaces in the account. Billing and support are also handled at the account level.
Databricks bills based on Databricks units (DBUs), units of processing capability per hour based on VM instance type.
See the Databricks on AWS pricing estimator.
Data science & engineering tools aid collaboration among data scientists, data engineers, and data analysts. This section describes the fundamental concepts.
A workspace is an environment for accessing all of your Databricks assets. A workspace organizes objects (notebooks, libraries, dashboards, and experiments) into folders and provides access to data objects and computational resources.
A web-based interface for creating data science and machine learning workflows that can contain runnable commands, visualizations, and narrative text. See Introduction to Databricks notebooks.
An interface that provides organized access to visualizations. See Dashboards in notebooks.
A package of code available to the notebook or job running on your cluster. Databricks runtimes include many libraries and you can add your own.
A folder whose contents are co-versioned together by syncing them to a remote Git repository. Databricks Repos integrate with Git to provide source and version control for your projects.
This section describes the interfaces that Databricks supports, in addition to the UI, for accessing your assets: API and command-line (CLI).
This section describes the objects that hold the data on which you perform analytics and feed into machine learning algorithms.
A filesystem abstraction layer over a blob store. It contains directories, which can contain files (data files, libraries, and images), and other directories. DBFS is automatically populated with some datasets that you can use to learn Databricks. See What is the Databricks File System (DBFS)?.
A collection of data objects, such as tables or views and functions, that is organized so that it can be easily accessed, managed, and updated. See What is a database?
A representation of structured data. You query tables with Apache Spark SQL and Apache Spark APIs. See What is a table?
By default, all tables created in Databricks are Delta tables. Delta tables are based on the Delta Lake open source project, a framework for high-performance ACID table storage over cloud object stores. A Delta table stores data as a directory of files on cloud object storage and registers table metadata to the metastore within a catalog and schema.
Find out more about technologies branded as Delta.
The component that stores all the structure information of the various tables and partitions in the data warehouse including column and column type information, the serializers and deserializers necessary to read and write data, and the corresponding files where the data is stored. See What is a metastore?
Every Databricks deployment has a central Hive metastore accessible by all clusters to persist table metadata. You also have the option to use an existing external Hive metastore.
A graphical presentation of the result of running a query. See Visualizations in Databricks notebooks.
This section describes concepts that you need to know to run computations in Databricks.
A set of computation resources and configurations on which you run notebooks and jobs. There are two types of clusters: all-purpose and job. See Compute.
You create an all-purpose cluster using the UI, CLI, or REST API. You can manually terminate and restart an all-purpose cluster. Multiple users can share such clusters to do collaborative interactive analysis.
The Databricks job scheduler creates a job cluster when you run a job on a new job cluster and terminates the cluster when the job is complete. You cannot restart an job cluster.
A set of idle, ready-to-use instances that reduce cluster start and auto-scaling times. When attached to a pool, a cluster allocates its driver and worker nodes from the pool. See Create a pool.
If the pool does not have sufficient idle resources to accommodate the cluster’s request, the pool expands by allocating new instances from the instance provider. When an attached cluster is terminated, the instances it used are returned to the pool and can be reused by a different cluster.
The set of core components that run on the clusters managed by Databricks. See Compute.* Databricks has the following runtimes:
Databricks Runtime includes Apache Spark but also adds a number of components and updates that substantially improve the usability, performance, and security of big data analytics.
Databricks Runtime for Machine Learning is built on Databricks Runtime and provides prebuilt machine learning infrastructure that is integrated with all of the capabilities of the Databricks workspace. It contains multiple popular libraries, including TensorFlow, Keras, PyTorch, and XGBoost.
Frameworks to develop and run data processing pipelines:
Jobs: A non-interactive mechanism for running a notebook or library either immediately or on a scheduled basis.
Delta Live Tables: A framework for building reliable, maintainable, and testable data processing pipelines.
Databricks identifies two types of workloads subject to different pricing schemes: data engineering (job) and data analytics (all-purpose).
Data engineering An (automated) workload runs on a job cluster which the Databricks job scheduler creates for each workload.
Data analytics An (interactive) workload runs on an all-purpose cluster. Interactive workloads typically run commands within a Databricks notebook. However, running a job on an existing all-purpose cluster is also treated as an interactive workload.
Machine Learning on Databricks is an integrated end-to-end environment incorporating managed services for experiment tracking, model training, feature development and management, and feature and model serving.
The main unit of organization for tracking machine learning model development. See Organize training runs with MLflow experiments. Experiments organize, display, and control access to individual logged runs of model training code.
A centralized repository of features. See What is a feature store? Feature Store enables feature sharing and discovery across your organization and also ensures that the same feature computation code is used for model training and inference.
An interface that allows you to automate tasks on SQL objects. See SQL API.
A presentation of data visualizations and commentary. See Databricks SQL dashboards.