Databricks tables concepts
A Databricks table resides in a schema and contains rows of data. The default table type created in Databricks is a Unity Catalog managed table.
The following example shows a managed table named prod.people_ops_employees
that contains data about five employees. As a managed table, the data files are stored in Unity Catalog's managed storage location in cloud storage.
Storage formats
Table types in Databricks define how data is owned and accessed. Separately, the storage format defines how the data is physically structured and tracked on disk.
Databricks supports two primary open table storage formats:
- Delta Lake is the default storage format for managed and external tables in Databricks.
- Apache Iceberg is supported on managed and foreign tables in Databricks. This format is useful when you're integrating with the Iceberg ecosystem.
These formats add a transactional storage layer that tracks metadata and enables Atomicity, Consistency, Isolation, and Durability (ACID) compliance, time travel, and other features.
Table types
Databricks offers three primary table types, each designed for different data management scenarios and ownership models. Your choice of table type determines how Databricks manages the underlying data files and metadata.
The primary differentiator for table types in Databricks is the owning catalog, as described in the following table:
Table type | Managing catalog | Read/write support | Performance optimization | Storage cost optimization |
---|---|---|---|---|
Managed | Unity Catalog | Yes | Yes | Yes |
External | None (files only) | Yes | Manual only | Manual only |
Foreign | An external system or catalog service | Read only | No | No |
Managed tables
Managed tables manage underlying data files alongside the metastore registration. Databricks recommends that you use managed tables whenever you create a new table. Unity Catalog managed tables are the default when you create tables in Databricks. See Managed tables.
External tables
External tables, sometimes called unmanaged tables, reference data stored outside of Databricks in an external storage system, such as cloud object storage. They decouple the management of underlying data files from metastore registration. Unity Catalog supports external tables in several formats, including Delta Lake. Unity Catalog external tables can store data files using common formats readable by external systems. See External tables.
Foreign tables
Foreign tables represent data stored in external systems connected to Databricks through Lakehouse Federation. Foreign tables are read-only on Databricks. See Foreign tables.
Tables in Unity Catalog
In Unity Catalog, tables sit at the third level of the three-level namespace (catalog.schema.table
), as shown in the following diagram:
Basic table permissions
Most table operations require USE CATALOG
and USE SCHEMA
permissions on the catalog and schema containing a table.
The following table summarizes the additional permissions needed for common table operations in Unity Catalog:
Operation | Permissions |
---|---|
Create a table |
|
Query a table |
|
Update, delete, merge, or insert data to a table |
|
Drop a table |
|
Replace a table |
|
For more information about Unity Catalog permissions, see Manage privileges in Unity Catalog.