User Guide
Welcome to the Databricks User Guide. Here you will find the key topics that you need to be successful with Databricks.
- Supported Browsers
- Clusters
- Display Clusters
- Create a Cluster
- Clone a Cluster
- Edit a Cluster
- Start a Cluster
- Terminate a Cluster
- Pin a Cluster
- Delete a Cluster
- Cluster Event Log
- Cluster Configurations
- Cluster Mode
- GPU-enabled Clusters
- Overview of Databricks Light
- Overview of Databricks Runtime for Machine Learning
- Python Clusters
- Cluster Access Control
- Metrics
- Cluster Driver Logs
- Cluster Information in the Spark UI
- Clusters and Libraries
- Clusters and Notebooks
- Tips and Troubleshooting
- Workspace
- Notebooks
- Visualizations
- Accessing Data
- Databases and Tables
- Libraries
- Jobs
- Secrets
- Developer Tools
- Databricks File System - DBFS
- Databricks IO Cache
- Business Intelligence Tools
- Advanced Features
- Security
- FAQ and Best Practices Guides
- Install and Compile Cython
- Databricks Datasets
- Running C++ Code
- Sending Email From Databricks Notebooks
- Spark 1.x Cluster Connectivity
- How to Delete All Jobs Using the REST API
- Cannot Read Databricks Objects Stored in DBFS Root Directory
- Problem: SSO Server Redirects to Original URL, Not to Vanity Databricks URL
- Incompatible Schema in Some Files
- How to Set up Apache Kafka on Databricks
- How to Create a Cluster with Anaconda
- How to Dump Tables in CSV, JSON, XML, Text, or HTML Format
- Distinguish Active and Dead Jobs
- Drop Tables with Corrupted Metadata from the Metastore
- How to Get the Full Path to the Current Notebook
- Hive UDFs
- How to Run SQL Queries from Python Scripts
- Multi-Part Upload Failure
- Listing Table Names
- EBS Leaked Volumes
- How to Update Nested Columns
- CosmosDB-Spark Connector Library Conflict
- Reading Large DBFS-Mounted Files Using Python APIs
- VACUUM Command on a Databricks Delta Table Stored in S3
- How to Improve Performance of Databricks Delta MERGE INTO Queries Using Partition Pruning
- Handling Partition Column Values While Using an SQS Queue as a Streaming Source
- Cannot Run Notebook Commands After Canceling Streaming Cell
- Problem: Access Denied when Writing Databricks Delta Tables to S3
- How to get Workspace, Cluster, Notebook, and Job Details
- Problem: Notebook Autosave Fails Due to File Size Limits
- Problem: Library Unavailability Causing Job Failures
- Problem: Access Denied When Writing to an S3 Bucket Using RDD
- Problem: Cluster Failed to Launch
- How to Persist and Share Code in RStudio
- How to Calculate the Number of Cores in a Cluster
- How to Use Apache Spark Metrics
- How to Speed Up Cross-Validation
- How to Handle Blob Data Contained in an XML File
- How to Improve Performance with Bucketing
- Problem: Apache Spark Job Doesn’t Start
- Problem: Spark Job Fails with
Driver is temporarily unavailable - How to Troubleshoot Several Apache Hive Metastore Problems
- Problem: Cluster Cancels Python Command Execution due to Library Conflict
- Problem: Cluster Cancels Python Command Execution after Installing Bokeh
- Problem: Python Command Execution Fails with
AttributeError - Best Practices for Dropping a Managed Databricks Delta Table
- Streaming with File Sink: Problems with Recovery if you Change Checkpoint or Output Directories
- Problem: Create Table in Overwrite Mode Fails When Interrupted
- Problem: Job Fails When Using Spark-Avro to Write Decimal Values to AWS Redshift
- How to Write to a Custom Sink Using
foreachBatchin Structured Streaming - Problem: Maximum Execution Context or Notebook Attachment Limit Reached
- How to Extract Feature Information for Tree-based Apache SparkML Pipeline Models
- How to Correctly Update a Maven Library in Databricks
- How to Specify Skew Hints in Dataset and DataFrame-based Join Commands
- Problem: Fitting an Apache SparkML Model Throws Error
- Problem:
ForbiddenError while Accessing S3 Data - How to Perform Group K-fold Cross Validation with Apache Spark
- How to Save Plotly Files and Display From DBFS
- Problem: Apache Spark Jobs Hang Due to Non-deterministic Custom UDF
- How to Restart a Structured Streaming Query From Last Written Offset