Databricks Runtime 7.5 for Genomics (Unsupported)
Databricks released this image in December 2020.
Databricks Runtime 7.5 for Genomics is a version of Databricks Runtime 7.5 (Unsupported) optimized for working with genomic and biomedical data. It is a component of the Databricks Unified Analytics Platform for Genomics.
Important
This documentation has been retired and might not be updated. The products, services, or technologies mentioned in this content are no longer supported.
The Databricks Genomics runtime has been deprecated. For open source equivalents, see repos for genomics-pipelines and Glow. Bioinformatics libraries that were part of the runtime have been released as a Docker container, which can be pulled from the ProjectGlow Dockerhub page.
For more information about the Databricks Runtime deprecation policy and schedule, see All supported Databricks Runtime releases.
For more information, including instructions for creating a Databricks Runtime for Genomics cluster, see Genomics guide. For more information on developing genomics applications, see Genomics guide.
New features
Databricks Runtime 7.5 for Genomics is built on top of Databricks Runtime 7.5. For information on what’s new in Databricks Runtime 7.5, see the Databricks Runtime 7.5 (Unsupported) release notes.
Conversion from Hail MatrixTable to Spark DataFrame
Glow now has the from_matrix_table function to import Hail MatrixTables as Spark DataFrames in Glow.
pandas-based linear regression with offset
Glow now offers the linear_regression function in Python to test association between genotypes and one or more phenotypes (Step 2 of GloWGR). This function is significantly faster than the Glow linear_regression_gwas function (up to 8x on 25 phenotypes) and is designed to work seamlessly with the output of Step 1 GloWGR through acceptance of an offset argument. Covariates and whether to include an intercept in fitting can be controlled as well.
Improvements
Fast VCF reader by default
As of this release, the default VCF reader is set to the fast reader. To use the htsjdk based reader, set the Spark config io.projectglow.vcf.fastReaderEnabled to false.
Hard calls option for BGEN reader
The BGEN reader in Glow now accepts has the new boolean emitHardCalls option to generate hard calls for samples when reading the BGEN file. This option is set to true by default. The probability threshold for hard calls is set by the new hardCallThreshold option (default = 0.9).
Improvements to joint genotyping pipeline
The joint genotyping pipeline was improved such that the targeted regions file is now translated into a filter that can be pushed down to the VCF data source, where tabix index can be leveraged for filtering. Previously, a range join was used for this purpose. This will improve the ingest time if a targeted regions file is provided (with <25 regions) and the input is tabix-indexed bgzipped VCFs. In addition, the default bin size used in the pipeline was reduced to 5000. This change speeds up shuffling by reducing the skew, resulting in a faster pipeline.