The transaction log for a Delta table contains versioning information that supports Delta Lake evolution. Delta Lake tracks minimum reader and writer version separately.
Delta Lake guarantees backward compatibility. A newer version of Databricks Runtime will always be able to read data that was written by an older version of Databricks Runtime.
Delta Lake will occasionally break forward compatibility. Older versions of Databricks Runtime may not be able to read and write data that was written by a newer version of Databricks Runtime. If you try to read and write to a table with a version of Databricks Runtime that is too old, you’ll get an error telling you that you need to upgrade.
For existing tables, you’ll be warned about the availability of new version and given the option to upgrade the table. Sometimes you may be required to upgrade before you are able to use new features (for example, append-only tables).
The reader version in Databricks Runtime 4.1 did not change, but the writer version in Databricks Runtime 4.1 was incremented. As a consequence:
You cannot write to tables created by Databricks Runtime 4.1 and above with Databricks Runtime lower than 4.1.
You can read tables written by Databricks Runtime 4.1 and above with Databricks Runtime lower than 4.1.
To take advantage of Databricks Runtime 4.1 improvements, you must upgrade tables created using versions of Databricks Runtime lower than 4.1. To upgrade a table, first upgrade all jobs that are writing to the table. Then run:
com.databricks.delta.Delta.upgradeTable("<path-to-table>" or "<tableName>")