Databricks has announced that the Delta Lake has made open source. Delta Lake is a storage layer that makes it easier to maintain data integrity as new data flows into an enterprise data lake, by bringing ACID transactions to the data repositories.
Ali Dhodsi, co-founder and CEO of Databricks, states that data lakes have proven to have little data reliability. “Delta Lake has removed these challenges for hundreds of enterprises. By making Delta Lake open source, developers can easily build reliable data lakes and convert them into Delta Lakes.”
Delta Lake has been a part of Databricks’ services for some time now and is already being used in production by companies such as Viacom, Edmunds, Riot Games and McGraw Hill, writes TechCrunch. The tool provides the ability to enforce specific schedules, take snapshots, and absorb streaming data or refill it more as a batch job.
Delta Lake also uses the Spark engine to handle the metadata of the data lake. In the long run, the company also wants to add an audit trail, as well as various other functions. It is also important to know that Delta Lake runs lakes on top of existing data and provides support for the Apache Spark APIs.
The company is still investigating how the project will be managed in the future. “We are still exploring multiple models of open source project governance, but the GitHub model is well understood and offers a good balance between the ability to accept contributions and governance from above,” says Ghodsi.
Delta Lake becomes available under the Apache License V2, the same license that Apache Spark uses. In addition, Databricks intends to accept contributions from outside, just as in the Spark project. “We want Delta Lake technology to be used everywhere on premise and in the cloud by small and large enterprises.”
“This approach is the quickest way to build something that can become a standard, by allowing the community to provide direction and contribute to the development.This news article was automatically translated from Dutch to give Techzine.eu a head start. All news articles after September 1, 2019 are written in native English and NOT translated. All our background stories are written in native English as well. For more information read our launch article.