Principles of Data-Intensive Systems

Many organizations store their largest datasets in data lakes, i.e. collections of files in a large-scale, low-cost storage system such as the Amazon S3 or HDFS. Unfortunately, making data lakes reliable and efficient is challenging: readers can see incomplete data from in-progress writes, there are no indexes to accelerate access, and rolling back incorrect updates made by a data pipeline is complex. To address these problems, Databricks recently open sourced Delta Lake, a transactional storage management system the runs over commodity cloud and HDFS storage to provide reliability and scalability. Delta Lake manages a transaction log that allows for efficient isolation, point-in-time snapshots and rollback, and accesses metadata using Apache Spark in order to scale to billions of files and petabytes of data. Delta Lake was first released to Databricks customers in 2017 and is now in production use at over 1000 organizations, where it processes multiple exabytes of data per month. We describe some of the largest-scale use cases and the motivation for Delta Lake's design.

Bio: Reynold Xin is a cofounder and Chief Architect at Databricks. In the open source community, Reynold is known as a top contributor to the Apache Spark project, having designed many of its core user-facing APIs and execution engine features. Reynold received a PhD in Computer Science from UC Berkeley, where he worked on large-scale data processing systems including Apache Spark, Spark SQL, GraphX and CrowdDB. Reynold also led the team that set the 2014 GraySort record in the Sort Benchmark.