Making Apache Spark™ Better with Delta Lake

No video

Making Apache Spark™ Better with Delta Lake

Рет қаралды 175,443

Күн бұрын

Join Michael Armbrust, head of Delta Lake engineering team, to learn about how his team built upon Apache Spark to bring ACID transactions and other data reliability technologies from the data warehouse world to cloud data lakes.
Apache Spark is the dominant processing framework for big data. Delta Lake adds reliability to Spark so your analytics and machine learning initiatives have ready access to quality, reliable data. This webinar covers the use of Delta Lake to enhance data reliability for Spark environments.
Topics areas include:
- The role of Apache Spark in big data processing
- Use of data lakes as an important part of the data architecture
- Data lake reliability challenges
- How Delta Lake helps provide reliable data for Spark processing
- Specific improvements improvements that Delta Lake adds
- The ease of adopting Delta Lake for powering your data lake
See full Getting Started with Delta Lake tutorial series here:
databricks.com/getting-starte...
Get the Delta Lake: Up & Running by O’Reilly ebook preview to learn the basics of Delta Lake, the open storage format at the heart of the lakehouse architecture. Download the ebook: dbricks.co/3IIcVCg

Пікірлер: 16

@RossittoS 3 жыл бұрын

Excellent features!!

@sonagy23 2 жыл бұрын

28:32 How does Delta Lake work? 28:50 Delta On Disk 29:59 Table = result of a set of actions 31:31 Implementing Atomicity 32:48 Ensuring Serializability 33:33 Solving Conflicts Optimistically 35:08 Handling Massive Metadata 36:32 Roadmap 38:20 QnA

@kbkonatham1701 2 жыл бұрын

hi kim thanks for support , you are from ? , i am from india.

@rakshithvenkatesh2773 3 жыл бұрын

I see this whole "Hierarchical Data Pipeline" strategy being talked about quite a bit these days. We did establish this as part of a ready solution we built for Manufacturing use case using Confluent Kafka + KSQL. But the Data Lake is something i believe will remain/continue to exist as a depot for long term retention of data where AI/DA platforms leverage data from these data lakes for batch processing. I see this story from DataBricks to be a Data-warehouse convergence towards Data Lakes !

@meryplays8952 3 жыл бұрын

The architecture comes with a nice VLDB 2020 paper (which the presenter did not mention).

@hidemisuzuki965 2 жыл бұрын

Where can I download the slides? Thanks!

@Sangeethsasidharanak 3 жыл бұрын

27.28 on automating data quality. .. isn't it same as we do quality check before we save using custom code..Will there be any additional benefits?

@gustavemuhoza4212 3 жыл бұрын

It's probably the same, but not sure how you could do that on a datalake consistently. As described here, Delta appears to make it easier to do and making it possible to do it as if you were doing it on a relational database.