Netflix Data Engineering Tech Talks - Psyberg, An Incremental ETL Framework Using Iceberg
Рет қаралды 4,972
8 ай бұрынAbhinaya Shetty and Bharath Mummadisetty, Data Engineers from Netflix’s Membership Data Engineering team, introduce Psyberg, an incremental ETL framework. Learn about how Psyberg leverages Iceberg metadata to handle late-arriving data, and improves data pipelines while simplifying on-call life!
#netflix
#datascience
#dataengineering
#etl
#bigdata
@rajvellaturi 8 ай бұрын
This is such an exciting talk. I faced this problem in my experience working as a DE. Identifying and reprocessing those late-arriving records is resource-intensive and time consuming for sure. Thanks to Iceberg for making it easy and possible to put together a solution with the help of metadata.
@iirdna 6 ай бұрын
how you avoiding too high tide of a changes? meaning - is any late data arriving triggers Psyberg? even just few thousand of rows? or you accumulating changes at some sort of gates/elevators and process when enough late data accumulated to justify downstream reprocessing?
@ViralDave_26 8 ай бұрын
👍👍
Amazon Web Services
Рет қаралды 17 М.
Network Security-Video Presentation-Formulating an Efficient Real-Time Monitoring and Countermeasure
Shailendra T
Рет қаралды 16
Netflix Data
Рет қаралды 32 М.
CelerData
Рет қаралды 596
Netflix Data
Рет қаралды 3,4 М.
Netflix Data
Рет қаралды 3 М.
AWS Events
Рет қаралды 10 М.