As data operations scale and teams manage many interdependent data pipelines, it becomes harder to deliver trusted data to the organization while maintaining high performance on the engineering team. Adopting a proper framework becomes essential for making the work repeatable and delivering operational efficiency and observability.
In this fireside chat between Sandy Ryza, Lead Engineer on the Dagster Project, and Pete Hunt, CEO of Dagster Labs, we will discuss Dagster's core framework, built around the core concept of Data Assets. By employing such a framework, Data Engineering, and ML teams can build repeatable processes while leveraging key capabilities like Continuous Integration, testing, and observability at scale.
This session will benefit heads of data engineering or ML looking to build data processes that truly scale, protecting the productivity of the core engineering team while delivering trusted and timely data to all stakeholders.
00:00 introductions
01:33 Building pipelines, building platforms, building platform tooling.
02:28 Key considerations in building a platform: Productivity and Trust
05:00 Software-defined Assets - what are they?
08:00 Defining the interfaces between assets and teams
11:00 How the Asset-centric approach lets you understand how data is generated
12:50 The deeper context that data assets provide
18:15 Data assets and pipeline automation
Try Dagster today with a 30-day free trial: dagster.io/lp/dagster-cloud-t...