Nick Schrock covers the principles and origin of Dagster. Dagster is a new type of workflow engine: a data orchestrator. Moving beyond just managing the ordering and physical execution of data computations, Dagster considers the entire data application lifecycle. Practitioners in Dagster build data-aware dependency graphs designed for local development and testing; deploy those graphs to a multi-tenant, cloud-native orchestration engine; and then monitor and observe the data assets produced by those computations.
In this talk, Nick discusses how Dagster differentiates itself across the three stages (dev & test, deploy & execute, monitor & observer) of the application lifecycle. Through a demo and code snippets, the talk shows how the Dagit web UI and Dagster programming model can power a variety of data practitioners.
Related video:
Data Driven NYC talk on Fundamentals of Data Engineering (kzfaq.info/get/bejne/o7aDra9psdvWo2Q.html)
#data #dataengineering #airflow #prefect #cloudnative #cloudcomputing #cloud #fundamentalsofdataengineering
-------------------------
About Nick Schrock
Nick Schrock is the founder and CEO of Elementl, the company behind Dagster. Previously, Nick worked at Facebook, where he co-created GraphQL. Nick believes deeply in the power of well-designed developer tools to make engineers more productive, accelerate their careers, make their lives more enjoyable, and transform the organizations in which they work.