Thinking in Partitions (A Dagster Deep Dive)

Рет қаралды 1,423

Күн бұрын

By embracing partitioning, you can speed up execution and greatly reduce the cost of running pipelines. Gain control over backfills and observe your data assets at a higher level of granularity.
You can catch all Dagster Deep Dives in the series here: dagster.io/events/dagster-dee...
You will find the Dagster repo for these sessions here: bit.ly/DagDemo
Be sure to bookmark (star) the Dagster repo for future reference: github.com/dagster-io/dagster
00:00 Intro
01:22 Why partition your pipeline?
05:06 An intro to Dagster's partitions API
06:37 Partitioning by category
11:24 Partition an asset by time
15:23 Recap of partitioning and partitioning patterns.
22:00 wrap up and other resources
22:40 Q&A

Пікірлер: 11

@shaounakn 4 ай бұрын

I like what you folks have done with this product.

@dagsterio 4 ай бұрын

Thanks - there is a lot more in store coming next month!

@quinnherden 4 ай бұрын

+1 I am rooting for you guys. Thank you for all of your hard work

@dagsterio 4 ай бұрын

We appreciate it - thanks @@quinnherden !

@shaounakn 4 ай бұрын

@@dagsterio Sure, thanks for making these sessions, these are really helpful.

@jordanfox470 17 күн бұрын

Is there native support for mapping time based partitions to static partitions defined like "today", "rest of month", "rest of year", "rest of history"? This is a common setup for power bi datasets, which can be represented as assets in dagster. Would be nice to take advantage of auto materialize policies.

@dagsterio 16 күн бұрын

Dagster does not natively support mapping time-based partitions to static partitions like "today," "rest of month," "rest of year," and "rest of history" directly out of the box. However, you can achieve similar functionality by defining custom partitioning schemes and using the appropriate partition mappings. You can define custom partitions using StaticPartitionsDefinition for static and TimeWindowPartitionsDefinition for time-based partitions.

@user-yh1hx9pe7u 4 ай бұрын

@dagsterio Do you have the source of the demo avaialble somewhere?

@dagsterio 3 ай бұрын

All the code for the demos from the deep dives are in this repository ( github.com/dagster-io/devrel-project-demos )! This one in particular is in the partitions directory.

@Jahaniam 3 ай бұрын

@@dagsterio unfortunately it is private/ link is broken.

@dagsterio 11 күн бұрын

@@Jahaniam Sorry, the final parenthesis got included by KZfaq in the URL - try this: github.com/dagster-io/devrel-project-demos