Thanks for the comment @JohnCF. If you go through the enhancements introduced with this Dagster+ launch, you will see that many of them (in fact, all of them except for Dagster Insights) benefit both the open-source and the commercial offerings. The data cataloging capability is a good example of that. From our perspective, these new additions are moving us forward on both the OSS and the Dagster+ roadmaps. In addition, by providing more value to those organizations that adopt Dagster+ we are able to guarantee the longevity and accelerated development of Dagster Open-Source.

@@dagsterio Does that mean what's mentioned at 7:15 about column lineage is available in open-source too? The phrasing definitely sounded like it's only available for Enterprise users...

@@JohnCF Correct. Column level lineage is a Dagster+ feature and is not available in Dagster Open-Source.

Dagster does not natively support mapping time-based partitions to static partitions like "today," "rest of month," "rest of year," and "rest of history" directly out of the box. However, you can achieve similar functionality by defining custom partitioning schemes and using the appropriate partition mappings. You can define custom partitions using StaticPartitionsDefinition for static and TimeWindowPartitionsDefinition for time-based partitions.

Hey Abishek! In your case, would you be able to represent the S3 files as source assets first, adding asset checks onto those, and running Embedded ELT only if those asset checks pass? Sling currently (afaik) is heavily focused on doing ingestion well, so you can defer to the rest of the Dagster ecosystem (such as asset checks) for validations.

@@tim-at-elementl Thanks, Tim. How would you rate dlt for my use-case? I see dlt is far more mature..

​@@AbhishekAgrawal-dv1id we've found that dlt is a powerful framework for ingesting from APIs and it's definitely mature enough for production settings. I'll also say that neither Sling's or dlt's integration currently allow for creating asset checks in-flight during ingestion. Instead, have you thought about ingesting the files into a quarantined dataset first using whichever tool you'd like, applying asset checks to that, and then moving that data to your real "analytics-ready" BQ datasets once you've vetted the data? This way, you can do ad hoc analysis to understand why the data failed data quality tests easily, but also keep it isolated from your production analytics.

Yeah, I am also leaning towards doing something like this. Thanks for this, Tim. Would you suggest using a similar approach to pull data from a different database? We'd still need to run minor validations on the incoming data, though. Would dlt help here at all?

Yep. We are big fans. Enjoy the Easter eggs! ;-)

For some teams, definitely, although it can be complementary to dbt docs, because it sucks in some of the data via the dbt integration. Essentially becomes a super set of documentation

Behind the camera, helping out with the teleprompter while recovering from an injury.

Wishing him a speedy recovery then. We miss him on KZfaq !

👋 Right here! I just happened to be unable to participate in the recording session for this. Team killed it!

@@schrockn yes they did. Keen to hear your take on all this Nick ; video from you soon ?

Many of the enhancements in the 1.7 release benefit all users (Open-source and paid Dagster+ users). In general, the open-source solution gains more capabilities with each release both to support open-source users and to unlock more capabilities in Dagster+ which are built on top of core.

Hi! I had to put it in a different repo to accommodate for running multiple code locations and not breaking our existing setup for the deep dive projects. The dedicated repo for the data mesh example can be found here! github.com/dagster-io/data-mesh-demo

@@dagsterio much appreciated

Thanks. The Dagster capabilities are expanding with each new release.

is it really hard? Im planning to learn this too. lmao

You might find this blog by Sandy interesting: dagster.io/blog/dagster-ml-pipelines. - Otherwise you can listen to the entire Podcast featuring Sandy here: datastackshow.com/podcast/machine-learning-pipelines-are-still-data-pipelines-with-sandy-ryza-of-dagster/

Sorry, one of our redirects got broken - here is the link: github.com/dagster-io/devrel-project-demos/tree/main

More specifically for this session: github.com/dagster-io/devrel-project-demos/tree/main/dagster-deep-dives/dagster_deep_dives/resources_and_configurations

All the code for the demos from the deep dives are in this repository ( github.com/dagster-io/devrel-project-demos )! This one in particular is in the partitions directory.

@@dagsterio unfortunately it is private/ link is broken.

@@Jahaniam Sorry, the final parenthesis got included by KZfaq in the URL - try this: github.com/dagster-io/devrel-project-demos

Thanks - there is a lot more in store coming next month!

+1 I am rooting for you guys. Thank you for all of your hard work

We appreciate it - thanks @@quinnherden !

@@dagsterio Sure, thanks for making these sessions, these are really helpful.

Try: github.com/dagster-io/devrel-project-demos

Hi @user-hs9lo5gh3r, the most common way to bring up this menu is to select an asset from the global asset lineage, and then in the top right where it says "Materialize selected...", open the dropdown menu and select "Open launchpad". Hope this helps!

Hey @Amapramaadhy, what you’re expressing is totally valid. The concepts of Assets, Ops, and Jobs and how to compose them can be a bit convoluted - this has become more noticeable as our APIs evolve. We’re aware of this, and it’s on our roadmap to improve. Thanks for taking the time to respond and sharing your thoughts.

No doubt that evey new powerful framework takes some investment up front to learn. Have you explored Dagster University? courses.dagster.io/courses/dagster-essentials

@@dagsterioThank you so much. It has helped enlighten many things

We explain the logic in a blog post here: arc.net/l/quote/euycfcsm

dlt is better. light years beyond sling imo, and they have a better and more commited and active team. dlt is like the dagster of the lightweight integration libraries.

Hi Joshi. Dagster is an open-source project and all the capabilities that Pedram describes in this video are available in the open-source solution, which you can self-host locally or on-prem. Dagster Cloud offers additional capabilities, which are detailed at dagster.io/cloud or on the pricing page here: dagster.io/pricing