STOP Using Plain Python Scripts! Do this instead (5 reasons)
Рет қаралды 13,245
Күн бұрынSign up for the totally free tier of Prefect Cloud here: prefec.tv/doug-mercer
-----------------------------------------
Sign up for 1-on-1 coaching at dougmercer.dev
-----------------------------------------
One of the most frustrating parts of the workday is doing something that you know could be automated, but just… isn’t yet.
In this video, we use Prefect to schedule a Python script to run every week. After, we find that scheduling was really only the first of five problems, and we use Prefect to solve all of them.
You'll learn how just one Python library, Prefect, and a few minor changes to your code can allow you to:
- Schedule your Python Script to run on local or remote systems
- Handle Errors with Retries
- Monitor your workflows in an incredibly powerful web-based UI
- Parallelize and scale your code's using on-prem compute clusters or Cloud platforms
- Persist Python results and Markdown reports
- and more...
Prefect changed the way that our data science and machine learning focused research team approached scripting.
Chapters
---------------
00:00 Intro
01:38 I want to take Monday off
03:28 I don’t want to babysit code
06:35 No one knows what’s going on
10:33 My code is embarrassingly slow
12:41 I keep throwing away our data
13:47 Wrap-Up
Special thanks to Siara M. for making artwork used in the intro sequence.
#python #prefect #scheduling #scripting #mlops
@dougmercer 5 ай бұрын
Sign up for the totally free tier of Prefect Cloud here: prefec.tv/doug-mercer
@khawarshehzad487 5 ай бұрын
Appreciate the effort you put in your videos. Thanks for sharing. keep it up!
@dougmercer 5 ай бұрын
Thanks so much! Will do =]
@LeveragedAlpha 5 ай бұрын
Could I say this video changed my life? I have gone on a prefect binge. And wow. Already put it to use, and wished I had it all these years instead of cron jobs.
@dougmercer 5 ай бұрын
That's so awesome! It was a game changer for us to. Thanks so much for sharing-- your comment made my morning =]
@LeveragedAlpha 5 ай бұрын
@@dougmercer No - thank you, Doug! So glad I had the privilege of discovering your youtube channel. It's already my favorite technical channels (and among my favorites, overall). You go deep into providing actionable advice for solving problems I face, with incredibly high-quality content, but without the puffery and cringe thumbnails etc. It has already helped me improve as a developer, and I look forward to continuing by watching (and implementing) more videos.
@Slightlymisshapen 10 күн бұрын
Beautiful presentation. I love your descriptions on libraries. The illustrative code break down puts context and concrete examples on what would otherwise be yet more abstract documentation. It helps to grasp the utility of the libraries uou describe not just in this video but all of them. Fantastic work.
@dougmercer 10 күн бұрын
Aw, thanks! glad you enjoyed it, and thanks for your super nice comment =]
@adamgkruger 2 ай бұрын
Such a good video! TY for the content. I would 100% take a more indepth course on using Prefect.
@dougmercer 2 ай бұрын
Thanks Adam! I may do another Prefect video in the future, but in the meantime you can maybe check out prefect.io for their blog/docs or their KZfaq channel @PrefectIO. They have a ton of cool integrations with other software that I barely scratched the surface of
@maurolimaok Ай бұрын
Nice channel. Hope it grows.
@dougmercer Ай бұрын
Thanks Mauro! I hope so too🤞
@MrApresmoiledeluge 5 ай бұрын
Fantastic stuff!
@dougmercer 5 ай бұрын
Thanks! I appreciate it =]
@MaxShapira2real 5 ай бұрын
Thank you for the sublime content! I'm curious about the font in your text editor (1:57).
@dougmercer 5 ай бұрын
Hey Max! That's Anonymous Pro. It's available for free on Google Fonts, fonts.google.com/specimen/Anonymous+Pro
@LucasVianaCodes 2 ай бұрын
Your videos are incredible! Would you share your editing stack? Cheers!
@dougmercer 2 ай бұрын
Hey Lucas! Thanks so much =] I currently edit on... Davinci Resolve 18.1 Krokodove plugin for Fusion (which is preventing me from updating to latest Davinci Resolve, because BlackMagic is dragging their feet on releasing an updated SDK 😡) A `pygments` formatter I wrote for converting code text to formatted text plus node for my code, which I then animate in Fusion in Resolve. (Which is not automatic, and a total pain in the butt) Epidemic sound for music Storyblocks/envato for other stock media In the future, I'd like to better automate my code animations. Previously I have dumped a ton of hours into trying to make a `manim` (python animation library) extension for animating the difference in code, but manim's Code mobject is really poorly implemented and requires too many workarounds to deal with. I've considered using reveal.js for code animations, but I think the result would be worse for what I do.
@dirtdart81 5 ай бұрын
This looks really exciting, doesnt seem to do much new, bit really easy to use and all in one place. I can think of some places to use it at work already and am definitely keeping prefect in mind. Ignore the trolls! 😂
@dougmercer 5 ай бұрын
Hah! thanks 😂
@alexandreramos8484 Ай бұрын
The combo prefect+dbt is amazing
@glaucojunior 5 ай бұрын
I used to run an Apache Airflow installation on an EC2 from AWS, to me its look a lot of an Airflow alternative, is that correct? Can you tell about advantages (and disadvantages) between they, if you know of course?
@dougmercer 5 ай бұрын
Sure! Quick caveat-- I've only used Airflow a bit, but I was so frustrated with the on-boarding and user experience that I gave up on it and have used Prefect ever since. My take is that Prefect is a more Python-native library. Writing Prefect flows/tasks feels a lot more like just writing Python than Airflow, which feels like I'm writing config files with a thin Python wrapper. There's actually quite a few breakdowns comparing the two on the web. - www.prefect.io/prefect-vs-airflow - neptune.ai/blog/argo-vs-airflow-vs-prefect-differences Beyond that, Prefect's blog has an interesting deep dive point out some frustrating "Airflow-isms" and explaining how Airflow can break down at scale-- www.prefect.io/blog/implications-of-scaling-airflow Some pros/cons that were most relevant to me... Prefect Pros: - more modern, user friendly API - really easy to get started (pip install one library, add some decorators, run a terminal command to get a server running for monitoring) - supports dynamic workflows (rather just static DAGs) Prefect Cons: - Some really cool features are Prefect Cloud exclusive. Airflow Pros: - Oldest workflow orchestration out there, so wide user community/documentation (i.e. Mature). Airflow Cons: - I hate their API. - Static DAGs
@glaucojunior 5 ай бұрын
@@dougmercer wow, thank so much for the information, just knwoed your channel and loved it, cheers from Brazil!
@dougmercer 5 ай бұрын
Cheers!
@fffklan3986 Ай бұрын
pros of airflow: - Mature, tried and tested, runs on 100% open source tech. - multiple ways of writing DAGS (they have a newer taskflow api) - If you're ever looking into using it in a production environment, entry level config of AWS managed airflow can be as little as $400/mo. - do cool stuff, learn data engineering, fun fun fun cons: - if you don't know linux/docker well, good luck. - xcom can be a painpoint (limits the amount of data that can be passed between tasks, meaning that large amounts of data need to be stored between tasks). - you have to use your brainpower and thoughts to accomplish things. pros of prefect: - easy cons: - paid version starts at $1,850. - can't really put it on a resume as a skill (if ur trying to get a job) - sponsoring youtubers for publicity in conclusion: if u want something done quick, use plain old cron. if you want a trusted, robust orchestration system, use airflow.
@martimlobao Ай бұрын
I’ve used both Airflow and Prefect for work, and can say that setting up and running Airflow is a million times more painful that Prefect. OTOH, Prefect can become extremely expensive with even just a few users, whereas Airflow is dirt cheap, which is why we ended up switching from Prefect to Airflow 😢
@famed3777 2 ай бұрын
This is how promotional videos should look like!
@dougmercer 2 ай бұрын
Thanks! It helps that I had already used their library for years, so I knew what I wanted to make before they reached out.
@adamgkruger 2 ай бұрын
Thanks!
@dougmercer 2 ай бұрын
Wow! Thanks Adam, you made my whole week =]
@FadkinsDiet 2 ай бұрын
How is this different from Apache Airflow?
@dougmercer 2 ай бұрын
I left a pretty detailed reply on @glaucojunior's comment, so maybe check that out High level gist is-- Prefect is a bit more modern, feels like you're writing Python, and has more dynamic features. Airflow is, uh, mature, but feels like you're basically just writing config files laundered. Here's a third party comparison of the two neptune.ai/blog/argo-vs-airflow-vs-prefect-differences
@dinoscheidt Ай бұрын
Looks like a python only alternative to Kestra - and still no sight of the software engineering practice to maintain build and ETL pipelines next to code in a CICD fashion… 😮💨
@robosergTV Ай бұрын
meh, there are alternatives like Flyte, Vertex AI, Kubeflow, etc. Or one can use Kedro to write the pipeline and export it to run in the cloud
@dougmercer Ай бұрын
Oh neat, I'll have to look into them a bit more. I've seen kubeflow but not the other three
@BenHammond 28 күн бұрын
…tech people are weird in their level of reflex dismissiveness (I say this as a tech person).
@doofyguck6186 Ай бұрын
Airflow user here 😂
@dougmercer Ай бұрын
Airflow is totally fine 😂! It's been the standard for a while for a reason... That said, when I tried to use it, I felt like I was basically creating a config file through Python kwargs instead of writing Python code. Wasn't the right choice for me
@doofyguck6186 Ай бұрын
@@dougmercer the way I tend to use Airflow (due to my company's architecture) is composing and extending the operators to fit our needs. So in our main airflow DAG file it's simply a config file but it uses our custom made extended operators. That's where we write most of our logic.
@dearheart2 Ай бұрын
Nice video, just wish the music was mute/gone.
@dougmercer Ай бұрын
Sorry about that. I saw you posted a similar comment on several videos. I agree that it'd be awesome if KZfaq supported multichannel audio that allowed the viewer to entirely mute music. In my past videos, I've definitely mixed the audio too loud. I really tried to keep it quiet in the 1brc, and will continue to be more mindful about future videos. However, I probably will never have a video without music. In my personal opinion, music has the potential to make videos more engaging. Admittedly, sometimes my mixing and song selection misses the mark. So, thanks for at least attempting to watch my videos, but sorry about the VO/music mix
@mohak9102 Ай бұрын
How is this better than airflow?
@dougmercer Ай бұрын
"better" is probably a matter of taste. I prefer it because it feels more like writing Python and less like writing a config file. There are several comparisons out there that dive into the differences. Here's a third party link comparing them + another approach (Argo) neptune.ai/blog/argo-vs-airflow-vs-prefect-differences
@rafaeldbx Ай бұрын
how do I make my application more reliable? rely on some dumb cloud ai startup. these all feel like already solved problems, this is just a new set of tooling the developer has to learn, except this time there's probably next to no documentation
@dougmercer Ай бұрын
Prefect's open source library is useful in and of itself-- no cloud required. Also, their documentation is good ¯\_(ツ)_/¯
@ardenthebibliophile Ай бұрын
Novice to automation workflows: why not jenkins?
@dougmercer Ай бұрын
Typically Jenkins is used for managing CI/CD pipelines (automating build, test, deployment). Though, I suppose you could use it for data pipelining tasks 🤔 Prefect is a bit more specifically tailored for data engineering, ML ops, or just generally writing pipelines with Python Code (and would not be well suited for administering CI/CD pipelines out of the box). To my mind, similar technologies to Jenkins include GitHub Actions, Gitlab CI, or bamboo. Similar technologies to Prefect include Airflow, Argo, Luigi, Dagster.
@ardenthebibliophile Ай бұрын
@@dougmercer appreciate it. The company I work for, while very large, is relatively new to the data engineering space. We've had to learn a lot as regular engineers and scientists as we had a bit of huxterism a couple years ago where worlds were promised and delivery was thin (at best). Add in our technology groups are limited in what is approved for use and it's a complicated scenario that Jenkins, somehow, made it through. Will check out airflow and Argo too!
@dougmercer Ай бұрын
My day job sounds really similar to yours. Hopefully you can find something that's approved!
@ardenthebibliophile Ай бұрын
@@dougmercer pharmaceutical industry is what I work in
@dougmercer Ай бұрын
I work with the government. Getting software approved is a whole process...
@marlan__ 5 ай бұрын
Why not just use Cron?
@dougmercer 5 ай бұрын
Good question! Basically-- scheduling is only the first of five problems (and is the only problem that CRON would solve). Later in the video, you'll see why. Our code had flaky upstream processes that needed retries, our team had no idea what was going on mid-script run (where the real script took multiple days), we weren't scaling/distributing work effectively, and we didn't have a good system for keeping track of our pipeline's results. CRON alone wouldn't be enough without writing our own... well.. workflow orchestration system. Prefect gave me a ton of features out of the box, and I was able to turn around a refactored script in an afternoon that blew my team's minds.
@dougmercer 5 ай бұрын
@ordinarygg As someone who has written their own general purpose pipelining library (because I wanted to) and multiple web dashboards (using Flask, FastAPI, Streamlit, Panel, etc.), I can promise you that it's way easier to pip install a dependency, add a few decorators, and run a few terminal commands than it is to replicate all the useful functionality that Prefect provides. Check out my comment I left above in response to @marlan__ to get a better idea where I'm coming from.
@Redditard Ай бұрын
Me: This all can be fixed using cron and internal error handling.
@AniaKovas Ай бұрын
Because it's about more than that, Cron is a crude tool anyway, but also you use the tools you know and are willing to learn. He's using Cron anyhow, and there's, just like I said, more going on. That's the point. I teach this stuff, this is a good lesson. Didn't know about prefect, and this is a good place for my students to look.
@arnabchatterjee2094 5 күн бұрын
basically prefect is celery with steroids
@dougmercer 5 күн бұрын
That's typically how I use the open source library! I recently spoke with someone from Prefect at PyCon, and they said the automation features and more celery-esque features are coming to the OS library soon. So keep an eye on that =]
@danielot Ай бұрын
Talented but definitely need a senior engineer or a tech lead. There are a lot of ways to solve all of these problems (khm airflow, mlflow😂) Was this a real situation or just made up to help illustrate why this is in fact a great library?
@dougmercer Ай бұрын
Yeah, there are a lot of ways to solve these sort of problems. We chose this and it worked well for us ¯\_(ツ)_/¯
@damonguzman 4 ай бұрын
This feels like a 15 minute ad. I dislike this video greatly.
@dougmercer 4 ай бұрын
Sorry you feel that way... Although the video is sponsored, this is a true story from my day job, and the script doesn't make any claims I don't stand by. That said, feedback taken. Thanks for commenting.
@kc12394 3 ай бұрын
Dude he literally said it's sponsored in the first 90 seconds of this video and you're still complaining it "feels" like an ad. No one is forcing you to watch a video. You can turn off the video as soon as you hear that it's sponsored you know...
The Pure State
Рет қаралды 20 М.
Tom Delalande
Рет қаралды 89 М.