The idea of explaining async by printing out real world tasks is genius

having additional things to try out at the end of the video is awesome. your content is always so great! a sincere thank you :)

Great video, for a while I have been hoping that I would just stumble upon a Python-specific async tutorial.

wow, this is by far the best asyncio tutorial I have encountered. thank you!

Thank you...the idea of waiting on multiple things at once is what made it "click" for me

The best advice I can give to anyone regarding python's async/asyncio is to read the documentation, as there are a lot of edge cases and it's well documented. For example you need to save objects from create_task with a class variable or they get garbage collected

2:53 time.sleep() blocks the current thread, asyncio.sleep() only blocks the current task. Remember that a “task” is not a concept of the Python language itself: it is purely a figment of the asyncio library -- it is a wrapper around a coroutine object, which _is_ a concept of the Python language. Tasks are schedulable entities managed by asyncio.

This channel is like top 3 channels for Python on the web. Thanks for this and keep it up!

Everything i needed and more . Async content is lacking on KZfaq.

4:46 You can have a nonempty pending list even in the absence of a timeout. There is another optional argument to asyncio.wait() which specifies whether to wait for all tasks to complete (the default), or just the first one.

Wow, this is crazy similar to Rust! The difference is pretty much only function names. And such a clear explanation with a good analogy, I can share this with Rust newbies.

Dude your videos are so great - making hard stuff easy - thanks!

Very interesting video, really helps understand how async works

I have been looking forward to this

An excursion into my usual playground! Very nice and succinctly explained. Perhaps another related one would be the concept of async wrappers?

Great video as always

i hit subscribe within first 40 secs, i knew finally i am going to understand this anync maddness

Hands down the best async python tutorial

great video, thank you very much

Wow that's crazy I was just searching for this yesterday

I did not know it had a built in HTML parser! Then why do so many people reach for beautifulsoup? Can you do a comparison, and explain if they are or aren't used for the same goals?

Great intro to async, though definitely feels like something I need to do myself a couple times before I understand it/remember it fully. Really appreciate the suggestions at 13:18 for that!!

10:54 You can't parse [X]HTML with regex. Because HTML can't be parsed by regex. Regex is not a tool that can be used to correctly parse HTML. As I have answered in HTML-and-regex questions here so many times before, the use of regex will not allow you to consume HTML. Regular expressions are a tool that is insufficiently sophisticated to understand the constructs employed by HTML. HTML is not a regular language and hence cannot be parsed by regular expressions. Regex queries are not equipped to break down HTML into its meaningful parts. so many times but it is not getting to me. Even enhanced irregular regular expressions as used by Perl are not up to the task of parsing HTML. You will never make me crack. HTML is a language of sufficient complexity that it cannot be parsed by regular expressions. Even Jon Skeet cannot parse HTML using regular expressions. Every time you attempt to parse HTML with regular expressions, the unholy child weeps the blood of virgins, and Russian hackers pwn your webapp. Parsing HTML with regex summons tainted souls into the realm of the living. HTML and regex go together like love, marriage, and ritual infanticide. The cannot hold it is too late. The force of regex and HTML together in the same conceptual space will destroy your mind like so much watery putty. If you parse HTML with regex you are giving in to Them and their blasphemous ways which doom us all to inhuman toil for the One whose Name cannot be expressed in the Basic Multilingual Plane, he comes. HTML-plus-regexp will liquify the n​erves of the sentient whilst you observe, your psyche withering in the onslaught of horror. Rege̿̔̉x-based HTML parsers are the cancer that is killing StackOverflow it is too late it is too late we cannot be saved the transgression of a chi͡ld ensures regex will consume all living tissue (except for HTML which it cannot, as previously prophesied) dear lord help us how can anyone survive this scourge using regex to parse HTML has doomed humanity to an eternity of dread torture and security holes using regex as a tool to process HTML establishes a breach between this world and the dread realm of c͒ͪo͛ͫrrupt entities (like SGML entities, but more corrupt) a mere glimpse of the world of reg​ex parsers for HTML will ins​tantly transport a programmer's consciousness into a world of ceaseless screaming, he comes, the pestilent slithy regex-infection wil​l devour your HT​ML parser, application and existence for all time like Visual Basic only worse he comes he comes do not fi​ght he com̡e̶s, ̕h̵i​s un̨ho͞ly radiańcé destro҉ying all enli̍̈́̂̈́ghtenment, HTML tags lea͠ki̧n͘g fr̶ǫm ̡yo​͟ur eye͢s̸ ̛l̕ik͏e liq​uid pain, the song of re̸gular exp​ression parsing will exti​nguish the voices of mor​tal man from the sp​here I can see it can you see ̲͚̖͔̙î̩́t̲͎̩̱͔́̋̀ it is beautiful t​he final snuffing of the lie​s of Man ALL IS LOŚ͖̩͇̗̪̏̈́T ALL I​S LOST the pon̷y he comes he c̶̮omes he comes the ich​or permeates all MY FACE MY FACE ᵒh god no NO NOO̼O​O NΘ stop the an​*̶͑̾̾​̅ͫ͏̙̤g͇̫͛͆̾ͫ̑͆l͖͉̗̩̳̟̍ͫͥͨe̠̅s ͎a̧͈͖r̽̾̈́͒͑e n​ot rè̑ͧ̌aͨl̘̝̙̃ͤ͂̾̆ ZA̡͊͠͝LGΌ ISͮ̂҉̯͈͕̹̘̱ TO͇̹̺ͅƝ̴ȳ̳ TH̘Ë͖́̉ ͠P̯͍̭O̚​N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝S̨̥̫͎̭ͯ̿̔̀ͅ

Wow that is some great content. But for me some of the webcrawling concepts are a little distracting. It would be so great if you could add a video for async work on Datapipelines ( get some csv files from an fftp, transfrom them and bulk load them to an relational db for example including logging , taks handling and retry logic). I think this would be a great lesson for many data scientists who are not really educated in data eneneering topics. Anyhow thanks for the homework, i hope i will not fail it :-)

Please do a video on the aiomultiprocess library.

if someone has done the improvements suggested at the end I would love to check out the final version

I thought I was experienced in python. I discovered "await" and "async" with this video.

The syntax seems to have improved a bit since I last tried async programming in python 3.6 Still javascript has the best syntax for async programming in my opinion.

I am looking for tips on how to properly use logging module with asyncio. Supposing all the coroutine functions emit some log messages, it will quickly become very difficult to find out which log message comes from which task as they are all executed simultaneously. Or am i missing something? :)

Another interesting topic would be using asyncio and coroutines in GUI programming. Because every GUI framework implements its own event loop. The key thing about asyncio is it provides a standard event-loop API for Python. You can replace its default event loop with wrappers around alternative event loops, and thereby use coroutine tasks in GUI apps.

While async is not threading, it would be great if James could clarify how different tasks may communicate properly, for example what happens with variables shared between tasks.

- Am I awaited? - You are awaited, shiny and chrome!

great vid

Yes, HTML is not a regular language, however your problem might be. For example, if you are searching for the only id="..." on the page, you can totally do that with a regex.

I'm hyped up

I remember one day I saw an example of this with a script that downloads videos from internet, instead of waiting to download each one sequentially, they used asyncio, but I was wondering, if the downloads happen simultaneously, won't that give less bandwidth to each video and then making each one download slower? How is it still giving benefits with this approach? (Or maybe my understanding of how things are downloaded is not right hehe)

Nice intro, a bit fast into those queues.

Does the HTML parser also support XML? Is there also one for JSON??

very common error: 32 if events._get_running_loop() is not None: ---> 33 raise RuntimeError( 34 "asyncio.run() cannot be called from a running event loop") then use: await main()

this async does not directly imply multithreading/core, right?

any particular reason you didn't use aiohttp?

The biggest problem I had with trying to implement concurrency is working on the same data from different threads. i.e. multiple threads insert into the same dict/list when they are done with their task.

1:57 Best to call it a “coroutine object”.

How much does it cost for cpu? Im actually curious.

finally an async video 🤩 discord gang 🤙

good async

I've written asyncio applications and it quickly becomes complicated once you mix multiprocessing (external processes having their own ioloop), listening on sockets and pipes (each having slightly different implementation that requires you to write different code for each), your code becomes too complicated, you have to manage everything yourself and navigate abstractions that do the same things but are different (reading lines coming from sockets vs pipes). To give an comparison, it's like if python only exposed an API for threading where you have to create the thread yourself, create local storage yourself and schedule work to run in the thread via a queue, for every thread, all by yourself. That'd be terrible API to use... well asyncio is at that level imo, not high enough (yet) to be awesome.

5yrs in python, never did threading, multiprocessing or async. Am I alone? Keep postponing learning it, seems too difficult.

Web crawlers should handle AJAX (XMLHttpRequest) generated dynamic content, shouldn't they? (I wonder how much of the total content is available these days, I guess not a trivial amount of percentage...) Nice video BTW.

Please make one for fast api calls. And not some low latency Wikipedia api calls

Discord gang!

Well, nothing is wrong in the original problem. Your only cook's hat got dirty and thus you bring it to the laundry, but without it you can't bake. However, today somewhere in between 10:00 and 15:00 you've got a courier delivery and thus must be at home. That leads to the situation when you first wait until the Amazon delivery (task with 3h timeout), then you go to the laundry and wait until your clothes wash and dry, and only then you can start baking a cake.

"asyncio is very much an afterthought" so true :( I did my own delve into asyncio very recently and I didn't have a good time. Maybe it's just my poor docs reading skills tho

Btw this background looks very gray comparing to this one from new UI

What would be the advantage of using async instead of Thread? from threading import Thread import time import random def slow_function(thread_index): time.sleep(random.randint(1, 10)) # simulates waiting time for an API call response for example print("Thread {} done!".format(thread_index)) def run_threads(): threads = [] for thread_index in range(5): individual_thread = Thread(target=slow_function, args=(thread_index,)) threads.append(individual_thread) individual_thread.start() # at this point threads are running independently from each other and the main flow of application print("Main flow of application") for individual_thread in threads: individual_thread.join() # joining threads insures that all threads are done before moving further in the flow of application print("All threads are done") run_threads() p.s. great video as always!

Nice, but I find your example too complicated for a video. And I'm an experienced async programmer. I can follow, but at least a diagram with the queues would be helpful for global comprehension. And a diagram of how the ioloop schedules coroutines, because there are a lot of assumptions and I believe it's hard for beginners to follow without visual cues.

"it doesnt matter if you use c or python, facebook wont respond any faster" rustaceans be like: lets rewrite whole facebook in rust now

notification gang

How did you fight off Tony the Pony?

I really try but somehow the async programming is just too much for my brain.

Waiting for part 2: adding this scrapped data into GraphQL with strawberry orm( source can be : stackoverflow issues)...

why not have one task for each crawl() and have crawl call asyncio.create_task(crawl(url)) for each found url? then you don't need worker functions or a queue.

Title: "Intro to async..." ..."I claim that you already understand asynchronous programming" Yes, but no

0:15 ... welp... another video I just cannot watch.

3:35 is why I think asyncio in python is pointless. You can EASILY replicate this behavious with more control by multithreading or multiprocessing. It will be more readable too

The initial code is how adhd brain works 🥲😂