You're NOT Managing Your Memory Properly | Python Generators (Yield)

Рет қаралды 12,763

Күн бұрын

How using Python generators (aka the yield statement) can be advantageous to program design and memory efficiency and management.
Source code - github.com/daniel-boctor/Dani...
0:00 - Introduction
0:25 - Lists
2:51 - Generators
5:47 - Rolling Returns
8:00 - Outro

Пікірлер: 42

@adityashukla7849 11 ай бұрын

Wow. I was just studying generators, but was finding it slightly difficult and unnecessary. Now I understand it's importance. Thank you for this video 🙏🏼

@DanielBoctor 11 ай бұрын

I'm glad I was able to help! Thanks for watching

@lilDaveist 11 ай бұрын

For a in depth view I recommend James Powell talks. PyData Seattle 2017 goes over generators about an hour in or so. That talk was in my mind and helped me greatly in a project I’ve done. 😃

@ar7eniyan 11 ай бұрын

1:59 the graph here actually confuses the viewer: in the first example memory consumption of the whole loop grows quadratically and in the second it grows linearly (like the graph shows for the first one). sys.getsizeof returns linearly growing size because list type stores only references to its values, but the strings themselves also grow linearly, which gives an overall consumption of O(n^2) in the first case

@DanielBoctor 11 ай бұрын

Very observant, thanks for sharing! The intention was to make the general concepts independent from the specific example used. The fact that the list contained strings was chosen in this case as it made for a better illustration. If you were to store integers in the list however, instead of strings, the size of each integer would not increase with the value. This would then result in a linear space complexity (O(n)). The intention wasn't to focus too much on this specific example, but you are certainly correct. I definitely should have made this more clear though! Thanks for pointing it out.

@re.liable 11 ай бұрын

And if for some reason you want to see all of the content to be generated, you can just cast it to a list e.g. list(your_generator) or [*your_generator]

@DanielBoctor 11 ай бұрын

This is definitely true!

@ermalgashimramori 10 ай бұрын

Great example, quality content

@CaseyFahey 11 ай бұрын

New subscriber, like your content, REALLY like that you just jump right into it. I think creators sometimes forget that minutes wasted really add up over time, thank you for wasting no time!

@DanielBoctor 11 ай бұрын

Thanks for the feedback! As a viewer, I completely agree with this myself. Glad you enjoyed

@williamrich3909 11 ай бұрын

Excellent clear and conscise. Looking forward to more content!

@DanielBoctor 11 ай бұрын

Thanks for watching! I appreciate the support

@JOHNSMITH-ve3rq 11 ай бұрын

Incredible explanation

@DanielBoctor 11 ай бұрын

I'm glad it was helpful

@domenicfieldhouse5644 11 ай бұрын

This is one of the cleanest edited videos I've seen, and great info

@DanielBoctor 11 ай бұрын

Glad you enjoyed!!

@erickgaldino7453 11 ай бұрын

This channel is criminally underrated

@DanielBoctor 11 ай бұрын

I glad you enjoyed! I appreciate the support.😊I got more on the way!

@TheControlMastr 11 ай бұрын

Your videos make me want to learn Python all over again 😮 amazing video!

@DanielBoctor 11 ай бұрын

I'm glad you enjoyed! Thanks for watching😊

@abhishekchoudhury 11 ай бұрын

Love the video, @DanielBoctor. Would love to see videos on advanced Django!

@DanielBoctor 11 ай бұрын

I'm glad you enjoyed! Django content is certainly on the way. I'll be uploading more frequently once the summer ends too.

@Darkev77 6 ай бұрын

Very practical insights, thanks a lot for this

@DanielBoctor 6 ай бұрын

I'm glad you found it helpful! Thanks for the support 😊

@Simple_Simon_UK 11 ай бұрын

Is there an equivalent to Generators in Cobol-72? Only joking. Compilers are wonderful things. Their enrichment of capabilities are opening up opportunities beyond our wildest dreams.

@anon_y_mousse 11 ай бұрын

Okay, so basically, don't process or generate data before you really need to. Good message, just needs some more real world examples.

@jacobjayme6280 11 ай бұрын

Very epic!

@waifuking7961 11 ай бұрын

pogchamp

@BaronXOfficial 11 ай бұрын

Thanks for sharing!

@DanielBoctor 11 ай бұрын

Glad you liked it 😊

@fire17102 11 ай бұрын

I almost always use yield generators ! Mostly because it's faster lazyier processing, for better speed performance, and or asynchronousity manipulation. But it's wonderful to see such a clear example of how this is also memory efficient and a must for big data processing. Thanks for the regular bracket trick! (ftw) didnt know about that! Love this channel! Love python! Subscribed! All the best

@DanielBoctor 11 ай бұрын

Thanks, I appreciate it! I'm glad you enjoyed. 😊 More to come!

@aflous 11 ай бұрын

@fire17102 Generators don't enhance performance. In fact, in most cases they will even hinder your program performance as the overhead for creating the full list in memory can be time costly. Only use generators when you know you would need a single value at a time. The benefits would mainly be memory effeciency, not better performance. There is however a scenario where you could incur some performance gains, and that's when you have a pipelined workflow. Having a generator yielding output from one precessing step to another can lead to huge performance gains compared to waiting to gather all the data at once in each step

@tunafllsh 11 ай бұрын

In case of list comprehension vs generator, most of the time list comprehension is much faster. If you don't have memory issue and have constant size elements, the speed is worth it. So in rolling returns, it is fine to use lists for windows and it would be much faster.

@darshanrajpattanaik2154 11 ай бұрын

Ery informative video... What are your thoughts about using it when using to (i) receive api get requests followed by a loop on the response data (ii) to sort the data without using it in any loop?

@DanielBoctor 11 ай бұрын

Thanks for watching! i) If you make a single API call, store the results in memory, and than iterate over the results with a generator, this would not be advantageous as the entire result would already need to be stored memory in the first place. api_response = requests.get(...) # API call outside of generator would be ineffective if all of your data is already in memory If you however make successive API calls within a generator, where the API returns different data each call, this would be a valid use of generators, as you would only need to store a slice of the data in memory at a time. def generator(): yield requests.get(...) # API call inside of generator would be effective as you can yield a single piece of the results, if the API returns different results each time it is called ii) Not entirely sure what you mean by this. Perhaps you could elaborate?

@darshanrajpattanaik2154 11 ай бұрын

@@DanielBoctor So the generator gives out a set of elements that I need to sort, but I get only one element one by one, can I perform sort operation in the generator element without using a for loop?

@DanielBoctor 11 ай бұрын

@@darshanrajpattanaik2154 I see. It actually is possible to sort a collection of data without storing it all in memory at once, with a technique called 'external sorting'. A common example of this would be an external merge sort, where you break up your dataset into smaller chunks, sort them separately, and than merge the sorted chunks together using a priority queue or a similar approach. You would likely still need some form of loop depending on the approach, however the generator would be effective for memory usage if paired with an external sorting algorithm. This topic goes a little deeper, and I would advise looking more into external sorting if you want to gain a further understanding. Hope this helps!

@john.dough. 11 ай бұрын

it seems like you're using the word "residual" to mean "minuscule". Given that "residual" means "remaining after the greater part or quantity has gone.", I'm not quite clear what greater part of the function's memory you'd be referring to.

@DanielBoctor 11 ай бұрын

Thanks for bringing this up. I used the word "residual" to signify that the size of the generator is a constant value that 'remains' after we convert a list of any size to a generator. In this sense it could be considered residual, as it is what we are left with (as opposed to the generator taking up no memory at all, since it needs some memory for overhead). That was just my thought process, and I definitely could have made this more clear. I appreciate the feedback!