No video

How Sets Can Truly OPTIMIZE Your Python Code

  Рет қаралды 43,562

Indently

Indently

Жыл бұрын

Today we will be looking at how sets can be used to significantly optimize your Python code.
▶ Become job-ready with Python:
www.indently.io
▶ Follow me on Instagram:
/ indentlyreels

Пікірлер: 88
@Indently
@Indently Жыл бұрын
As one of you were kind enough to point out, the speed_difference calculation was completely off. It should have been obvious to me immediately, but sometimes when you're writing tutorials these details pass you. In the example I provided, sets are MUCH faster than 99%. Sorry for that bad snippet of silly math :)
@Dent42
@Dent42 Жыл бұрын
If it makes you feel any better, GitHub has been using the same incorrect calculation for their Copilot efficiency gains for months now, saying their product increases efficiency by 55% (presumably coming from the fact that tasks took 55% less time), when in reality it increases efficiency by 127%
@Indently
@Indently Жыл бұрын
That's a hilarious little piece of trivia, thanks for sharing 😂
@callbettersaul
@callbettersaul Жыл бұрын
I've thought about it and I'm still quite certain, that the math is correct in the video. My reason for believing this is that the 100% represents the time the list took to complete. That means 100% faster would mean that set took 100% less time to complete (which would be 0s). However, if you compared it the other way around (list took x% longer to complete), then it would be quite a large number, for example, at 3:00 it would be correct to say, that the list was 2,848,073% slower, than the set or that the list took 2,848,073% more time to complete. But the other way around, it can't go over 100%, because it would go into negatives that way.
@callbettersaul
@callbettersaul Жыл бұрын
@@Dent42 Isn't it true tho? When they're comparing to the old value and get the new value, that is 45% of the original value. That means the old value was reduced/improved by 55%. Their efficiency is based on the time it took and the time can't be 127% less than the original time, because nothing takes below 0s to complete. Edit: Nevermind, when talking about efficiency/performance increase, then it would be 122% I think (tried 3 different formulas, all gave 122%, not 127%).
@Indently
@Indently Жыл бұрын
I like the idea of something taking 100% less time to complete
@callbettersaul
@callbettersaul Жыл бұрын
To add to this, when lists check if an element is there, it uses '==' operation on each member, which is why it takes so long. However, set takes the hashcode of the element (hash(element)) and checks if that hashcode is in the table. In your own classes you can modify the behaviour of the operations, by writing your own implementation of the magic methods __eq__() and __hash__().
@BrianStDenis-pj1tq
@BrianStDenis-pj1tq Жыл бұрын
I'd suggest that for many applications, using a dict() is what you want. Not only do you get the speed of a set, you can store a value as well. Dict or list are the go-tos, while set can be used if you really need the speed of a dict and you really don't want to store anything.
@vuvu7005
@vuvu7005 Жыл бұрын
if you whant to handle more infos (like counts, index in an array, order, etc...) you can use dicts, the keys of the dict will behave practicaly the same as the set and you will be able to store aditional infos
@pistolangpaltik
@pistolangpaltik Жыл бұрын
I'm new to programming and I'm, as always, learning something new from your short videos. However, there is something wrong with the calculation on speed increase. As the result of 'set' approaches zero, the calculation approaches 100% and never beyond. I think you need to have the result of set as base. Thanks for the videos!
@Indently
@Indently Жыл бұрын
That's absolutely my fault for not double checking that bit, I used ChatGPT for inspiration and didn't notice that until you brought it up. I need to be more strict on the code it generates. Thanks for bringing it up!
@billvvoods
@billvvoods Жыл бұрын
@@Indently Not sure if it was gpt3 or 4 that you used because 3 is notorious for math mistakes. In any case your video was very good. Straight forward and to the point.
@callbettersaul
@callbettersaul Жыл бұрын
It approached zero, because the list was the base and going above 100% would mean, that the set took negative seconds to complete. If he used set as base, he would have to write that the list was slower by 2,848,073% for example. In video it stated that set was faster by
@GabiruM
@GabiruM Жыл бұрын
I love your videos, man. They are simple, effective and very useful. Hope you get the recognition you deserve.
@CatalinSoare
@CatalinSoare Жыл бұрын
While I did know that sets were a lot faster than lists, I didn't know by how much. This was very interesting and informative. However, what I didn't know and just saw in your video was pep-515; being able to separate digits with underscores for visual aid seems awesome!
@NickDanger3
@NickDanger3 Жыл бұрын
The fact that sets are implemented as hash tables, and that the performance is O(1) are not two separate “reasons”: they are the exact same reason expressed in two different ways.
@apsifiable
@apsifiable Жыл бұрын
You should search zero instead of 999_999. Or in other words, not that your conclusion isn't correct, but it is almost like some marketing department came up with your example: the most unoptimal case from the point of view of lists.
@Efebur
@Efebur Жыл бұрын
That is the point though. You should compare worst cases (or at least average), not best cases. Obviously if he searched for 0, it would come up immediately for his list. Instead, he should shuffle the list and search for multiple random numbers and make an average.
@apsifiable
@apsifiable Жыл бұрын
@@Efebur Indeed. What I learned is that sarchasm is a difficult sport.
@Indently
@Indently Жыл бұрын
The amount of friends I've lost through using sarcasm in text form 😂
@dcollett
@dcollett Жыл бұрын
Thanks so much! Your videos are the best on the internet.
@WinfriedKastner
@WinfriedKastner Жыл бұрын
And he speaks the best and understanding English of the whole world!!
@Indently
@Indently Жыл бұрын
That's too kind, Winfried :)
@WinfriedKastner
@WinfriedKastner Жыл бұрын
@@Indently That's absolutely true what I said. I'm assuming you're not British born or something because of your first name. Ialso bought your Udemy course last weekend, even though I know Python well by now. But it is a pleasure to work through and follow. And thank you for your great videos and hopefully many more to come on KZfaq!!!
@VisibleWater
@VisibleWater 14 күн бұрын
Very good explanation, thank you
@the_infinity_snake
@the_infinity_snake 2 ай бұрын
The thumbnail showed the word "why", so i thought this video would explain the reason why a set is faster to search for a specific value than a list.
@VisceralBoredom
@VisceralBoredom 14 күн бұрын
3:35
@alidev425
@alidev425 Жыл бұрын
I'm, as always, learning something new from your short videos. great content keep it up 👌👌
@Mulakulu
@Mulakulu Жыл бұрын
I believe you made a mistake with the percentage code and result. Your result doesn't show how many times faster the fast code was. That would be determined by 9.16/(3.2•10^-5)=286250, so a quarter of a million times faster. Basically the fast code could run a quarter of a million times in the same time the slow code could do it once. I recognize your way of calculating it to be the way to find percentage error, where 100% seems like a reasonable result, but it's not what you're looking for I presume.
@James_08_07
@James_08_07 Жыл бұрын
Just about to say exactly this, the limit in his approach is 100% faster which is accurate but not the best way IMO to show how much faster something is.
@Indently
@Indently Жыл бұрын
The math is silly, I was fooled by the shiny ChatGPT-4 model. I need to be much more harsh on it.
@laz272727
@laz272727 Жыл бұрын
One thing sets do as opposed to lists is eat twice the ram. It's not twice of a lot of ram, since lists tend to be storing pointers and those are int sized, but it's still more.
@Indently
@Indently Жыл бұрын
Definitely interesting :)
@infiniteplanes5775
@infiniteplanes5775 Жыл бұрын
There are always trade offs, and I am interested in hearing some more stuff about lists vs sets
@laz272727
@laz272727 Жыл бұрын
@Infinite Planes There are generally two types of lists used commonly. Lists you see by default in OOP languages are generally Array-backed - literally an array of a size similar to the list, but rounded to some pre-set number. They're more expensive to edit, since you need to make a new array every time, but can be accessed by index O(1) (it's a sum operation of array's beginning pointer and the index). They are generally the best if you need an array you would want to edit later. However, in Functional languages, the default is a Linked List - where every list member has a pointer to the next one. This means editing it is easy, since the next member can be anywhere in memory and you just have to change the pointer; however, finding a specific member is very difficult since you literally have to iterate over the entire piecemeal list. Also, since they store one extra pointer, they're slightly bigger. They are better for FP-like usages of lists, though.
@laz272727
@laz272727 Жыл бұрын
@Infinite Planes There are two types of construction for tables (the video talks about sets, but values don't have to be unique and the difference in speed is very minor - in fact, many languages implement sets as tables without a value, or with the value being the actual value and key being hashed). Note that while calculating space taken by tables is relatively easy (they, like array lists, are slightly bigger than their elements put together), explaining them isn't quite as easy. They have a lot of optimizations related to the fact that they're meant for really fast lookups by keys. Hell, in a perfect world, hashmaps are O(1) (if you use an array indexed by the hash - obviously that is nowhere near space efficient).
@IlariVallivaara
@IlariVallivaara Жыл бұрын
​@@laz272727 Array-backed lists are not necessarily or even usually more expensive to edit: it all depends on how you edit it. Further, linked lists offer asymptotic removal/addition benefits only if you can obtain the corresponding pointer for free or very cheaply, which is not usually the case. It is a common misconception, though, because people don't realize the cost of traversing the linked list to obtain the pointer. You also seem to mix random (index-based) access to finding a specific number in the collection. These are completely different operations, and the latter is O(n) for both (unsorted) arrays and linked lists.
@devocracy1089
@devocracy1089 Жыл бұрын
Very cool!
@artistpw
@artistpw 3 ай бұрын
Sets seem far more efficient than lists. Nice you can convert them back and forth to get items sorted.
@abdultheseekerofknowledge4453
@abdultheseekerofknowledge4453 Жыл бұрын
Wow bro, everytime i see one of your videos i immediately know i'll learn something new, thank you
@Smbrine
@Smbrine Жыл бұрын
Just use ctypes. Don't be lazy, c/cpp is about 100-200 times faster (it takes about 0.2sec to load a cpp function in python but even considering this it will be 10 times faster at least)
@Indently
@Indently Жыл бұрын
We use Python because we are lazy.
@Smbrine
@Smbrine Жыл бұрын
@@Indently python is also lazy so that sums into a very poor performance :)
@Indently
@Indently Жыл бұрын
@@Smbrine Poor performance, big salary 😎
@tanmaypatel4152
@tanmaypatel4152 Жыл бұрын
Which IDE are you using in this video?
@mailonbido
@mailonbido Жыл бұрын
im curious too
@tanmaypatel4152
@tanmaypatel4152 Жыл бұрын
@@mailonbido I got it. It's pycharm. It looks different because of the latest UI update in the 2023 version I guess.
@mailonbido
@mailonbido Жыл бұрын
@@tanmaypatel4152 oh nice, thanks!
@tanmaypatel4152
@tanmaypatel4152 Жыл бұрын
@@mailonbido You're welcome
@sirynka
@sirynka Жыл бұрын
My first thought was that you're going to talk about list vs tupple comparation and I was confused about how immutability could make iteration faster.
@octonwoomy1150
@octonwoomy1150 Жыл бұрын
W video as always, thanks for the great content
@sushantpawar3703
@sushantpawar3703 Жыл бұрын
What about tuples compared to lists?
@k.kaiserahmed8013
@k.kaiserahmed8013 Жыл бұрын
Somewhat slow... Just a lil bit....
@MilChamp1
@MilChamp1 Жыл бұрын
tuples are faster to create and index, and very similar to search through. Lists are useful for collections that need to be edited.
@k.chriscaldwell4141
@k.chriscaldwell4141 Жыл бұрын
Good info. Thanks.
@thegothaur
@thegothaur 5 ай бұрын
funny. it is O(1) because of sets being hash tables :)
@LogicVertix
@LogicVertix Жыл бұрын
Thanks For this valuable practical , I have only studied this in books only. Do we have any other way to find out the time of execution of a program so that I can find the efficient programs
@Indently
@Indently Жыл бұрын
Maybe this will help: kzfaq.info/get/bejne/p758m5ikv7LUeHk.html&ab_channel=Indently
@vvv3876
@vvv3876 Жыл бұрын
Python 3.10.4 Windows 8.1 I dont have the library timeit and when try to install it through pip i get the error: No matching distribution found for timeit
@saiakhileshande8486
@saiakhileshande8486 Жыл бұрын
Hey @Indently, what if I am given a very long list and asked to check if an element exist in the list or not. In this case, which one is optimal - checking in the list or converting the list to set and checking in the set? The question is that is it better to convert list into set and check?
@dmytrk
@dmytrk Жыл бұрын
Converting a list to set takes N hashing calls, so you don't win in efficiency. Either store your data in a set, if order doesn't matter, or just use a list
@computerfun6724
@computerfun6724 Жыл бұрын
Love from India bro 🇮🇳
@chx75
@chx75 Жыл бұрын
How about dicts?
@mysticplayz5649
@mysticplayz5649 Жыл бұрын
Doesn't that mean if sets are unordered the number we are finding will be at 2nd index or in the middle where in list we know that what we are finding is always at the last index so set find it quicker than list ?😮
@Efebur
@Efebur Жыл бұрын
No. If it were in the middle (on average) it would be half as slow as the list, instead it is really fast, always. It's because sets are using a completely different algorithm than lists (hashing).
@kbcat
@kbcat 2 ай бұрын
what ide is this
@KeiraraVT
@KeiraraVT Жыл бұрын
Looking at this makes me wonder if I can optimize my function by making a set instead of a dict for speed. Or if a dict is faster than a. List but slower than a set.
@appelnonsurtaxe
@appelnonsurtaxe Жыл бұрын
From an implementation perspective, a set is often implemented as a simple hashmap (dict) where the keys are the set element and the values are all void (empty). So the performance of indexing into a dictionary and checking if an element is in a set is pretty much the same under the hood
@link_safe
@link_safe Жыл бұрын
is a frozenset() faster than a set()?
@laz272727
@laz272727 Жыл бұрын
They take slightly less ram, since they don't need to be mutable.
@link_safe
@link_safe Жыл бұрын
@@laz272727 Thanks 🧑‍💻
@rishiraj2548
@rishiraj2548 Жыл бұрын
👍
@ThankYouESM
@ThankYouESM Жыл бұрын
Is there a way to generate pixel-by-pixel smooth, colorful images incredibly fast with set()?
@waylonbarrett3456
@waylonbarrett3456 Жыл бұрын
You could have a set for every integer value of a color channel. So, one for 0 red, one for 1 red, one for 2 red, etc. And the same for green and blue channels. Then, in each set, store the pixel indices for that color value.
Learn TOML in 10 Minutes (Tutorial)
10:51
Indently
Рет қаралды 24 М.
5 Good Python Habits
17:35
Indently
Рет қаралды 494 М.
Ouch.. 🤕
00:30
Celine & Michiel
Рет қаралды 43 МЛН
Smart Sigma Kid #funny #sigma #comedy
00:40
CRAZY GREAPA
Рет қаралды 38 МЛН
Magic trick 🪄😁
00:13
Andrey Grechka
Рет қаралды 39 МЛН
Gli occhiali da sole non mi hanno coperto! 😎
00:13
Senza Limiti
Рет қаралды 13 МЛН
Compiled Python is FAST
12:57
Doug Mercer
Рет қаралды 106 М.
The Fastest Way to Loop in Python - An Unfortunate Truth
8:06
mCoding
Рет қаралды 1,4 МЛН
6 CLEAN Tips To IMPROVE Your Python Functions
9:55
Indently
Рет қаралды 76 М.
1,000,000 backtest simulations in 20 seconds with vectorbt
3:48
10 Important Python Concepts In 20 Minutes
18:49
Indently
Рет қаралды 36 М.
Modern Graphical User Interfaces in Python
11:12
NeuralNine
Рет қаралды 1,5 МЛН
Python dataclasses will save you HOURS, also featuring attrs
8:50
15 Python Libraries You Should Know About
14:54
ArjanCodes
Рет қаралды 382 М.
5 Useful Python Decorators (ft. Carberra)
14:34
Indently
Рет қаралды 96 М.
threading vs multiprocessing in python
22:31
Dave's Space
Рет қаралды 574 М.
Ouch.. 🤕
00:30
Celine & Michiel
Рет қаралды 43 МЛН