No video
When should I use a "groupby" in pandas?
Рет қаралды 243,904
Күн бұрынThe pandas "groupby" method allows you to split a DataFrame into groups, apply a function to each group independently, and then combine the results back together. This is called the "split-apply-combine" pattern, and is a powerful tool for analyzing data across different categories. In this video, I'll explain when you should use a groupby and then demonstrate its flexibility using four different examples.
SUBSCRIBE to learn data science with Python:
www.youtube.co...
JOIN the "Data School Insiders" community and receive exclusive rewards:
/ dataschool
== RESOURCES ==
GitHub repository for the series: github.com/jus...
"groupby" documentation: pandas.pydata.o...
"agg" documentation: pandas.pydata.o...
"plot" documentation: pandas.pydata.o...
== LET'S CONNECT! ==
Newsletter: www.dataschool...
Twitter: / justmarkham
Facebook: / datascienceschool
LinkedIn: / justmarkham
@coolfooly 6 жыл бұрын
Seriously this was an amazing explanation. I cant like it enough.
@dataschool 5 жыл бұрын
Thanks! :)
@MagnusAnand 3 жыл бұрын
It is indeed
@soumenhalder4831 4 жыл бұрын
Me a Ph.D. student in experimental high energy physics getting spirit from all these tricks, and avoiding some boring tasks. Thank you very much, sir.
@dataschool 4 жыл бұрын
You're very welcome!
@krishnendusaha5940 6 жыл бұрын
Great video. It's better than many of the so- called great courses on the internet! Cheers!
@dataschool 6 жыл бұрын
Awesome, thank you! :)
@mitchellyula4447 Ай бұрын
Thank you so much! I never used the groupby function until now and this helped me complete my pandas assignment! very clear explanation and your videos have helped me a ton!
@dataschool 28 күн бұрын
Glad it helped! 🙌
@mkaberli 5 жыл бұрын
You’re my go to person for informative videos on pandas, Numpy, and Matplotlib.
@dataschool 5 жыл бұрын
Thanks!
@marcinzaremba1811 3 жыл бұрын
That is exactly what I was looking for and couldn't find at the same time. Thank you!!!
@dataschool 3 жыл бұрын
You are welcome!
@shahzan525 4 жыл бұрын
Your content is amazing , I visited some others channel , they are not taught like you, I see this video in suggestions & I learn perfectly how to use groupsby and where to use ... Thanks man........
@muyideenjimoh6672 4 жыл бұрын
First video I clicked on and it answered all my confusion. Many thanks.
@dataschool 4 жыл бұрын
Great to hear!
@robertobarriosduran2622 4 жыл бұрын
Even for my bad english level, this explanation was so clear. Big Thanks!
@dataschool 4 жыл бұрын
Great to hear!
@diegodesouza302 3 жыл бұрын
ok...i just left my afforded course from edx to watch only Data School videos...no regreted!!!! The way this guy explain should be a pattern to all teaches in the world! Love from Brazil!!!
@dataschool 3 жыл бұрын
Thank you so much!
@arashchitgar7445 5 жыл бұрын
WOW... I wish all the tutorials were that clear. Thanks man!
@dataschool 5 жыл бұрын
You are too kind! 😄
@uguree 3 жыл бұрын
Corey and Kevin you are brilliant. Thanks Corey as he advertised your channel, so pleased. It's like here go how to make a you tube content to teach
@fezulhasan3647 2 жыл бұрын
Man you are amazing there is no one who can explain better than you. Love from India buddy.
@dataschool 2 жыл бұрын
Thank you so much!
@BigAC7 3 жыл бұрын
This is great! I'm taking an absolutely hateful pandas course that's basically a needle-in-a-haystack scavenger hunt. Faced with a .groupby() problem where the course materials in no way logically lead to the answer, I wound up here. The problem doesn't precisely conform to your tutorial, but just three of your lines were enough to convey the basic mechanism of action that led to an answer. Thanks!
@dataschool 3 жыл бұрын
Awesome to hear!
@uguree 3 жыл бұрын
I will start your Sci-kit series too as I enjoyed so much watching all these pandas series as very clear.
@dipakshah1008 3 жыл бұрын
You explain pandas very well! Pls keep posting ...would love to see something on how to apply custom functions to tables (avoiding iteration use)
@ChartExplorers 3 жыл бұрын
Hi, it sounds like what you are looking for is vectorization. I would recommend reading up on numpy where() and numpy select(). I looked through Kevin's (data school's) videos and didn't see anything on this topic. He is amazing at explaining things, in the mean time you many be interested in this video where I talk about vectorizing a little bit at the end of the video here: kzfaq.info/get/bejne/ea1jeLlouqipfHU.html
@ruslanzakharov4218 5 жыл бұрын
It was really useful for me. I don't know english well, but I understood almost all words in this video
@dataschool 5 жыл бұрын
That's awesome to hear! 🙌
@robhubert8350 6 жыл бұрын
Nice job with the explanation. I combed through dozens of videos and articles online specific to leveraging groupby and your's by far is the best.
@dataschool 6 жыл бұрын
Thanks very much for your kind words!
@weskitlasten417 6 жыл бұрын
Thanks! I realize this is years old, but worth a shot... One thing I would love to see clearly explained is how to change values in one column based on a condition on values in another column. For example, if drinks.spirits > 100 and drinks.total_liters_of pure_alcohol >10, change drinks.continent (or other column) to "lush". As an addition, if values in one column (say drinks.name) contain the word "dirty" change drinks.briny to True. Etc
@dataschool 6 жыл бұрын
For your first example, the code would look something like this: drinks.loc[(drinks.spirits > 100) & (drinks.total_liters_of pure_alcohol >10), 'continent'] = 'lush' loc is super useful! Here's more information on loc: kzfaq.info/get/bejne/rtyggaRox6mck6c.html
@asadtanvir4065 6 жыл бұрын
Thank you Man! You are doing a great job here by showing Pandas tutorials in such easy way. Pray that you can carry out this benevolent work more
@dataschool 6 жыл бұрын
Thanks very much for your kind words!
@TexaSol2020 3 жыл бұрын
excellent description. Much better than many on LinkedIn.
@dataschool 3 жыл бұрын
Thanks!
@daudujonnie Жыл бұрын
You are simply fabulous. I think you deserve a grand master award.
@dataschool Жыл бұрын
Wow, thank you!
@sulaimankhan8033 4 жыл бұрын
Your'e rate of speech is inline with your coding..., I appreciate it.. thanks
@dataschool 4 жыл бұрын
You're welcome!
@scorpsallday33 7 жыл бұрын
Are you fucking kidding me.... you sir are the man. These tutorials have been blessing me. Where have you been all my life 😂
@dataschool 7 жыл бұрын
Awesome! Thanks so much for your very kind comments :)
@rafaelzanfolin7745 2 жыл бұрын
Thank you sir! Eu really saved my life, hugs from Brazil!!
@dataschool 2 жыл бұрын
Glad it helped!
@janaal-sokhon2462 4 жыл бұрын
Thank you so much for all of your videos that explain pandas. You seriously explain better than my teachers. THANK YOU
@nitishkumar-bk8kd 4 жыл бұрын
the best way of explaining pandas ,big fan for your teaching
@nelsonkayode106 Жыл бұрын
This is the best groupby video I've seen. Thank you for your commitment. You just gained a subscriber.
@dataschool Жыл бұрын
Thank you so much!
@cyruswpl 7 жыл бұрын
This is really helpful and easy to understand! Looking forward to your future great works, keep it up.
@dataschool 7 жыл бұрын
Great to hear!
@fadilyassin4597 Жыл бұрын
the way you teach is unique and good
@dataschool Жыл бұрын
Thank you!
@julieye2260 4 жыл бұрын
Thank you so much! This video is very helpful and great quality! I appreciate your speaking speed; it is definitely friendly for foreigners to follow.
@redcat7467 4 жыл бұрын
Man, you got one hell of a channel. Thanks!
@dataschool 3 жыл бұрын
I appreciate that!
@josephvargas6756 6 жыл бұрын
You sir, are a Gentleman and a Scholar. Thanks for your videos.
@dataschool 6 жыл бұрын
Thanks!
@sarandam5125 5 жыл бұрын
Thank you very much for an amazing explanation. I doubted .groupby method for a long time and finally found this video, This video totally clear my doubt out :D
@dataschool 5 жыл бұрын
You are very welcome! So glad it was helpful to you :) You might be interested in my latest video, which has some usage examples of groupby: kzfaq.info/get/bejne/mranf8970LmsfKM.html
@brandonwarfield5611 9 ай бұрын
Wow! You explained that so well that it sticks. Thank you!!
@dataschool 7 ай бұрын
You're very welcome!
@Kristina_Tsoy 2 жыл бұрын
Amazing video as always, Kevin! Thank you for great tip at 03:48
@dataschool 2 жыл бұрын
My pleasure!
@Jettiesburg 7 жыл бұрын
Thanks, top video! Only suggestion would be to expand on this with some strategies for creating a new DF with the grouped/agg values. Thanks again
@dataschool 7 жыл бұрын
Thanks for your suggestion!
@fawadkhan8905 5 жыл бұрын
How nice of you to come up with such great videos. Thanks
@dataschool 5 жыл бұрын
Thanks for your kind words!
@davebeckham5429 4 жыл бұрын
Excellent tutorials as always Kevin. Thanks for sharing.
@dataschool 4 жыл бұрын
Thank you!
@karmakast806 5 жыл бұрын
oh my go.. such clarity in explanation.
@dataschool 5 жыл бұрын
Thanks! :)
@ramakanthrayanchi8888 8 жыл бұрын
Awesome. The tip at last would be very much helpful if we want to apply a function on all the numeric columns at once for each group independently.
@johnnyrotten43 4 жыл бұрын
You made that so easy. Thank you.
@dataschool 4 жыл бұрын
You're very welcome!
@astronauta8132 6 жыл бұрын
man, your explanations are so good. keep it straight, keep it simple, keep it upPPP!!!!
@dataschool 6 жыл бұрын
Thanks very much for your kind words! :)
@panchopaulo111 4 жыл бұрын
This video was just amazing!! You just got a new subscriber.
@dataschool 4 жыл бұрын
Thank you!
@user-mx8pn1ky5s 5 жыл бұрын
I have already written before and want to say it again. Great lessons, thanks teacher!
@dataschool 5 жыл бұрын
You are so very welcome - thanks for your kind comment! 🙌
@terrancedejesus625 7 жыл бұрын
Fantastic video. Using query() is another great way to filter through data and add multiple filters. I have often found it to be less typing and alot more simple. Example: drinks.query("continent == 'Europe'").beer_servings.mean(). Thanks for the video!!
@dataschool 7 жыл бұрын
Thanks for your kind words! I personally don't use "query" because the pandas documentation lists it as "experimental", which tells me that they might change the API or eventually remove it: pandas.pydata.org/pandas-docs/stable/indexing.html#the-query-method-experimental However, functions can always change regardless of whether they are listed as "experimental", so I guess there's no harm in using it! :)
@abhishek-hb1vg 5 жыл бұрын
wooooooowowowowowoow this is what i felt when u used the matplotlib function. U r amazing.
@dataschool 5 жыл бұрын
Thanks!
@praveenudayakumar5128 7 жыл бұрын
Good explanation, audio clarity and good English too thank u bro.
@dataschool 7 жыл бұрын
You're welcome!
@kleczekr 8 жыл бұрын
You can also try to run the code: import seaborn as sb before running the code plotting the graph. The difference is really amazing (and you can see the length of bars for Europe and North America, which in the standard graph are behind the legend).
@dataschool 8 жыл бұрын
Great tip! Yes, the default plot aesthetics will change simply by importing Seaborn.
@freeandeasy9795 3 жыл бұрын
Nailed it! Exactly what I was looking for. Thanks
@dataschool 3 жыл бұрын
You're welcome!
@majidm4215 4 жыл бұрын
you are awesome, I have never watched that many videos in one day Thank you so much
@iampujan 6 жыл бұрын
Easily understandable. Thanks for the video and keep making such videos.
@dataschool 6 жыл бұрын
You're very welcome!
@kwphysics 2 жыл бұрын
This was a useful video and I appreciate your explanation!
@dataschool 2 жыл бұрын
Glad it was helpful!
@rajpaul1501 5 жыл бұрын
You are simply awesome. Need more and more of your tutorials.
@dataschool 5 жыл бұрын
Thank you!
@scottlucas3710 7 жыл бұрын
Excellent job of explaining pretty complicated subject matter !
@dataschool 7 жыл бұрын
Thanks so much!
@Mark-sf4kh 6 жыл бұрын
Data School you guys have a slack channel or blog where one can post questions?
@parthraghuwanshi2929 2 жыл бұрын
Man you really bring great content
@dataschool Жыл бұрын
Thank you!
@hariaiyar1119 4 жыл бұрын
Amazing explanation bro.
@dataschool 4 жыл бұрын
Thanks!
@andreazecchi812 3 жыл бұрын
that's a fantastic explaination! Thanks so much!
@dataschool 3 жыл бұрын
You're very welcome!
@theodoretourneux5662 2 жыл бұрын
fantastic! exactly what I needed clearly explained and organized. thank you!
@dataschool 2 жыл бұрын
You're welcome!
@tarekkhalefa1836 4 жыл бұрын
thanks for your simple explanation
@dataschool 3 жыл бұрын
You're welcome!
@rin_645 3 жыл бұрын
4:49 and dot aggggggggg~ hahaha 😂 But seriously tho, what an absurdly good explanation of groupby function. I wish you could be a professor in my university🤪
@dataschool 3 жыл бұрын
Thank you! 😄
@robindong3802 6 жыл бұрын
Very clear and simple. great demo.
@dataschool 6 жыл бұрын
Thanks!
@maggietang1369 4 жыл бұрын
Great videos,helped a lot.Thanks!
@dataschool 4 жыл бұрын
You're welcome!
@rohitjoshi1777 3 жыл бұрын
Clear explanation please make video on enumerate function in pandas
@dataschool 3 жыл бұрын
Thanks for your suggestion!
@hakunamatata-qu7ft 4 жыл бұрын
please do a video on all possible string operations possible in data preprocessing
@dataschool 4 жыл бұрын
Does this help? kzfaq.info/get/bejne/mNWWlKZhrsnPeKM.html
@339059331 3 жыл бұрын
Awesome as usual! thanks very much!
@dataschool 3 жыл бұрын
Thank you!
@honprarules 4 жыл бұрын
Earned a sub in 3 mins. Amazing work!
@dataschool 4 жыл бұрын
Thank you!
@raghavendra4955 4 жыл бұрын
Y there are less views🙃🙃.. Its an amazing explanation..
@dataschool 3 жыл бұрын
Thank you!
@jakeb2553 4 жыл бұрын
What do you recommend if I want to group by conditionals? Let's say I have a column "Blood Pressure" and I want to group by values 120.
@edilsonsousa3743 Жыл бұрын
Thank you for making this video, really helped!
@dataschool Жыл бұрын
Glad it helped!
@khalidhassani6173 4 жыл бұрын
agg is the best. thanks a lot
@dataschool 4 жыл бұрын
You're welcome!
@tayebzlezel9222 3 жыл бұрын
u r a life saver my friend
@magelauditore333 4 жыл бұрын
U r awesome man. Pls Pls make numpy series for data science, stats
@KimmoHintikka 7 жыл бұрын
On the agg method. Is there a way to simply return which countries the min and max belong to? I know that now that, min 0 for specific continent I could just filter dataframe based on continent and beer_servings value of 0, but could I somehow do that in the initial agg call to also print out the biggest and smallest consumers?
@dataschool 7 жыл бұрын
Great question! If there's an easy way to do this, I can't think of it at the moment. Let me know if you come up with something!
@user-un1rg9db6k 7 жыл бұрын
Any update on that? Thanks!
@adelsaleh5769 7 жыл бұрын
import pandas as pd drinks =pd.read_csv('data/drinks.csv') print('*****************************') max_bear= drinks['beer_servings'].max() min_bear= drinks['beer_servings'].min() print ('max_bear:',max_bear,',min_bear',min_bear) print('*****************************') print(drinks[drinks['beer_servings']==max_bear]) print('+++++++++++++++++++++++++') print(drinks[drinks['beer_servings']==min_bear])
@BNOOOOGERS 6 жыл бұрын
You can get the corresponding index values of a maximum with groupby().idxmax(). But if the country is in a column rather than the index, I think this is a 2-step process where you then index the country column by the results of the groupby.idxmax.
@derdere7803 6 жыл бұрын
df.loc[df.reset_index().groupby('continent').beer_servings.idxmax()] df.loc[df.reset_index().groupby('continent').beer_servings.idxmin()]
@Shiva-zy7jq 5 жыл бұрын
Bonus tips for comment reaaders. nunique() aggregator is used to calculate the number of unique values in a series
@dataschool 5 жыл бұрын
Thanks for sharing!
@Stella-rs5qf 4 жыл бұрын
Very great explanation
@Dmitrycheburirtrie 3 жыл бұрын
Best explanation !
@dataschool 3 жыл бұрын
Thank you!
@SarmitaMajumdar 3 жыл бұрын
Amazing handy guide!!! I must thank you for the same! Wish to receive more in near future!!!
@dataschool 3 жыл бұрын
Glad it was helpful!
@terngun Жыл бұрын
This was helpful! Thanks for sharing.
@dataschool Жыл бұрын
Great to hear!
@ManojSharmaX 4 жыл бұрын
2:58 filter and group bu 5:35 agg
@RichardGreco 4 жыл бұрын
Excellent tutorial! If possible maybe another video to refresh and/or to address Pandas 1.0.x?
@dataschool 4 жыл бұрын
Thanks for your suggestion!
@sajeeb4 5 жыл бұрын
Group By multiple column data_frame.groupby(by=['continent','country']).beer_servings.mean()
@dataschool 5 жыл бұрын
Thanks for sharing!
@MrCraptakular 7 жыл бұрын
Hello. Firstly, thank you for this magnificent video series. I really mean that! I have a question, continuing with your beer example. If you didn't have a continent column in the source data how would you accomplish the same output?
@dataschool 7 жыл бұрын
Glad you like the series! Regarding your question, the continent column has to be part of the DataFrame, otherwise you can't group by it. Hope that answers your question!
@carolinegh601 5 жыл бұрын
I wanna give you a hug! Thank you so much!
@dataschool 5 жыл бұрын
You are so kind! 😊
@RayedWahed 8 жыл бұрын
Great Series! Can you please upload a couple of detailed videos on plotting graphs and graphically visualizing data to spot noise and to determine which features may be less important? Thank you
@dataschool 8 жыл бұрын
Thanks for the suggestion! I will eventually cover visualization in this series. Regarding feature selection, I'm unlikely to cover it in this video series, though I will cover it in an upcoming course. Make sure you are subscribed to my newsletter to hear about upcoming courses: www.dataschool.io/subscribe/
@harijayaram 7 жыл бұрын
very clear example of groupby ...thanks
@dataschool 7 жыл бұрын
You're very welcome!
@nevilparekh6400 5 жыл бұрын
Great Tutorials. :) Thanks for such a nice series. Could you please come up with videos on Heat Map & Correlation Table? thanks in advance.
@dataschool 5 жыл бұрын
Thanks for your suggestion!
@mustafaadelamer2789 5 жыл бұрын
Thank you so much You really made it very simple
@dataschool 5 жыл бұрын
That's great to hear! :)
@wardomedet 7 жыл бұрын
thank you for this tutorial series you are really a good teacher it helped me a lot I have a question would you consider making a tutorial video for pandas memory handling methods(I am not sure if the terminology is right ) how can we remove objects from memory when we are done or is there a way to control these processes thank you!!
@dataschool 7 жыл бұрын
Thanks for the suggestion! I'll consider it for the future!
@brianstampe7056 6 жыл бұрын
I second this request!
@rajeshd3574 5 жыл бұрын
I second this request even
@syntel13764 5 жыл бұрын
i am a great fan of you .. you rocks !!
@dataschool 5 жыл бұрын
Thank you! 🙌
@shiyunjiang8085 5 жыл бұрын
Can I ask how to calculate the group by aggregation and only count unique number?Thanks ( BTW YOU ARE AMAZING)
@dataschool 5 жыл бұрын
Thanks for your kind words! As for your question, could you clarify? I don't quite understand. Thanks!
@luisfernandoperezarmas7198 5 жыл бұрын
Thanks for this video bro, great explanations.
@dataschool 5 жыл бұрын
Thanks!
@maggietang1369 4 жыл бұрын
Great video, helps me a lot ,thanks!
@dataschool 4 жыл бұрын
Great!
@fet1612 5 жыл бұрын
Q&A Series: Video #14 _04(TC01:14) ========================================== drinks.beer_servings.mean() What if you wanted to know beer_servings across continents? You wanted to know, how it varied from Europe to Asia to Africa. So this is where a GROUPBY (groupby) is USEFUL. (TC01:34)
@johnrogers3315 4 жыл бұрын
succinct and effective, many thanks
@dataschool 4 жыл бұрын
Thanks!
@russnagel1 3 жыл бұрын
Like many others have said. More great content, thank you!
@zohaibhasan5061 7 ай бұрын
I am a beginner to pandas. Amazing videos that were suggested by a friend. I have a doubt where I am stuck. For me in Python 3 drinks.groupby('continent').mean() does not work. min, max and count works but this does not (not without specifying 1 or more column names)
@dataschool 7 ай бұрын
Excellent question! In the current version of pandas, if a DataFrame contains non-numeric data and you want to calculate the mean of all numeric columns after a groupby, you have to include the argument numeric_only=True. Thus: drinks.groupby('continent').mean(numeric_only=True). Hope that helps!
@zohaibhasan5061 7 ай бұрын
@@dataschool Yes it sure helps!!! Thanks for pointing out this argument
@brianbii1104 3 жыл бұрын
I really enjoy ur video. i was wondering if you had any videos on how to do a report and if not then can u point to some resources that may have a way to explain it to me.
@ranger52289 Жыл бұрын
Great explanation! How could you show the information for the country in each continent with, say, the highest beer servings?
@elilavi7514 8 жыл бұрын
Thanks a lot for the video . I like you series on pandas , every lesson I learn new things ! Sometimes with a large dataset , the groupby( ) could be a little bit heavy for certain server and slow . Does the groupby have any alternatives for a big datasets ?
@dataschool 8 жыл бұрын
+Eli Lavi Thanks for all the great questions... keep them coming! :) I'm not aware of any alternatives, but I will keep your question in mind in case I come across a solution!
@mohamedalibouchhioua9734 5 жыл бұрын
Very helpful video thank you and good luck.
@dataschool 5 жыл бұрын
Thanks! :)
Data Nash
Рет қаралды 67 М.