Exploratory Data Analysis in Pandas | Python Pandas Tutorials

No video

Exploratory Data Analysis in Pandas | Python Pandas Tutorials

Рет қаралды 125,986

Күн бұрын

Take my Full Python Course Here: www.analystbui...
In this series we will be walking through everything you need to know to get started in Pandas! In this video, we learn about Exploratory Data Analysis in Pandas.
Dataset in GitHub:
github.com/Ale...
Code in GitHub: github.com/Ale...
Favorite Pandas Course:
Data Analysis with Pandas and Python - bit.ly/3KHMLlu
____________________________________________
SUBSCRIBE!
Do you want to become a Data Analyst? That's what this channel is all about! My goal is to help you learn everything you need in order to start your career or even switch your career into Data Analytics. Be sure to subscribe to not miss out on any content!
____________________________________________
RESOURCES:
Coursera Courses:
📖Google Data Analyst Certification: coursera.pxf.i...
📖Data Analysis with Python - coursera.pxf.i...
📖IBM Data Analysis Specialization - coursera.pxf.i...
📖Tableau Data Visualization - coursera.pxf.i...
Udemy Courses:
📖Python for Data Analysis and Visualization- bit.ly/3hhX4LX
📖Statistics for Data Science - bit.ly/37jqDbq
📖SQL for Data Analysts (SSMS) - bit.ly/3fkqEij
📖Tableau A-Z - bit.ly/385lYvN
Please note I may earn a small commission for any purchase through these links - Thanks for supporting the channel!
____________________________________________
BECOME A MEMBER -
Want to support the channel? Consider becoming a member! I do Monthly Livestreams and you get some awesome Emoji's to use in chat and comments!
/ @alextheanalyst
____________________________________________
Websites:
💻Website: AlexTheAnalyst.com
💾GitHub: github.com/Ale...
📱Instagram: @Alex_The_Analyst
____________________________________________
0:00 Intro
1:51 First Look at Data
3:45 Info()
4:40 Describe()
5:47 Counting all Null Values
7:09 Count of Unique Values
8:15 Sorting on Values
10:40 Correlation between Columns
11:53 Heatmap using Seaborn
14:43 Grouping Data
25:02 Visualizing Grouped Data
26:17 Boxplots for Outliers
29:07 Data Types of Columns
30:41 Outro
All opinions or statements in this video are my own and do not reflect the opinion of the company I work for or have ever worked for

Пікірлер: 181

@santiagofajardo4949 Жыл бұрын

Hello, at minute 24:24, I managed to reverse the range of column names using [5:13][::-1]. The expression [::-1] is used to reverse ranges and it is very useful: df2 = df.groupby('Continent')[df.columns[5:13][::-1]].mean(numeric_only=True).sort_values(by='2022 Population', ascending=False) df2 Thank you very much, Mr. Alex, for these tutorials.

@user-zq6cp7lh3s 7 ай бұрын

Thank You!

@renanz21 7 ай бұрын

Alternatively, start counting columns backwards, df2 = df.groupby("Continent")[df.columns[-5:-13:-1]].mean().sort_values(by='2022 Population', ascending=False) df2

@satrapech6107 11 ай бұрын

the correction of df.corr() is: numeric_columns = df.select_dtypes(include=[np.number]) correlation_matrix = numeric_columns.corr correlation_matrix()

@pradiptanugraha6841 10 ай бұрын

Thanks it works. Why df.corr() not working on me ?

@rajkumarjadi7061 9 ай бұрын

thanks man.

@francescab1413 9 ай бұрын

df.corr(numeric_only = True) worked for me

@arrofifahmi7708 7 ай бұрын

@@francescab1413 me too mate! Thanks a lot!

@SDMNKhan 6 ай бұрын

name 'np' not defined?

@pbp7 Жыл бұрын

Man, “Oceania” was so funny 😂, tks for the class!

@JW-pu1uk Жыл бұрын

This is absolutely top tier content. I can't stress this enough to people new, or going into the DA/DS field: you WILL be exploring and cleaning data sets much more than you will be visualizing and building models. Thanks for this, Alex!

@rafaelmarques5623 8 ай бұрын

Oceania is one of the 7 Continents (North America, South America, Europe, Asia, Africa, Oceania, Antartica). It's basically Australia and the countries (islands) around it. Hope that helps!

@AlastorGarcia Жыл бұрын

Thanks Alex! Right now i'm applying to my first DA Job and you have no idea how useful your videos have been for me!!

@ermano5586 Жыл бұрын

Hey? How is it going? Did you succed in applying for the job you want?

@kartikgupta370 8 ай бұрын

We can also write this to save time writing all the column names in the list "df2 = df.groupby('Continent')[df.columns[12:4:-1]].mean(numeric_only=True).sort_values(by='2022 Population', ascending=False) "

@frenamakenson9844 6 ай бұрын

Hello, 100000000 thanks for sharing For the Corealtion part at 11mn df.corr(numeric_only=True) # pass numeric only param to not having error

@matthewchristian9969 15 күн бұрын

Thank you!

@toygar8699 8 ай бұрын

For those get error in heatmap: import matplotlib.pyplot as plt numeric_columns = df.select_dtypes(include=['float']) sns.heatmap(numeric_columns.corr(), annot=True) plt.rcParams['figure.figsize'] = (20, 7) plt.show()

@asmitaupadhyay4656 5 ай бұрын

thank you

@nointernetnarwhal7615 4 ай бұрын

THANK YOU!!!!!! I almost quit for good.

@nassrmohamed278 4 ай бұрын

i had that error in corr : " could not convert string to float: 'AFG'" do you know how to solve this

@user-vy8kk9ob3s 3 ай бұрын

thanks a lot toygar

@yanpaucon1043 3 ай бұрын

@@nassrmohamed278 df.corr(numeric_only=True)

@staquatica1607 10 ай бұрын

I got some error's (using pycharm) that I solved by using "mumeric_only=True". For instance: df.corr(numeric_only=True) and df.groupby("Continent").mean(numeric_only=True)

@mohammedshadaabkhan3228 9 ай бұрын

Hey use this code instead numeric_df = df.select_dtypes(include='number') # Select only numeric columns plt.figure(figsize=(20, 7)) # Set the figure size sns.heatmap(numeric_df.corr(), annot=True) # Create the heatmap with annotations plt.show()

@DevanshAsawa 7 ай бұрын

helped a ton thanks

@haley2486 7 ай бұрын

Thanks for posting! I had to do SHIFT+TAB on the corr() function to find out how to get only numeric values.

@nassrmohamed278 4 ай бұрын

thaaaaaaaaaaaaaaank youuuuuuuuuuuuuuuuu

@shankarmidatala2049 28 күн бұрын

Namaste! I found your tutorials "Simple, Easy to follow, and To the point". Thanks.

@sj1795 7 ай бұрын

EXCELLENT SUPERB video!! I can't believe it--I'm 6/7 videos away from the end of your FANTASTIC bootcamp series! Wahoo! I learned a lot in this video. :) As for "ending on a low note", hardly Alex lol All your content is uplifting and rewarding! As always, THANK YOU!

@pradiptisimkhada292 Жыл бұрын

I just finished all the videos in you bootcamp playlist few hours ago and I'm excited to do this again..

@DuckingDuck-th2lt 6 ай бұрын

Hello, Alex! Once again, thanks a lot for all your hard work! At 13:10 I got an error ValueError: 'box_aspect' and 'fig_aspect' must be positive" Solved it by putting the plt.rcParams BEFORE the sns.heatmap The other problem was that some functions didn't work until I added the parameter numeric_only = True, e.g., df.corr (numeric_only=True) or .mean(numeric_only = True) Hope, it can help someone!

@yanpaucon1043 3 ай бұрын

Thank you, You are the Best!

@MaximKazartsev Жыл бұрын

Alex, thank you for this great video and everything you do! In order to avoid manual ordering of the population years, there is a way to use df.columns method, by adding reversed. The whole construction looks like df2 = df.groupby('Continent')[list(reversed(df.columns[5:13]))].mean().sort_values(by='2022 Population', ascending=False) And it works )

@languagewanderlust 5 ай бұрын

thank you!

@DEDE-ix9lg Жыл бұрын

I always enjoy a video from Alex. Making one of the best videos , while some other channels just can be a real headache

@abhishekchaudhary7913 7 ай бұрын

df4=df3.sort_index(ascending=True) df4 at 26:11 as alex is sorting manually you sort the year directly by this command

@user-yp1ej5ou6b 8 ай бұрын

Hey, just a quick note here, when we're plotting the populations, it's only related to the numeric values compared to the highest populations, in fact (for example) Oceania's population increased in around 2.5 times Anyway, thanks for the content, it's amazing

@kogureyoeh Жыл бұрын

at 24:00 you can just simply add ".sort_index()" on the "df3 = df2.transpose()", so that we don't have to manually rearrange the columns. df3 = df2.transpose().sort_index() worked on my end, hope on your end too.

@abisolalumous5505 7 ай бұрын

thank you

@quotesdiary310 Жыл бұрын

Hi Alex Thank you so much for your support for freshers in the field of data analytics.

@keluargaindo-timordiuk Жыл бұрын

For the grouping data I do df2=df.drop(columns=['CCA3','Country','Capital']) df3=df2.groupby('Continent').mean(numeric_only=True).sort_values(by="2022 Population",ascending=False) df3 to get to the same output as seen in the video

@danielmariobuchberger 10 ай бұрын

Me too, this should be explained, because Strings can not get easy a mean...to long is most the problem!

@bolajiawofuwa8116 8 ай бұрын

THANK YOU!!!!!!

@Inc0gnit030 Жыл бұрын

I really enjoyed this introduction to Pandas! Keep up the good work!

@tranguyen4462 4 ай бұрын

omg I laughed out loud at the "Oceania" part ;)))) Alex is so funny and brutally honest about things he didn't know ;)))

@ngwamalfred8151 11 ай бұрын

Where would l have been without this video .

@kevindeschepper8140 2 ай бұрын

To exclude rank from being display in the numerice data: columns_to_include = df.select_dtypes(include=['number']).columns.difference(['Rank'])

@LaMeeLifestyle Жыл бұрын

Thanks for all you do. I’m loving the bootcamp. Just finished excel project. However, please can you make a video on story telling?

@nadarioferguson6276 4 ай бұрын

Thank you so much for this. I really enjoyed it and learned a lot of what I had forgotten a few years ago.

@SoggyBagelz Жыл бұрын

Lets goo!

@aishwaryapattnaik3082 Жыл бұрын

Thanks a lot for this clear cut explanation. Can you make something similar for NLP projects end to end ?

@r10053506 2 ай бұрын

why is my program when running corr() is not automatically detecting numbers and runs into an error

@moniquebrasilbaptista1989 10 ай бұрын

I am sure I am going to use some of these tips. Thank you!😍❤

@minasghazaryan9344 Жыл бұрын

Hi, Alex. First of all thanks for a great video and explanations in it. If you could help out with the issue I get running your exact code I would be more than grateful. Running the df.corr() line gives me the following error: ValueError: could not convert string to float: 'AFG' . Same comes for the heatmap,etc. What could it be here? Thanks a lot in advance.

@ReneePieschke Жыл бұрын

Getting the same errors.

@11zaad Жыл бұрын

try this ==> df.corr(numeric_only=True)

@dustin3320 Жыл бұрын

Best to use df.corr(numeric_only=True) to get around this

@Batira583 9 ай бұрын

you saved my life thanks so much @@dustin3320

@fede77 8 ай бұрын

df.corr(numeric_only = True)

@innocentnduaguba 8 ай бұрын

Thank you so much Alex, truly great content you put out there. I have a question please; when I run df.groupby('Continent').mean() and df.corr() I get errors, please what could be the cause and what can I do to remedy it.

@sabithsaqlain1367 7 ай бұрын

use df.corr(numeric_only = True)

@sj1795 7 ай бұрын

@@sabithsaqlain1367 THANK YOU for this!! This was driving me a little nutty. Really appreciate you sharing this. :)

@SDMNKhan 6 ай бұрын

I could not fix the mean() issue.

@chriscurtis95 3 ай бұрын

df.groupby('Continent').mean(numeric_only=True)

@user-ml2qj4fm9x Ай бұрын

@@chriscurtis95 🙏 Thank You!

@Zenitsu-mq7fq 5 ай бұрын

24:50 df2 = df.groupby('Continent').mean(numeric_only=True).iloc[:, -5:-13:-1].sort_values(by = '1970 Population', ascending = False) df2 = df2.transpose() df2.plot() This way we don't use the copypasting and changing columns, just use reversed indexes)

@jeffrey6124 16 күн бұрын

Hope you also make a Pyspark series 🤓

@sarayusemesta6132 2 ай бұрын

26:00 you can just add this to inverted columns df2 = df.groupby('Continent')[df.columns[5:13]].mean(numeric_only=True).sort_values('2022 Population', ascending=False) df2_inverted = df2.iloc[:, ::-1] df2_inverted

@abdulsami6117 Жыл бұрын

Love from Pakistan Alex, Really Helpful and Enjoyable. I also like the OOPS sound you make 😂😂

@iqraasif3783 9 ай бұрын

Hi, can someone help. When I plot figures that have been grouped, it doesn't show the figure, just says .

@user-tm7uw4os1n 6 ай бұрын

21:09 I just figured it out. Simply add another line after the plot, like: df2.plot() plt.show()

@OkallTheAnalyst 5 ай бұрын

Incase you are running into an error at minute 11:12, add numeric_only = True to the corr. i.e df.corr(numeric_only = True).

@mananagrawal4114 4 ай бұрын

thanks man !

@Hamzahnahmad 26 күн бұрын

thank you. really helpful!

@Charlay_Charlay 7 ай бұрын

Thank you for the Pandas class!

@kevindeschepper8140 2 ай бұрын

another way to select the columns (think of a big data sets where indicing with numbers would be challeging) columns_to_include_2 = df.select_dtypes(include=['number']).filter(like='population').columns

@kevindeschepper8140 2 ай бұрын

columns_to_include_2 = df.select_dtypes(include=['number']).filter(like='Population').columns.difference(["World Population Percentage"]):P

@sivasagarchakkarai1687 Ай бұрын

If "df.corr()" doesn't work for the same data set were using in this Video. And It throughs an error : could not covert string to float: 'AFG'. Like this, Try : df.corr(numeric_only = True)

@user-hr5bm4oe8f Ай бұрын

same

@user-hr5bm4oe8f Ай бұрын

numeric_columns = df.select_dtypes(include=[np.number]) correlation_matrix = numeric_columns.corr correlation_matrix()

@neildelacruz6059 11 ай бұрын

Thank you Alex this is very helpful.

@aayushitrivedi3481 Жыл бұрын

love your videos alexx ;)

@user-re4ip5ms9w 2 ай бұрын

my heatmap is broken its not showing all the values even if I wrote the annot = True anyone have a fix? i tried almost everything when I hit shift+tab

@HarshKumar-ws3wv 5 ай бұрын

Sir, in your opinion : Jupyter vs Pycharm? Which is better for Exploratory Data Analysis ?

@user-dx2hx2rd4g Жыл бұрын

Thank you for the useful information!

@arpitmaheshwari122 7 ай бұрын

hey, can anyone tell if the correlation command is working in vs code? I'm getting a value error in this part. please share the solution if you have one thanks :)

@Shashankkundena 5 ай бұрын

Hey, just use numeric_only = True

@anuarroho2561 21 күн бұрын

mean(numeric_only=True)

@philiprhome3824 Жыл бұрын

as R user, the syntax of pandas is just weird in compare to tidyverse (dplyr and tidyr)

@adminravi 10 ай бұрын

Is it ok if I use: pd.set_option('display.float_format', '{:.2f}'.format) instead of pd.set_option('display.float_format', lambda x: '%.2f' % x)

@rohallav 10 ай бұрын

or even better you can do lambda x: f"{x:.2f}"

@vitorribeirosa Жыл бұрын

Neat... Thanks for sharing this content. Cheers

@gauravpunera3256 Жыл бұрын

Alex please make video on how to get international remote data analyst job

@haithammontaser7769 Жыл бұрын

Hello Alex. Thanks for the video and content. Is there any video for data per-processing?

@enix492 Жыл бұрын

Hello Alex. I read a few reviews on your recommended course on Udemy. People are saying that it is a bit outdated especially the last section. Do you think I should still go for it and the non updated part doesn't matter? Love your content and thanks for everything you do here.

@AlexTheAnalyst Жыл бұрын

I haven't taken it in a while - worth listening to more recent comments. Could be outdated?

@donvious 4 ай бұрын

hi, where is the link for the csv format document?

@rjk537 Жыл бұрын

I'm a law graduate without any experience or qualifications in data analysis whatsoever but i want to get into data analysis. Will i be able to get a job in this field? and if yes then what possible skills and certifications will help me to achieve the same? please give me some tips and insights it would be really helpful!

@ermano5586 Жыл бұрын

Yes, you can, from skills I would prefer mostly analytical thinking, learn probability and statistics, other high math stuff. From certification mr Alex said that Amazon and Tableau certifications, and others will help, but anyways if it's long-term learning certificate, I think it is ok to have it on CV. But the thing that highlites you it is the projects that you have done mostly for your job and I mean not only portfolio projects but another ones to show your uniqueness.

@osiomogieasekome8799 Жыл бұрын

I couldn't get seaborn to import... I tried online solutions about installation but it didn't work

@jjsansano 3 ай бұрын

This is great! Thank you!

@octaverius762 Жыл бұрын

Alex which continent do you think Australia is in 😮

@AlexTheAnalyst Жыл бұрын

@chefernandez563 Жыл бұрын

Australia is also a continent tho😂 sometimes ppl will also refere to NZ ans Aus as the "Australias" but Oceania includes the other surrounding islands

@octaverius762 Жыл бұрын

@@chefernandez563 Oceania is a continent, Australia is a country. How people often speak is not relevant

@dragoneer121 Жыл бұрын

@@octaverius762 Actually it is relevant. Though different countries do have different models and its entirely up to convention. Australia the continent is usually considered the 3 islands of mainland Australia, Tasmania and Papua New Guinea

@karanvaghela4668 11 ай бұрын

Hey alex why we should use python instead of SQl Because SQl is easy

@elfridhasman4181 Жыл бұрын

Thank you Alex💯🔥

@TheRobinCreations 10 ай бұрын

Thank you so much it was very informative.

@orlumbuseuw5646 Жыл бұрын

Was there here an adult ignorant of what Oceania is or is this some inner joke in the channel?

@octaverius762 Жыл бұрын

I can't believe this

@litoavila. Жыл бұрын

Also FYI America is just one continent, in case you doubt it

@MatthewBreithaupt 9 ай бұрын

OceanEeeA

@MatthewBreithaupt 9 ай бұрын

FYI Australia is not a *small* island. Oceania doesn't "mean" anything, it's the name of a continent containing the countries listed right in front of you since you already filtered the data 😂😂

@Marcusram Жыл бұрын

we can do df3=df3.iloc[::-1] to solve the problem with the date order

@OazadOMER 10 ай бұрын

Thank you very much Alex I'm shifting from Ph to Data Analyst with your bootcamp I had an issue with plt.show() AttributeError: module 'matplotlib' has no attribute 'show' i's deprecated and I counldn't find something sameller and also my chart not showing numbers 14:10 Best regards

@dishanbhandari 4 ай бұрын

Hi there, did u find the solution to your problem of not showing numbers? I ran into the same problem too.

@olaleyeboluwatife949 Ай бұрын

@@dishanbhandari hey mate, you found the solution?

@rnjesus9950 7 ай бұрын

This worked for me where df.corr() did not: # Select numeric columns (excluding any non-numeric columns) numeric_columns = df.select_dtypes(include=['float64', 'int64']) # Calculate the correlation matrix correlation_matrix = numeric_columns.corr() correlation_matrix

@ermano5586 Жыл бұрын

I have one problem, which is that the table does not display columns starting from "area (km^2)" when we call "df" to view the table, I mean there is no scrollbar for horizontal data, can anyone help for this, please?

@ruchirmittal9207 8 ай бұрын

Try another browser. Some browsers doesn't support that feature.

@quotesdiary310 Жыл бұрын

Thank you so much alex

@user-zk3be2uh8l 2 ай бұрын

corr_matrix = df.select_dtypes(include='number').corr() # Then proceed with creating the heatmap sns.heatmap(corr_matrix, annot=True) plt.rcParams['figure.figsize'] = (20, 7) plt.show() I have used this code for heatmap but the notebook doesn't populate the heatmap with individual correlation values rather colored tiles only. please anyone can help?

@ramanjeetkaur8969 Ай бұрын

pip install --upgrade seaborn matplotlib Update seaborn and matplotlib. It worked for me

@dishanbhandari 4 ай бұрын

My heatmap doesn’t contain the data values inside them as in 14:18 instead it just shows a heatmap with column values as in the top most band. I have written the code just as shown above df.corr(numeric_only=True) as well as that ‘annot’ but still no data values. Pls Anyone help

@NyeinHtutSwe 3 ай бұрын

i am also run into same problem :). I still cant find the solution

@jDub997D 2 ай бұрын

upgrade your seaborn package pip install seaborn --upgrade restart your kernel and rerun all the boxes

@olaleyeboluwatife949 Ай бұрын

@@jDub997D 1000 thanks bruv... bless you

@user-fx9eq7zm2v Жыл бұрын

Again, thank you were much!

@truthgaming2296 7 ай бұрын

its spells 'O-Ce-A-Nia' btw btw thank for this guidance SIr Alex :)

@meredithleonor5035 Жыл бұрын

why use anaconda instead of google collab, just curious looking forward in visual tutorial at python and statistics thanks i really need this type of tutorial i am studying cohort analysis and RFM analysis

@peaceandlove8862 9 ай бұрын

Oceania is the continent that includes Australian and New Zealand.

@youssefbekk4453 Жыл бұрын

high level , thanks

@srijanrawat4014 Жыл бұрын

i am having problem in downloading the file , can anyone help me out

@diegomartins7214 9 ай бұрын

Thank you!

@dragoneer121 Жыл бұрын

Continents are mostly a social convention. The english spekaing countries tend to use 7, while spanish speaking countries have a 6 continent model where it uses Oceania and combines North and south America. Australia is the continent but Oceania is a geopolitical convenience. If it was not included most of the pacific isalnd countries would not be associated with a continent. North and South America are another convenience and Central america is only a region by American standards. As an example of how ridiculous it is as a continent, Hawaii would be included if it was independant.

@ayoubchouket 5 ай бұрын

thank you

@taroge5464 11 ай бұрын

no explanation.................pd.set_option('display.float_format',lambda x : '%.2f' % x)

@l7932 2 ай бұрын

thanks sir

@akademy_performance_digital 7 ай бұрын

great

@chefernandez563 Жыл бұрын

Am I the only one who knew Oceania was Australia, New Zealand, Samoa and those places😂😂

@roshandhumal1193 Жыл бұрын

Sir Alex. I am Roshan Dattaram Dhumal I live in India from Mumbai. I want to start my career in data analysis but I don't know how to start and I want to know what steps you have to take to become Data analytics. I would like to request you to please explain to us and give us some steps. Please sir I will definitely do hard work.

@hammadahmed7192 Жыл бұрын

try passing numeric only argument. In recent version, default value of this argument has changed to false so it tries to correlate string values as well. df.corr(numeric_only = True)

@naagarhive6581 6 ай бұрын

OOPs

@Ben-qe8ju Жыл бұрын

O-she-ana

@alikoohi8265 Жыл бұрын

informative video thanks.Just found an easier way to reverse order of rows: df3 = df2.transpose().loc[::-1] 😉

@DatabaseAdministration 5 ай бұрын

It's funny american don't know the continent of australia.

@gogor8017 7 ай бұрын

You said 'Oceania' so many times, now it sounds like meaningless word.

@aayushitrivedi3481 Жыл бұрын

first pin me

@ermano5586 Жыл бұрын

@marypazcuessy3004 2 ай бұрын

Can anyone help me? My heatmap wont load all the numbers, just the Rank row starting at 1 I used df.corr(numeric_only = True) sns.heatmap(df.corr(numeric_only = True), annot = True) plt.show()

@RaihanRisad 5 ай бұрын

i couldnt able to do df.corr() because it was saying some columns are not numeric so that case i had to use numeric_df; numeric_df = df[['2022 Population', '2020 Population', '2015 Population', '2010 Population', '2000 Population', '1990 Population', '1980 Population', 'Area (km²)', 'Density (per km²)', 'Growth Rate', 'World Population Percentage' ]] numeric_df.corr()

@dwbrow3 4 ай бұрын

Try df.corr(numeric_only=True)