Python Pandas Tutorial 5: Handle Missing Data: fillna, dropna, interpolate

Рет қаралды 404,116

Күн бұрын

In this tutorial we'll learn how to handle missing data in pandas using fillna, interpolate and dropna methods. You can fill missing values using a value or list of values or use one of the interpolation methods.
Topics that are covered in this Python Pandas Video:
0:00 Introduction
2:30 Convert string column into the date type
3:15 Use date as an index of dataframe usine set_index() method
4:10 Use fillna() method in dataframe
7:35 Use fillna(method="ffill") method in dataframe
8:57 Use fillna(method="bfill") method in dataframe
9:56 "axis" parameter in fillna() method in dataframe
11:18 "limit" parameter in fillna() method in dataframe
13:46 interpolate() to do interpolation in dataframe
15:34 interpolate() method "time"
16:50 dropna() method Drop all the rows which has "na" in dataframe
17:50 "how" parameter in dropna() method
18:33 "thresh" parameter in dropna() method
Code link: github.com/codebasics/py/tree...
Do you want to learn technology from me? Check codebasics.io/?... for my affordable video courses.
Popular Playlist:
Complete python course: • Python 3 Programming T...
Data science course: • Data Science Full Cour...
Machine learning tutorials: • Machine Learning Tutor...
Pandas tutorials: • Pandas Tutorial (Data ...
Git github tutorials: • Git/Github Tutorial
Matplotlib course: • Matplotlib tutorial
Data structures course: • Data Structures And Al...
Data Science Project - Real Estate Price Prediction: • Machine Learning & Dat...
To download csv and code for all tutorials: go to github.com/codebasics/py, click on a green button to clone or download the entire repository and then go to relevant folder to get access to that specific file.
🌎 My Website For Video Courses: codebasics.io/?...
Need help building software or data analytics and AI solutions? My company www.atliq.com/ can help. Click on the Contact button on that website.
#️⃣ Social Media #️⃣
🔗 Discord: / discord
📸 Dhaval's Personal Instagram: / dhavalsays
📸 Instagram: / codebasicshub
🔊 Facebook: / codebasicshub
📝 Linkedin (Personal): / dhavalsays
📝 Linkedin (Codebasics): / codebasics
📱 Twitter: / codebasicshub
🔗 Patreon: www.patreon.com/codebasics?fa...

Пікірлер: 297

@codebasics 2 жыл бұрын

Check out our premium machine learning course with 2 Industry projects: codebasics.io/courses/machine-learning-for-data-science-beginners-to-advanced

@cromllo7162 2 жыл бұрын

This Man its Amazing ❤❤👍👍👌👌

@mixmaza6489 Жыл бұрын

@codebasics CAN we replace missing value using Na_values like we did in pervious videos cleaning messy data ?

@RHCIPHER Жыл бұрын

It's been 5 years since you posted this video but still, we can't find a better video than this to understand the concepts. Not only this but your complete playlist is GEM. Thanks a ton.

@MrMaipeople 5 жыл бұрын

I just don't know how exactly can I say THANK YOU enough to convey my thank you to you THANK YOU, THANK YOU, THANK YOU for all the tutorials in ALL your playlists, so awesome and helpful. ALL ARE SUPER EXCELLENT.

@PaulColclough47 7 жыл бұрын

Thanks very much for this! I'm doing an edX course and I was doing fine until this very topic came up. Your video perfectly cleared everything up which is a great relief!

@codebasics 5 жыл бұрын

Step by step roadmap to learn data science in 6 months: kzfaq.info/get/bejne/fpqJltSGsb-VXYU.html Machine learning tutorials with exercises: kzfaq.info/get/bejne/ndOmqcSgx9OblYU.html

@codebasics 3 жыл бұрын

Step by step roadmap to learn data science in 6 months: kzfaq.info/get/bejne/fpqJltSGsb-VXYU.html 5 FREE data science projects for your resume with code: kzfaq.info/get/bejne/b5tnmbR00pvFdKM.html

@juanliu5778 5 жыл бұрын

Best course that I can find after 2 months research, you are the only guy who know why we need pandas to do excel operations

@adnanhowlader143 3 жыл бұрын

I think your playlist has more rich content than paid courses

@sumit121285 3 жыл бұрын

100% true....

@NOTHING-j2h 2 жыл бұрын

I too agree.

@sameer9045 Жыл бұрын

I sure has

@pavankumararyasomayajulu6365 11 ай бұрын

100%

@amandaahringer7466 2 жыл бұрын

I love the way you used the graph/visual to explain the different interpolation techniques!

@koumospecial 6 жыл бұрын

What an amazing presentation!! You did a fantastic job there! Very professional approach! You just made me a subscriber with that! :) Big thanks!

@MyChusko Жыл бұрын

great tutorial. A very comprehensive explanation of the topic. I have a good command of pandas but I still learnt some new tricks I didn't know. This video has convinced me to buy some of your courses. Serious and in depth approach. Totally recommended!

@sangorilla1 6 жыл бұрын

Excellent presentation of these tools. I like the practical way you have demonstrated HOW to use the tools. Thanks.

@StevenSmith68828 5 жыл бұрын

Finally! Thank you so much I was trying figure out how to drop only the columns that have NaN. I had done it by hand it took me so long Pandas is amazing.

@sippu9722 5 жыл бұрын

LIFE SAVER BRO... THANKS A LOT THIS IS WHAT I EXACTLY WANTED

@purushothamgowda6449 5 жыл бұрын

Good presentation in simple words, easy to understand. Great job Sir!

@jasonwong8315 3 жыл бұрын

the threshold parameter is so useful! never heard about this before! thank you!

@sep69 5 жыл бұрын

Thanks for your great pandas tutorials. Just what I was looking for. You explain it very clear. Thanks again :)

@matthiasmusterman5139 5 жыл бұрын

Very well organized and intuitive tutorial. Great job. Thanks a lot

@saeedbaig4249 7 жыл бұрын

OMG this is exactly what I was looking 4! Thanks so much man u just earnt yourself a new sub

@juliocesarbenavente9590 6 жыл бұрын

With this kind of tutorials I can say to myself: I can do it!

@NidhiSharma-vd6wm 2 жыл бұрын

This tutorial should be awarded as the best tutorial, understood all the concepts ............thankyou so much.

@vatsalsingh4668 5 ай бұрын

Thankyou so much. I love how you have given all ipynb files with such good markdowns and details. Love you for it. Youre a life saver...got my Python exam exam

@vishaljhaveri6176 3 жыл бұрын

Awesome video. Loved all the concepts and mostly your teaching style. :) Thank you, sir.

@pramodpoudel6020 4 жыл бұрын

This guy is a great teacher. Very simple way to teach. Very basics. great dude. Thanks for sharing you skill for free.

@codebasics 4 жыл бұрын

I am glad you liked it pramod 😊

@vaibhavkumar38 5 жыл бұрын

Simple, lucid explanations with examples. Wonderful!!!

@codebasics 5 жыл бұрын

Vaibhav, I am glad you liked it

@prateekkachoria301 4 жыл бұрын

completeness of the material is commendable. keep it up thanks a lot :D

@MasterCoder99 3 жыл бұрын

Interpolate will definitely boost my kaggle score! Thanks so Much!

@himaniagarwal3671 4 жыл бұрын

Very well explained sir. I appreciate that you suggested those little tricks rather than just sticking to the concept.

@codebasics 4 жыл бұрын

Thanks Himani, glad you liked it. 😀

@saurabh9416 5 жыл бұрын

You need more subscribers buddy ! Simply awesome

@Lukas7360 3 жыл бұрын

clean and simple, yet effective, understandable and efficient. Thanks !

@codebasics 3 жыл бұрын

Glad it was helpful!

@surewin3006 3 жыл бұрын

Thanks! Great tutorial! Just finished python and now continue right away with pandas...still am enjoying and learning...

@codebasics 3 жыл бұрын

Glad it helped!

@azmaryzannataurin2844 3 жыл бұрын

The only video that makes sense to me :') Thanks a lot for this tutorial!

@leythecg 4 жыл бұрын

As always very well and understandably explained! Many thanks for that!

@codebasics 4 жыл бұрын

Glad it was helpful!

@vishwa4908 5 жыл бұрын

the perfect example had taken to explaining all the scenarios and nicely presented

@codebasics 4 жыл бұрын

How to learn coding for beginners | Learn coding for free: kzfaq.info/get/bejne/edakpc-qqZawemw.html

@garylanigan1 3 жыл бұрын

Simple and clear demonstrations coupled with simple and clear explanations. Very well presented!! kudos to you sir!!

@codebasics 3 жыл бұрын

Glad it was helpful!

@julesboileau 5 жыл бұрын

This one of the best explain step by step tutorial,

@minhazarnab8660 4 жыл бұрын

Well explanation than most paid courses. Thanks a lot.

@iloveno3 5 жыл бұрын

You have done an amazing job. Thanks a lot!

@emailamit08 2 жыл бұрын

Awesome method of teaching code. it is so clear to the beginners, you even tell minute things like what shortcuts you are using and how to find them. Really boss you deserve a respectful SALUTE. I am so satisfied with your teaching method that i am writing this pausing the video in between.

@codebasics 2 жыл бұрын

🙏👍👍👍

@alexandrelucas5621 6 жыл бұрын

Sir, you are top! Keep up the good work!

@SrisairamRajasekar 4 ай бұрын

When you googled, I saw the pangolin valentine Google doodle. I remember that vividly, as I played that game in my old office in 2017. I can't believe it's been 7 years and wish I had known these videos 7 years back.

@jennythedancer5139 Жыл бұрын

Nice one, I watched it second time and finds very helpful for EDA starters.

@ijeffking 6 жыл бұрын

I have been learning a lot from watching your videos. Than you so much.

@ashutoshranjan4644 Жыл бұрын

It is a real gem🤩 for those who wants to learn pandas

@bouseuxlatache4140 5 жыл бұрын

man I hope you are lecturing at the best university where you are. you have such a pedagogical approach.

@curious8338 2 ай бұрын

Thank you for the great explanation!!!!

@debashissahoo5031 6 жыл бұрын

thank you sir for clear explanations. one small doubt: after setting up day as index value, there is a gap in the first row, why is it so and how to avoid that.

@fadimehdibourarach3666 6 жыл бұрын

Superb Explanation ! Thank you :)

@optimizacioneningenieria3385 2 жыл бұрын

Great video. Thanks!

@glauberbrito8685 4 жыл бұрын

Great job. Congrats!!

@sakshidevi353 Жыл бұрын

you are a magician sir...thanks a lot

@oscarquintero6981 6 жыл бұрын

Thank you, you got yourself a new subscriber.

@kunwar_divyanshu 3 жыл бұрын

🔥🔥🔥🔥🙏Thank you sir for these all videos ! Really helpful .

@tkrath8348 4 жыл бұрын

Really a life saver bro!! Thanks a ton!!

@armagaan007 5 жыл бұрын

Beautifully explained! :)

@chitrajain5059 6 жыл бұрын

Your tutorials are great. Thanks so much. While executing the df in the jupyter notebook, why I can't see the table outline as I can see the video?

@samuelnikhade5612 3 жыл бұрын

Thanks a lot it was helpful !!

@nithyagowda9800 2 жыл бұрын

I was using fill('np.nan') that changed the dtype to 'object' from 'float64' that did not allow me to interpolate. I was able to pick-up on that because of your video! Now, I have to try and use linear interpolation.

@abhinandansingh39 5 жыл бұрын

really nice videos and much better than paid courses

@michaeltwiton2276 5 жыл бұрын

Thank you sir. This video is fantastic!

@vedanthbaliga7686 3 жыл бұрын

This is a very useful video! Thank you🙂

@rohinijadhav744 3 жыл бұрын

Just wow.... Thank u so much!!!!

@cihangiraydoner7962 3 жыл бұрын

Thank you for step by step explanation. Good job!

@codebasics 3 жыл бұрын

😊👍

@evitaooo7 3 жыл бұрын

Thank you so much! So helpful!

@codebasics 3 жыл бұрын

Glad it was helpful!

@talamuslu 5 жыл бұрын

perfect job. keep going

@geekyprogrammer4831 5 жыл бұрын

This is much better than Coursera Intro to Data Science course

@JustPython Жыл бұрын

Best clearing understand video

@erfanebrahimi9748 5 жыл бұрын

Great tutorial. Thank you.

@kmnm9463 4 жыл бұрын

Hi, Excellent examples and explanation. I am facing an issue , after using dictionary with fillna method for replacing 0 values in 'event' column , the df still has only 0s. Krish

@shockey3084 4 жыл бұрын

Good One bro keep it up.

@EcExplorer 2 жыл бұрын

TBH, this is better than the paid course contents like datacamp

@AnimeshSinghIITM 2 жыл бұрын

T hank you very much, it was really helpful.

@chiragmakwana7577 3 жыл бұрын

Sir thank u so much... With one dataset i am facing problems but by seeing u video i got to know sir ... Thanksss sir !

@codebasics 3 жыл бұрын

I am glad you liked it

@mostafaserag4035 3 жыл бұрын

Can you please make a series of videos for the datetime and os libraries? Thanks for your awesome vids!

@abdulmalekwalayeh4790 5 жыл бұрын

what a great video thanks a lot!!

@swamyabhishek2393 4 жыл бұрын

Excellent work!!

@codebasics 4 жыл бұрын

Thank you! Cheers!

@ireneashamoses4209 4 жыл бұрын

Excellent video!! Thank you!! 😊👍👍

@codebasics 4 жыл бұрын

Irene, I am glad you liked it

@anmol4390 4 жыл бұрын

Very very helpful.

@joseluisbeltramone599 3 жыл бұрын

Very good explanation! Thank you very much.

@codebasics 3 жыл бұрын

Glad it was helpful!

@pranitbhisade3174 6 жыл бұрын

wait for it....................Awsome..:)

@debashissahoo5031 6 жыл бұрын

thank you sir, really nice explanation.

@TheMarComplex 2 жыл бұрын

You're the best!

@HawkingMerchant 3 жыл бұрын

This is the best rich in content best for free you are a man of god

@codebasics 3 жыл бұрын

glad you liked it Vinay

@yusuffarah5602 4 жыл бұрын

brother what kind of method do you use to record this video so clear and professional please let me know i am trying to videos for data analysis. Thanks.

@nagavidya6780 Жыл бұрын

Superb!!!

@bhairavidhakras7933 Жыл бұрын

Have a few questions: (1) your data is already sorted by the date column, hence using just the interpolate method makes sense. But if my data is not sorted, should I first create a data frame by sorting on the dates and then use that data frame as an input to the interpolate method? (2) when you use interpolate with "time", how does the program know that it has to use the date column for the time? What if I had date1 like you do and had another column date2 with some other dates, how would the program know that it has do a time-based interpolation on date1 and not date2? (2) can interpolation be done for specific columns only? What if I wanted to do interpolate for temperature and forward fill for windspeed? what would be the syntax like? Thank you! Great video!

@arashalizade9583 4 жыл бұрын

Thank you very much it is very efficient to me , Really helpful !!

@codebasics 4 жыл бұрын

👍🤓

@elysel9424 6 жыл бұрын

really well done!

@manoj9 6 жыл бұрын

Very nicely explained

@sumit121285 3 жыл бұрын

thanks for all your videos...

@codebasics 3 жыл бұрын

Glad it was helpful!

@frankservant5754 4 жыл бұрын

Assuming from your data that you have all the events, how can you fill in the temperature based on the event, eg if the event was "sunny" fill in 32. etc ?

@skkkks2321 4 жыл бұрын

Wow ,learned a lot to handle datasets. Thank you Sir

@codebasics 4 жыл бұрын

You're most welcome

@vishnujatav6329 3 жыл бұрын

Very helpful. Thank you

@codebasics 3 жыл бұрын

Glad it was helpful!

@stevemungai3542 2 жыл бұрын

You are a smooth talking pro

@karthikc7288 5 жыл бұрын

Good job..thank u . My request is make some videos for "seaborn" it will be more useful..

@hanaaghaouti5069 4 жыл бұрын

the perfect tutorial thanks a lot

@mehmetkaya4330 4 жыл бұрын

Great! Thank you!

@ayushy2537 4 жыл бұрын

At 20:20 , you passed dt in DatetimeIndex() to make it DateIndex type. But when we will create a date range from pd.date_range it itself is DatetimeIndex type and we can skip the pd.DatetimeIndex function part.

@arunkumar-us8ei 6 жыл бұрын

Lovely explanation

@rahulsailwal4025 4 жыл бұрын

BTW - i have also subscribed. Thank you once again. Wow. Thank you for uploading series on pandas. Currently going through each and every video and it seems to be a better video. Could you please help me to understand below scenario - 16:45 - Lets assume, we have two dates...Eg. Invoice Pay date, Invoice rec date..is it possible to specify particular date for guessing using interpolate ?