Python Data Science Tutorial: Analyzing the 2019 Stack Overflow Developer Survey

Рет қаралды 60,859

Күн бұрын

In this video, we will be learning how to use analyze survey data in Python.
This video is sponsored by Brilliant. Go to brilliant.org/cms to sign up for free. Be one of the first 200 people to sign up with this link and get 20% off your premium subscription.
In this Python Programming video, we will be learning how to download and analyze real-world data from the 2019 Stack Overflow Developer Survey. This is terrific practice for anyone getting into the data science field. We will learn different ways to analyze this data and also some best practices. Let's get started...
The code from this video can be found at:
bit.ly/SO-Survey-2019
CSV Tutorial - • Python Tutorial: CSV M...
Jupyter Notebooks Tutorial - • Jupyter Notebook Tutor...
✅ Support My Channel Through Patreon:
/ coreyms
✅ Become a Channel Member:
/ @coreyms
✅ One-Time Contribution Through PayPal:
goo.gl/649HFY
✅ Cryptocurrency Donations:
Bitcoin Wallet - 3MPH8oY2EAgbLVy7RBMinwcBntggi7qeG3
Ethereum Wallet - 0x151649418616068fB46C3598083817101d3bCD33
Litecoin Wallet - MPvEBY5fxGkmPQgocfJbxP6EmTo5UUXMot
✅ Corey's Public Amazon Wishlist
a.co/inIyro1
✅ Equipment I Use and Books I Recommend:
www.amazon.com/shop/coreyschafer
▶️ You Can Find Me On:
My Website - coreyms.com/
My Second Channel - / coreymschafer
Facebook - / coreymschafer
Twitter - / coreymschafer
Instagram - / coreymschafer
#Python

Пікірлер: 145

@coreyms 4 жыл бұрын

I hope everyone finds this useful. Going out and practicing with real-world data like this is one of the best (if not the best) way to practice your skills. If any of you do your own analysis that's interesting then be sure to share it with us in the comment section below! PS I forgot to mention that as usual, the code for this video can be found in the description section.

@shpazhist 4 жыл бұрын

Corey, can you please do a video where you explain how to get to that level of intermediate-advanced developer where one can play with code really-really easily? In essence, say if one wants to learn about new library or framework or a new challenge - - can you please explain your process how do you go about actually learning that field rather than doing videos on standalone topics as IMO that would really help a lot of people to sorta move to the next level a lot. Thx

@bodekolton5961 3 жыл бұрын

I know it's quite off topic but do anybody know a good website to stream new tv shows online?

@troylorenzo6430 3 жыл бұрын

@Bode Kolton Lately I have been using Flixzone. Just google for it :)

@maddoxronin9643 3 жыл бұрын

@Troy Lorenzo yea, have been watching on Flixzone for since april myself =)

@bodekolton5961 3 жыл бұрын

@Troy Lorenzo thank you, I went there and it seems like a nice service :D I really appreciate it!

@mrityunjayahiremath3951 4 жыл бұрын

Python + Data Science + Corey = ❤️

@Ghasakable 4 жыл бұрын

Man, you are awesome, you are the reason that I open KZfaq, please if you come to Japan one day, I will do my best to show you good places here in Tokyo.

@coreyms 4 жыл бұрын

That'd be awesome!

@mylesmontclair 4 жыл бұрын

Great to have you back producing great content for us. Thanks, Corey!

@AvantjerTube 4 жыл бұрын

Thanks Corey! I’ve been following your excellent tutorials for quite awhile. Just wanted to say that I’m excited and looking forward to your Pandas course(s).

@sanukurien2752 4 жыл бұрын

@Corey, I love watching your videos when I have my morning coffee. It's elegant and super informative. Thank you!

@anaesthesiadolorosa7100 4 жыл бұрын

It's awesome that you've started making data science videos. Please keep them coming! ;)

@mancalos 3 жыл бұрын

Finally a project not using either the Titanic or Iris datasets. Corey, you are a gem. Thanks for all the effort you put into this and every other tutorial.

@Sauhros 4 жыл бұрын

Just what I needed to start off my winter break. I can't wait for your pandas series! Keep up the great work Corey.

@BoukariYameogo 4 жыл бұрын

This is what I was waiting for! Thank you!

@GuilhermePalazzoRodrigues 4 жыл бұрын

That was incredible, really easy to follow and replicate, thank you very much.

@kizzavincent 4 жыл бұрын

Very nice to have you back. Many thanks.

@dadoll1660 4 жыл бұрын

Wow, this is Amazing! Please make more videos for this DS Tutorial series. Thank you Corey...

@Lfppfs 3 жыл бұрын

Awesome tutorial, thanks Corey!

@DistortedV12 4 жыл бұрын

Wow Corey you do it all; thanks for the helpful tutorials!

@HunniiChii 4 жыл бұрын

Hope to see more of these videos from you in the future!

@user-gx9hk8gt3k 4 жыл бұрын

Thank you for Python Data Science Tutorial! Keep it up, please!

@afternoontiger9319 4 жыл бұрын

I'm definitely interested in a future tutorial of using pandas with this data! Otherwise a fantastic job

@ajb4964 4 жыл бұрын

Wow, please do more like this.

@rangavembar 4 жыл бұрын

Brilliant as always!! Looking forward to the Pandas series!!

@nikhillingam4630 4 жыл бұрын

absolutely brilliant work well done

@ramil17998 4 жыл бұрын

Thanks a lot for the video! I love it cause its as real as it can get!

@gtosXD 4 жыл бұрын

Thank you!!!! Really great video!

@philinnc 4 жыл бұрын

Looking forward to seeing a future video where pandas is used to analyze this same data. Really appreciated your stepping through the thought process, something I haven't seen in many other data science tutorials. Approaches to data cleaning is another interesting topic that would be nice to see covered in a future video. Recently there were these FEC reports... :-)

@Vasko10100 4 жыл бұрын

Finally, i understand analyzing Data in Python! I went from Coursera via EDX, UDEMY etc. But this.... this is a pure gold sir! Please, do some course or learning path for Data Science with Machine Learning in Python in depth( let it be if needed a paid learning path). You are one of the best! If you somehow manage to come in Macedonia, tell me, i want to bow before you!

@coreyms 4 жыл бұрын

Thanks! Im still working on a Pandas series. 4 videos are complete but I'm working to finish the rest soon. Glad to have you as a subscriber and member. You all make it possible for me to release these for free. Thanks again! media.giphy.com/media/WvUyMX1gB1gnC/giphy.gif

@gaunterodimm3569 4 жыл бұрын

Great video! This video helped me soooooooo much!

@borgestheborg 4 жыл бұрын

Oh baby! Never clicked on a video so fast in my life! :D

@yosefco3 4 жыл бұрын

Hi great Corey ! Thank you very much on your channel , your flask tutorial is the best i ever seen ! Please could you do a flask-api tutorial too ?

@akhilrapalli4118 4 жыл бұрын

wow what a video from an expert on python like u verymuch corey

@SagarSapkota-eo4ei 4 жыл бұрын

Hi Corey great and amazing content ..thanks a lot buddy

@Jo-id9zm 4 жыл бұрын

You're doing God's work, Corey!

@jayjay6804 4 жыл бұрын

As hobbiest my first inclination would be to extract the data using pivot tables in libreoffice. Thank you for demonstrating the use of python for this purpose.

@rangabharath4253 4 жыл бұрын

Awesome as always

@yomajo 4 жыл бұрын

It seems like Counter() could be a topic of its own. I need to dig more into that. Awesome code-along experience!

@someonesomebody716 4 жыл бұрын

Data Science + Corey Sir = YAY!!!

@syedabdulsalam4659 4 жыл бұрын

Finally you are back :p

@_boris 4 жыл бұрын

Your videos are so brilliant as I want to keep them to myself only... 💎 ...however, way too good not to share them with others. Shared with my classmates - hopefully, they find it useful :)

@KevinOMalleyisonlysmallreally 4 жыл бұрын

You produce some of the best tutorials on python. Itd really help if you did one on advancing jupyter notebook use, and seeing up sublime to work with pandas.

@arnopisspot5115 4 жыл бұрын

Looking forward the panda series boss. Thank you for all your hard work

@amnah.-.4708 4 жыл бұрын

Hi Corey, thank you for the content you put out, it is beyond exceptional. Also, it would be great if you could make an in-depth course on SpaCy, and its application in Data Science.

@Strider1066 3 жыл бұрын

I liked the incremental approach of coding by examples .

@muralitechtimes6487 Жыл бұрын

You are true MASTER of coding. We are now a days not finding your other videos. Do it for us which re very help us to grow.

@BrendanMetcalfe 4 жыл бұрын

nice vid! thanks

@alexanderten5497 4 жыл бұрын

Notification from Corey and Iam here to like the video

@bishantadhikari3440 4 жыл бұрын

Your videos are great and is probably the most easily understandable in all of youtube. Also, can you make videos about implementing AI in python cause there are not any good ones that meet your standard

@michaelworkspace 4 жыл бұрын

In case anybody ran into an error like "UnicodeDecodeError: 'charmap' codec can't decode byte 0x81" while reading the csv file like I had, the fix is to add a encoding='utf-8' parameter to the with open() function. e.g. with open('survey_results_public.csv', encoding='utf-8') as f:

@Mayur7Garg 4 жыл бұрын

Thx man! You are a savior.

@shivstn98 4 жыл бұрын

It can be also fixed by using raw string I.e. use r before first quote Eg. r"hello world"

@jemand4036 4 жыл бұрын

thank u so much, i had no idea what was going on!

@irfandy4hmad 4 жыл бұрын

Thanks bro. U saved our time

@michaelworkspace 4 жыл бұрын

Irfandy Ahmad No problem bro. Have a good one!

@anandinamdar4054 4 жыл бұрын

Legend 🙌🙌

@aydinjalilov2328 4 жыл бұрын

it blew my mind.

@rolandgerard6064 4 жыл бұрын

man, you are good 👍

@MAli-os2yj 4 жыл бұрын

Hi Corey! Love your videos man! I have finished your Flask series and currently doing the Django series. I have a few suggestions for future videos around the Blog Website that involve some ML and Data Science: 1. Introduce 'like and comment' functionality to the application which allows a user to comment and like posts by other users. 2. Then, integrate a recommendation system engine to the Blog application, the system will recommend content to the individual user based on his own posts and the posts that he has liked. 3. Also, a user can look into the profile of other users and see a word cloud of all posts that the other has submitted, this way they can get a rough idea of what the other uses posts. Keep Doing the good work man!

@hserdarkuyuk 4 жыл бұрын

Please do similar live coding, these are great to improve our perspective

@sandeepgaur2407 4 жыл бұрын

Awesome tutorial like always. Please make video on asyncio in python.. please please.

@736939 4 жыл бұрын

Thank you Corey! Continue please: ScikitLearn, TensorFlow

@nowyouknow2249 4 жыл бұрын

Nailed it

@rosyndapee8129 4 жыл бұрын

Nice! What about a future video with some visualization of geospatial data? I.e. using some GPS tracks converted into CSV file with all points lat and long. Would you suggest to use postrgres or postgis to then make queries to the geodata?

@daltonmarhufu9549 4 жыл бұрын

Great vid! Really helpful, could you possibly do a Pandas version of this?

@barungh 4 жыл бұрын

Badass !👍👌

@theDrewDag 4 жыл бұрын

Please make this the first of a Data Science playlist. PLEASE.

@nitishchhabra_me 4 жыл бұрын

Hey @corey please make videos for Airflow DAG's also. They are basically python codes so it would fit nicely within your python series

@md.akib5124 4 жыл бұрын

heeiiii corey. awesome to see another amazing tut from you

@Medhusalem 4 жыл бұрын

Hey Corey, I have seen the previous videos on DataFrames with Pandas where you used Jupyter as well. You mentioned in this video that it is nice that you can run code brackets or just certain lines of code. My question would be then, what is the difference between Jupyter and Spyder? Since Spyder can also execute certain code lines, without loading everything in again and has a super awesome functionality where you have a variable manager in which you can view every variable you declared.

@javierrivasseau8956 Жыл бұрын

Hi Corey, how are you? i have been watching your series and videos, and think they are the very best out there, i really like to learn with you, I would like to ask you some things if you could solve them, when you make the matplotlib series, you separate the convertedcomp for all the devs and for the python dev and javascript dev, can you upload a video to solve it? I tried anyway and could not. ?

@user-bs3jr5ix5r 4 жыл бұрын

Hello, Corey! Please append your matplotlib tutorial with a lesson about Legend Picking

@skandrskon7091 4 жыл бұрын

for line in csv_reader : counts[line['hobbyist'] ] += 1 really great idea for increment on counts dict thanks

@jackleone4150 4 жыл бұрын

If anyone else is facing the UnicodeDecodeError try adding this second argument to the open function => with open('data/survey_results_public.csv', encoding="utf8")

@sylvanmasha 4 жыл бұрын

wow, thank you so so much. i was facing the same problem

@kingslypaul2999 4 жыл бұрын

Can u make a video on pyttxs3 ...or something better .....thanks for all your God skilled contents that makes ppl understand anything u teach

@morganfree100 4 жыл бұрын

More please sir..(:-

@iangitonga2811 4 жыл бұрын

Great content.

@DistortedV12 4 жыл бұрын

From this, looks like you can't go wrong with javascript, python, SQL and a C type language. Wonder how long these will dominate.

@strange_man 4 жыл бұрын

The video that I need :D

@opst1704 4 жыл бұрын

If one day I decide to learn Python, then do I waste my time by learning C language now? I am learning C, primarily because I want to understand how memory works and DSA is easier to understand with C. Will this knowledge be helpful if I decide to learn Python? Seems like C and Python have a lot of differences, in Python there are so many things that I don't need to worry about. One more question, is it difficult to find out what is happening behind scenes in Python, for example: implementation of input() function? Great videos, the main reason why I am considering to learn Python.

@rahulsharma-st5yg 4 жыл бұрын

@corey: when to expect some data science real life projects?

@aryanvikash1967 4 жыл бұрын

Corey please create a tutorial on python coroutine.. async await its so confusing .. please

@sprk2009 4 жыл бұрын

Your videos are awesome. I am new to Python and preparing for Python certification. Can you please let me know which one of the following programs are good to proceed? Thanks in advance. 1. PCAP: Programming Essentials in Python from Netacad 2. Microsoft 98-381 Introduction to Programming Using Python

@pingojr4048 4 жыл бұрын

Can you do something for home bill management?

@mitos95176 4 жыл бұрын

Now I want to perform this task using pandas.

@arkhamian7586 4 жыл бұрын

Don't stop doing this.

@chinmayabarik557 4 жыл бұрын

Please do a pandas tutorial with this survey results sir..

@graybits4111 4 жыл бұрын

thanks for this great video the total % for each dev type should be 100%, which is not the case

@coreyms 4 жыл бұрын

Not necessarily. They can choose multiple languages. So for example, for front-end developers, most said they know JavaScript, but they also knew other languages as well. So each language is a percentage of how many developers said they know a particular language. Hope that clears it up

@graybits4111 4 жыл бұрын

@@coreyms it is clear now, thanks

@ashissahoo4158 3 жыл бұрын

Is the pandas version of the last question has been uploaded?

@ebrahimm.roshdy4487 4 жыл бұрын

Please do it using pandas since it's way easier and so common for data scientists

@shivamkumraa 4 жыл бұрын

Who else liked before watching this. Just 1000% sure this guy can't make any shit.

@Jmzz542 4 жыл бұрын

what makes you say that?

@amankothiyal7074 3 жыл бұрын

Hey can you analyse it using numpy library

@darrenlefcoe 4 жыл бұрын

On inspection of the data. The csv.DictReader(f) throws 594 errors in the for loop >> line in csvReader: Not sure what causes the error, but this exists. Needs to be handled with try: except UnicodeDecodeError: who else sees this ?

@Medhusalem 4 жыл бұрын

It didn't work somehow and I had to do a google search on my error: 'charmap' codec can't decode byte 0x81 in position 5739. So I had to change the with open part and specify the encoding part, after that everything worked fine. *with open('data/survey_results_public.csv', encoding='utf-8') as f:*

@rahulpatel4701 4 жыл бұрын

thanks for help

@yuliu1105 4 жыл бұрын

Please update pandas version of the 2nd half of the problem, my forloops don't work, bummer. Thank you

@natesh1 4 жыл бұрын

Please do a pandas tutorial series with this survey results too.

@bluekeybo 4 жыл бұрын

You didn't watch the video till the end did you

@gnuter3000 4 жыл бұрын

greatgreatgreat

@johnnya590 4 жыл бұрын

I don't know if you realize this, but at the start of EVERY video you say "Hey there how's it going everybody?" :)

@marinbeslo7841 4 жыл бұрын

Is the video with pandas-way out? :)

@hemanth6951 4 жыл бұрын

i saw you in another KZfaq channels

@coreyms 4 жыл бұрын

Which one was that?

@hemanth6951 4 жыл бұрын

@@coreyms some thing pyconf channel with some one interviewing with you last year

@coreyms 4 жыл бұрын

@@hemanth6951 Oh yeah! I think that was Dan Bader. He's a great guy.

@hemanth6951 4 жыл бұрын

@@coreyms yes yes yes finally I saw you and this second time I'm chatting with you I'm so happy and luky 😊😀😀

@boolloop 4 жыл бұрын

I love your videos, but I wondered why you didn't use pandas for this?

@coreyms 4 жыл бұрын

I mentioned the reason in the video. I wanted to show how to do this without Pandas first, and once I release my Pandas series then I will redo this video using Pandas as well. That way people who are unfamiliar with Pandas can do this as well

@boolloop 4 жыл бұрын

@@coreyms neat, looking forward to it!

@darrenlefcoe 4 жыл бұрын

for line in csvReader: failed on line 16. UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 5739: character maps to problem with the data-set or other reason ?

@sylvanmasha 4 жыл бұрын

f anyone else is facing the UnicodeDecodeError try adding this second argument to the open function => with open('data/survey_results_public.csv', encoding="utf8") this is a comment i found by Jack Leone. you're welcome

@pavanmandikal 4 жыл бұрын

Please do make a video on numpy also

@christiaan3315 3 жыл бұрын

Sorry. I am getting error messages... 'for line in csv.reader: row = next(self.reader) etc

@adildeshmukh4140 4 жыл бұрын

yes, please pandas

@samishabli5191 4 жыл бұрын

i got this error: Yes Traceback (most recent call last): and it just shows me the first yes and no and in the end it says: return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 5739: character maps to

@atishayshukla1117 4 жыл бұрын

Its an encoding issue with the file, open a file by specifying proper encoding something like: with open('data/survey_results_public.csv', encoding='utf8')

@ThePasupathi 4 жыл бұрын

@@atishayshukla1117 thank you..it works

@Ajithkumarsekar96 4 жыл бұрын

Almost everyone has used Javascript and html/css? From front-end developer to system-administrator

@shaguftakhan302 3 жыл бұрын

I clicked on this video just to see what data scientists do.

@olicairns8971 4 жыл бұрын

This is much easier if you use pandas

@coreyms 4 жыл бұрын

Agreed. I mentioned this in the video, but I wanted to show how to do this without Pandas first, and once I release my Pandas series then I will redo this video using Pandas as well. That way people who are unfamiliar with Pandas can do this as well