Python Project to Scrape YouTube using YouTube Data API | Analyze and Visualize YouTube data

Рет қаралды 138,146

Күн бұрын

In this video, we build a Python Project to Scrape KZfaq data using KZfaq Data API. Using KZfaq API, we extract the data and then load this data into a Python Pandas DataFrame and then analyze this data. Finally, we build simple visualization from this data using the Python Seaborn library.
Link to create KZfaq API Key: console.cloud.google.com/
Link to Google KZfaq Data API documentation: developers.google.com/youtube/v3
Link to my blog where you can download this project for free: techtfq.com/video/python-proj...
Timestamp:
00:00 Intro
01:19 How to generate KZfaq API Key
03:27 Google KZfaq Data API Documentation
07:48 Pre Requisite for project (Creating virtual environment and install python packages)
09:45 Part 1 - Scrape Channel statistics from youtube. Analyze and Visualize KZfaq channel
32:32 Part 2 - Scrape Video details for a youtube channel. Analyze and Visualize video details.
We start this project by first creating an KZfaq API Key which will be our credential to access youtube data. I will should you in detail, how to create an API Key.
Once the API Key is generated, we will then learn how to use this API key to access different youtube data. I.e. we will walk through the documentation given by google to use youtube API. We will look at the different sections in the documentation to access different data we need to build this project. We will also look at the sample python code given by google to call different resources and methods to fetch youtube data.
Finally, we will get into writing the python code to build this project. I will be using Jupyter Notebook to write my python code. Since it is a new project, we will create a new virtual environment for this project. We will use anaconda for this. Once the virtual environment is set, we will then install all the required python packages. So we will install "google-api-python-client" (which is the google python package required to access youtube api data), we will also install pandas and seaborn. I will show you how to create a virtual environment and also how to install all these packages in detail.
Once our environment is set and required packages are installed, we will then start writing the code in Jupyter Notebook. I have divided this project into 2 parts.
In the first part, we extract channel details from youtube. I.e. we extract details such as youtube channel name, total no of subscribers, total views and total number of videos posted by each channel. We gather these details for few Data Analyst/Data Scientist kind of channel and then compare these channel data with each other. We shall see who has the highest subscriber and who gets the most views and the amount of videos posted by these channels. We will be loading all of this data into a pandas dataframe and then analyze it. We will also generate some basic visualization using this data so we can easily compare these multiple channels.
In the second part of the video, we shall build a logic to extract video details from a particular channel. We shall extract details such as video title, total views each video has got, total number of likes, dislikes and comments each video has got. We shall extract these details for all of the videos posted by a particular channel. We will then analyze this data by loading it into a pandas dataframe. At the end we will create some simple visualization using Seaborn python library.
Hopefully, this can be a good starting project for anyone aspiring to become a Data Analyst. If you find this video useful then please make sure to like the video and subscribe to the channel.
🔴 WATCH MORE VIDEOS HERE 👇
✅ SQL Tutorial - Basic concepts:
• SQL Tutorial - Basic c...
✅ SQL Tutorial - Intermediate concepts:
• SQL Tutorial - Interme...
✅ SQL Tutorial - Advance concepts:
• SQL Tutorial - Advance...
✅ Practice Solving Basic SQL Queries:
• Practice Solving BASIC...
✅ Practice Solving Intermediate SQL Queries:
• Practice Solving INTER...
✅ Practice Solving Complex SQL Queries:
• Practice Solving COMPL...
✅ Data Analytics Career guidance:
• Data Analytics career ...
✅ SQL Course, SQL Training Platform Recommendations:
• SQL Course / Training
✅ Python Tutorial:
• Python Tutorial
THANK YOU,
Thoufiq

Пікірлер: 293

@digdra Жыл бұрын

Best tutorial about getting KZfaq data I found here! Your code is cleaner than the one proposed in other tutorials. Thank you!

@laurak5196 2 жыл бұрын

Hi Thoufiq, this is very impressive! I’m watching this purely out of curiosity bc I’ve not started my python class yet; however, I would love to see SQL project ideas, maybe for data cleaning and/or analysis. Thanks as always for your clear explanations and guidance 👍🏻 Also very glad to see you’re closing in on 15K, friend!! Congratulations in advance 😀👌🏻🙏🏻

@techTFQ 2 жыл бұрын

☺️😃 thanks Laura Yeah I’ll make some SQL projects for Data analysis in the near future.. Yeah feels great to see the channel growing 😍 Grateful for all your well wishes 🙏🏼

@sethpai Жыл бұрын

Brilliant tutorial. Appreciated the pace and detail. With this tutorial, I can focus my time on exploring KZfaq data and not worrying about how to get it!

@hoekbrwr Жыл бұрын

I am happy I struggled over this. I was googling for a Google Apps script and ended in Python. It is a very clear explanation and I am so happy about your English pronounciation. Lot of your country men(or women) have a strong accent i get very very tired of, so I give up or miss too much. So I subscribed to your channel!

@pzuazu8636 Жыл бұрын

Wow, I love how you bring it all together. Thank you again.

@urvindarkaur0278 Жыл бұрын

I haven't found anything better on youtube than your videos for data analysis. Only request I have is for you to make as many projects as you can from easiest to most difficult projects so that we can be thorough in this field. Even if I have to pay for it, I'd be more than happy. BUT , please don't stop here especially for projects...learning languages is slightly different than when we have to put together all of the learned tings in the project

@techTFQ Жыл бұрын

Noted and will come up with more projects in the near future

@manickmax6818 Жыл бұрын

@@techTFQ I'm also waiting bro do more projects bro for beginners it's very helpful

@MichaelThompson-pe1wi 2 жыл бұрын

I had to do a project like this for my internship. You are a lifesaver!

@user-uv5lr7bk8n 3 ай бұрын

did you get internship?

@AlexandreDias18 Жыл бұрын

Wow, as a fellow data scientist, I am truly amazed by the incredible work you've done! Your expertise in data analysis and modeling is truly inspiring. Your ability to harness the power of data to uncover insights and make informed decisions is truly invaluable. Your work has helped businesses optimize their operations, improve customer experiences, and drive innovation. Thank you for your hard work and dedication in making the world a better place through data-driven insights. Keep up the amazing work!

@maulairfani9474 8 ай бұрын

WOWWW

@mhapich Жыл бұрын

This is the first video that I've watched on scraping ANYTHING that I was able to follow from beginning to end, understand, and be able to apply to my own channels and project...without running into problems that couldn't be easily fixed! I could fix my own mistakes because you explained each step so thoroughly. Thank you!!

@ashkraze Жыл бұрын

hi i was scrapping vids from pewdiepie,mrbeast channel and i encountered an error i am able to fetch title and published date of all video_ids but in fetching likes it is showing "" 'KeyError' : 'lkeCount "" and same for viewcount, comment count.. Did you face the same issue and if yes how did youresolve it??

@mhapich Жыл бұрын

@@ashkraze I did run into that! What I did was use a nested try except block. I put the code from this video on creating the dictionary in try, then, under except, I put the same code, but I commented out the line that was giving me the error. Does that make sense? I needed to nest three more blocks depending on the error that was getting thrown. In each except block, I commented out a different statistic in the dictionary.

@ashkraze Жыл бұрын

@@mhapich yes that MAKES SENSE, fixed it and its running absolutely fine, Thanks mate!

@d4lm0 Жыл бұрын

I'm having a error (even after closing the execute wit () ). Help plz

@kalyanijog 11 ай бұрын

@@ashkraze only title,published_at and views work well for me, for like and comment I got keyError

@kiruthickagp 10 ай бұрын

Thanks a loy techTFQ. I was struggling a lot to do the initial steps of this project . Please do lot of videos like this !!!!!!!

@Anonymous-le2zr 2 жыл бұрын

One of the best Analysis video I have seen so far Amazing Content , Expecting videos on some more projects . Appreciate your hard work !!

@techTFQ 2 жыл бұрын

Thank you 🙏🏼 Glad you liked it 🙂

@zechariaparas9885 Жыл бұрын

Awesome Video! Subscribed because of this! Keep up the good work! Godsent

@ExcelNWithInisa Жыл бұрын

I'm only 25 minutes in and I've learned sooooooooo much already!! Thank you :)

@wingdigitalmarketing 9 ай бұрын

That was wonderfully helpful, Thoufiq. I am a complete novice and while elemets of the coding, such as programs you used, where to find them, how to intergrate are aspects I need to further study, the process you took was clear and inciteful. I paid attention throughout and flatter myself that I saw a few of your return errors before you did 😉 Now to study some more and see where this takes me. Thanks again. David

@samsonkiflom5954 2 жыл бұрын

Thanks for creating this project. I watched it like an action movie :) and going to learn bit by bit again by creating the same project again.

@AK-Star007 Жыл бұрын

This was exactly what i was looking for, to the point explanation. Thanx

@mohammedshahil4898 2 жыл бұрын

Was waiting for this😃Thanks a lot brother for this very helpful video👌🏻 Way to go🙌🏻

@techTFQ 2 жыл бұрын

Thank you hero 🙏🏼 Glad you are finding these videos useful 😍❤️

@dakshbhatnagar 2 жыл бұрын

Thanks, Brother. This was a bit intricate for me as I have just started out learning Python for Data Analysis but the tricks you have used are new and I am eager to learn and use them and create a new project based off of this information. Don't worry, keep at it, you will be bigger than Ken Jee one day, just stay consistent.

@techTFQ 2 жыл бұрын

Thank you Daksh 🙏🏼 I am glad you found this useful 🙂 And there is no competition with any other KZfaqrs, everyone is amazing at what they do .. I chose this example as I felt anyone and everyone could relate to this example and easily understand the concepts.. But thank you for your kind words 🙂

@user-uv5lr7bk8n 3 ай бұрын

for a non-tech background like me, your tutorial is super easy to follow.

@ChristinaK-r6n 27 күн бұрын

Thank you for putting this together! It was super helpful for me

@user-ye8xi6kf3q 5 ай бұрын

I have to say that you are very great...I am following your channel.Your projects are just amazing for getting job.

@fathimafarahna2633 2 жыл бұрын

Fab as always... concise and on fleek. Keep up👌🏻👌🏻🙏

@techTFQ 2 жыл бұрын

Thank you ❤️😘

@tmh3929 2 жыл бұрын

Thanks a lot techTFQ! It's a great project indeed.

@techTFQ 2 жыл бұрын

Thank you :)

@jeeyuuni Жыл бұрын

thank you so much for this ! it was super easy to follow along you are a great teacher!

@techTFQ Жыл бұрын

Thank you very much 🙏🏼 Glad it helped

@swarnimsrijan7470 2 ай бұрын

what a brilliant lecture! Really impressed by you sir, thanks for helping out I the journey

@annarocha9769 Жыл бұрын

OMG, this is awesome, thanks a lot!👏👏👏👏👏

@mayankjacky 5 ай бұрын

A great video which covers all aspects of creating a project in Python for Data Science. Thank you for your service to the community at large. Regards.

@techTFQ 5 ай бұрын

Your welcome, glad it helps

@sultankhatri7230 2 жыл бұрын

Thanks a lot for this video!!! 😊😊 Btw liked and ofc already subscribed. Please upload more videos of projects

@techTFQ 2 жыл бұрын

Thank you Sultan 🙏🏼 Really happy to know you liked this video.. 🙂 Yeah I plan to upload more project videos..

@sultankhatri7230 2 жыл бұрын

@@techTFQ thanks a lot brother 😊😊

@techTFQ 2 жыл бұрын

Your welcome bro 🙏🏼

@yichienchong7474 2 жыл бұрын

Thank you Thoufiq, this is a great project.

@techTFQ 2 жыл бұрын

Glad you like it

@jarosawszyc8287 Жыл бұрын

Absolutely amazing video!

@savyhash1855 5 ай бұрын

I have never seen such a detailed ml video..great

@Empeiro Жыл бұрын

Brother what a nice video, thanks for the time and sharing!

@KingofZamunda. 2 жыл бұрын

This is impresive and easy to follow,👌🏻 however KZfaq disabled the dislike count( The statistics.dislikeCount property was made private as of December 13, 2021. This means that the property is included in an API response only if the API request was authenticated by the video owner.) and this will affect the function to get video details.

@sabaamanollahi5901 2 жыл бұрын

Thank you Thoufiq for this incredible video

@muhammaddanialtanoli 11 ай бұрын

Nicely explained each and every detail thank you so much

@fbalseiro 2 жыл бұрын

Hi, great tutorial! Congratulations for your work! It really helped me a lot to know how to work with KZfaq API.

@techTFQ 2 жыл бұрын

I am glad to read this Filipe 🙏🏼

@soumilyade1057 Жыл бұрын

30:52 well, now you have 215K subscribers xD Congrats on that, and thanks for the video....!

@khanofficial5370 2 жыл бұрын

Assalamualaikum Toufiq , It's really a best ever vedio which i look in KZfaq for web scrapping.

@techTFQ 2 жыл бұрын

Walaikum salaam and thank you 🙏🏼

@kowsergazi 11 ай бұрын

Great video. Looking for more like this.

@fortunusgames 7 ай бұрын

Hi Thoufiq, this is a very impressive tutorial. I followed this and managed to create my own visualizations for my Master of Arts project, which analyzes several political KZfaqrs' video output. Very helpful!

@techTFQ 7 ай бұрын

Thanks and glad it helped:)

@justiconblack 6 ай бұрын

@@techTFQ Hello Thoufiq, I'm trying this exact same thing except I'm using username instead of channel Id, But the response only displays pageInfo : total results: 0 Any idea on what is wrong?

@rahulrajeev9763 2 ай бұрын

Thank you so much for this stuff.

@fahnub 2 жыл бұрын

this is such a good practice project. definitely going on my github.

@varalakshmidanda2684 2 жыл бұрын

Excellent work and great explanation sir.. waiting for more of challenging videos

@techTFQ 2 жыл бұрын

Thank you 🙏🏼 Glad you liked it …

@Wexler111 Жыл бұрын

Great Tutorial! Thanks a lot!

@solkarim2950 2 жыл бұрын

You are a great teacher!

@kalheer Жыл бұрын

Love this easy to follow 👍🏻👍🏻

@hassansemlali7450 Жыл бұрын

Nice content brother, you help me a lot, i really appreciate your efforts

@bobbymcclane2639 6 ай бұрын

your coding skills are very good

@bose19942 Ай бұрын

Thank you for this video

@dodgewagen 6 ай бұрын

Thank you!

@ImranAhmed-vl5po 2 жыл бұрын

good work taufiq thanks for sharing

@techTFQ 2 жыл бұрын

Your welcome Imran Glad you liked it

@gopia5095 Жыл бұрын

Clear explanation ❤

@hoperight5280 5 ай бұрын

Amazing!

@adrianafressato8489 2 жыл бұрын

Great class!

@techTFQ 2 жыл бұрын

Thank you Adriana 🙏🏼 Glad this helped ..

@brainstalk4577 2 жыл бұрын

Very useful ! Thank u

@techTFQ 2 жыл бұрын

Your welcome 🙏🏼

@sameermehdi3143 2 жыл бұрын

Thankyou so much for this wonderful tutorial and from my anlysis you posted most video on Jan and feb :-p

@techTFQ 2 жыл бұрын

Haha that’s pretty right 👍

@sameermehdi3143 2 жыл бұрын

Sir waiting for SQL project

@sameermehdi3143 2 жыл бұрын

One more question do we need to create another API key for another youtuber analysis?

@kancherlavenumadhav9360 2 жыл бұрын

Really interested in this stuff and fabulous explanation ...Also please do videos on Data science how to begin from scratch ......🙌🙌

@techTFQ 2 жыл бұрын

Thank you Kancherla 🙏🏼 Glad you liked it.. Yes will cover videos on data science soon..

@dataneeds2266 2 жыл бұрын

Wow. Amazing

@techTFQ 2 жыл бұрын

Glad you liked it ☺️

@fahnub 2 жыл бұрын

thank you for this

@190HITMAN 2 жыл бұрын

Hi Thoufiq, excellent video absolutely loved it please keep creating many more project videos. Can you make a short video on how to visualize data using python as Dynamic Radar Charts (i.e Player vs player comparision/company vs company - you can filter which two competitors to compare)

@techTFQ 2 жыл бұрын

Thank you Abhishek 🙏🏼 I’ll consider your suggestion for a future video

@Rapha_Carpio 2 жыл бұрын

@shubhangikalambe3435 2 жыл бұрын

Thanks for valuable information

@techTFQ 2 жыл бұрын

Glad you liked it

@ronaldchrishton7969 Жыл бұрын

Tnq .its usefull for me

@kamalverma6064 2 жыл бұрын

You are so brilliant 🥳🥳🥳🙏🙏💐

@techTFQ 2 жыл бұрын

Thank you Kamal 🙏🏼 Glad you liked it 🙂

@Abutturab 3 ай бұрын

top !

@pranayjain._ Жыл бұрын

Thanks a lot❤

@ferdinandosomma8885 Жыл бұрын

It was very useful and easy to understand! Just subscribed. Anyway, I'd like to ask you a question: I have to do a project for my master course, using youtube api, but I need a large amount of datas, basically a lot of channel id of my country (Italy). Is there any method or code that I can use in ordet to fetch all the Ids? Because to me It's impossible to insert them manually one by one. Appreciate any help!

@sandyputrautama9279 2 жыл бұрын

Thank you, very awesome tutorial 🔥😎

@ayushjaiswal350 21 күн бұрын

this is good

@manasasm5061 Жыл бұрын

Thank you

@techTFQ Жыл бұрын

You're welcome

@VictoireRey-pr7qh Жыл бұрын

Hello, I'm trying to run your code to get video details from a list of video_ids but I have a SSL error when running video_details=get_video_details(youtube, video_ids) and don't know how to fix it... Error is the following : "SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)" do you know how I can fix this? Thanksssss

@rajeshjakkawar3338 2 жыл бұрын

project is done...........thank you

@techTFQ 2 жыл бұрын

cool 🙏

@ayuuniyal5657 Жыл бұрын

HI, ModuleNotFoundError: No module named 'googleapiclient' I am geting this error can you help me?

@andersonfernandes6998 9 ай бұрын

Thank you very much for this tutorial. I am starting my studies in data analytics, and this video has helped me a lot in understanding ETL. After seeing the results, I had a question: is it possible to achieve a similar result by extracting comments from each video?

@houses_namibia 2 жыл бұрын

Great video! One thing? Can you write this project in another editor. Pycharm?

@animespot1448 Жыл бұрын

Great videos sir! Its very helpful! Sorry for my bad English, but I want to ask you about groupby in your videos around 1:03:10 How can I show the number of uploaded videos per month in ordered years (years not omitted)?? Thank you in advance.

@arpittrivedi6636 Жыл бұрын

super duper boss

@techTFQ Жыл бұрын

Thank you

@gastonalvarez3386 Жыл бұрын

Hi! congrats for the tutorial, is really clearly explained! I have a question, what if I want to extract the comments from every video of a given playlists of a channel? How can I do that? Thank you!!

@FireFly969 Жыл бұрын

thank you

@techTFQ Жыл бұрын

Your welcome

@shivsharma9153 2 жыл бұрын

Amazinnnggggg

@techTFQ 2 жыл бұрын

Thank you Shiv 🙏

@ShrutiKhandelwal007 Жыл бұрын

Can we also extract random channel names/IDs using python code for the data analysis to know who's at the top in any field? We have to search the channel name to find its id as of now. I had to search google for the top subscribers and then analyzed it. Can't we just randomly sample this whole youtube dataset to get the channel and do the analysis?

@thetruth4461 11 ай бұрын

For those of you who are seeing this project in 2023, in the function to get video ids remove the dislike code as now youtube api does not give information on dislikes.

@denisaroxana4006 Жыл бұрын

Can you get video data from videos posted in a certain week or a specific time frame?

@ryang8192 4 ай бұрын

Hello. i want to ask, how to receive video data (mp4) from youtube?

@haileeyuuno6233 2 жыл бұрын

I am stuck.. I wanted to do a live view counter display but the viewCount does not return live data. Is there anyway to do this? Please help

@manassr7840 Жыл бұрын

Is it possible to use the extracted data and implement an ML model? If so, which would be the best suitable model?

@LP-ig8tg 2 жыл бұрын

Great tutorial, thanks Bro. Also would highly appreciate If you can make similar video on "how to grab each viewers Data". I have been looking for extracting periodical data (Day, Week, Month etc) of viewership, subscribers, comments, likes & Dislikes etc of each user, so as to export it to the Google Sheet for rendering the process. Further any other references or sources to this subject referred shall be highly appreciated

@praveennavin8342 2 жыл бұрын

great video....just a small doubt...can we present this into Excel using CSV module?

@mayankjacky 5 ай бұрын

Thanks

@techTFQ 5 ай бұрын

Thanks Mayank! Very kind of you 🙏

@66.divyajaiswal13 5 күн бұрын

I liked the video but from college prospective: 1)how this project will be helpful 2) why should we develop this project Please answer me....

@YoussefMrabetYMF68 Жыл бұрын

Hi Thoufiq, I would like to follow your instructions but I didn't know how to do the part, " I already imported the build" ? Any help would be appreciated, thank you !!!

@srvos 2 жыл бұрын

Hi Thoufiq, great tutorial! Just getting started here with Python. One quick question. I don't see the YT-env in my jupyter notebook, but did start jupyter from the YT-env in my terminal. Any idea on where this went wrong? Kind regards! Stephan

@techTFQ 2 жыл бұрын

Hi Stephan, Thank you! The list of environments you see in jupyter notebook is actually the kernel and not your virtual environment. You can use the default "Python 3 (ipykernel)" from any virtual environment to run your code. But in case you want to create a kernel by the name of your virtual environment then you can do that as well using the below command. You can run it in terminal from the desired virtual environment. ipython kernel install --user --name=YT-env

@TamaJoki 2 жыл бұрын

Hi! I already running (data.to_csv('Data.csv')) and succeed. But I don't now where that file. Can you answer that? Thank youu

@maryamazam402 2 жыл бұрын

Amazing video.. thank you so much. I would like to know how to get video details of multiple channels at a time. Can you please suggest how to do it.

@ExiledGypsy 2 жыл бұрын

I used to be a C and SQL programmer but now I am retired. I want to download all my comments I have made over the years on different KZfaq into a table or database I can store locally. You seem to have done exactly what I need to do. Can I access all the necessary codes that I can just copy and paste? I will be most grateful.

@saimahmad1445 Жыл бұрын

It was the best video on this topic... I have a question can you please tell me how to use filter on tags while retrieving videos data

@martinbenum5653 Жыл бұрын

Hi Thoufiq. I am trying to follow your process on this and have hit a snag trying to get a channelid for any one using your hack of double clicking as found around 12:30 of your video. Seems KZfaq may have locked that functionality.

@sruthysunil1423 Жыл бұрын

Hi Sir, I would to like to ask you about a doubt regarding the channel ids, i am not able to get the channel id from the above mentioned method , even i have tried to select a video from the channel and checked the url now it all are customised and i wasn't able to get the channel id. Is there any other way to get the channel ids.

@ahmedhassan_saver 2 жыл бұрын

Bro, is there any way with js to get channel id for channels that have customURL?

@roshangupta7844 Жыл бұрын

i wanted to ask if it is possible to get video of a channel from specific country like if i wanted video of creator from india or China only then how can i do that what should be my approach?

@enjoyourjoint5012 11 күн бұрын

Hello! To find the channel ID doesnt work anymore, many of them now starts with @bla bla, so basically i feel like now that channel ID is connect with google, so it shows basically their username of google account. There's some other way we can find it? Thanks!