Corrections: 16:50 I say "66", but I meant to say "62.48". However, either way, the conclusion is the same. 22:03 In the original XGBoost documents they use the epsilon symbol to refer to the learning rate, but in the actual implementation, this is controlled via the "eta" parameter. So, I guess to be consistent with the original documentation, I made the same mistake! :) Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/

I got my first job in Data Science because of the content you prepare and share. Can't thank you enough Josh. God bless :)

I'm starting writing my Master Thesis and there were still some things I needed to make clear before using XGBoost for my classification problem. God Bless You

That's why I pay my Internet.

Can I just say I LOVE STATQUEST! Josh does the intuition of a complex algorithm and the math of it so well and then to make it into an engaging video that is so easy to watch is just amazing! I just LOVE this channel. You you boosted the gradient of my learning on machine learning in an extreme way. Really appreciate these videos

This dude puts the STAR in starmer. You are an international treasure.

I am a graduate student at Duke, since some of the materials are not covered in the class, I always watch your videos to boost my knowledge. Your videos help me a lot in learning the concepts of these tree models!! Great thanks to you!!!!! You make a lot of great videos and contribute a lot in online learning!!!!

Hi Josh, I just bought your illustrated guide in PDF. This is the first time I've supported someone on social media. Your video helped me a lot with my learning. Can't express how grateful I'm with these learning materials. You broke down monster maths concepts and equation to baby monster that I can easily digest. I hope by making this purchase, you get the most contribution out of my support. Thank you!

After watching your video, I understood the concept of 'understanding'.

I wanted to watch this video last week, but you sent me on a magical journey through adaboost, logistic regression, logs, trees, forests, gradient boosting.... Good to be back

I have never seen an data science video like this....good informative, very clear, super explanation of math and wonderful animation and energetic voice....Learning many things very easily....thank you so much!!

Wow, I am really interested in Bioinformatics and was learning Machine Learning techniques to apply to my problems and out of curiosity, I checked your LinkedIn profile and turns out you are a Bioinformatician too. Cheers

Fantastic explanation for XGBoost. Josh Starmer, you are the best. Looking forward to your Neural Network tutorials.

Best Channel for anyone Working in the Domain of Data Science and Machine Learning.

I've never I had so much fun learning something new! Not since I stared at my living room wall for 20min and realized it wasn't pearl, but eggshell white! Thanks for this!

In my heart, there is a place for you! Thank you Josh!

An incredible job of clear, concise and non-pedantic explanation. Absolutely brilliant!

Nowadays I write a "bam note" for important notes for algorithms.

Man, the quality and passion put into this. As well as the sound effects! I'm laughing as much as I'm learning. DAAANG. You're the f'ing best!

I just LOVE your channel! Such a joy to learn some complex concepts. Also, I've been trying to find videos that explain XGBoost under the hood in detail and this is the best explanation I've come across. Thank you so much for the videos and also boosting them with an X factor of fun!

I had left all hope of learning machine learning owing to its complexity. But because of you I am still giving it a shot..and so far I am enjoying...

Hey Josh! This is fantastic. As an aspiring data scientist with a couple of job interviews coming up, this really helped!

Thanks Josh for your explanation. XGBoost explanation cannot be made simpler and illustrative than this. I love your videos.

You make learning math and machine learning interesting and allow viewers to understand the essential points behind complicated algorithms, thank you for this amazing channel :)

Clear explanations, little songs and a bit of silliness. Please keep them all, they're your trademark. :-)

I watched all of the videos in your channel and they're extremely awesome! Now I have much deeper understanding in many algorithms. Thanks for your excellent work and I'm looking forward to more lovely videos and your sweet songs!

Man, you do deserve all the thanks from the comments! Waiting for part2! Happy new year!

xgboosting！This must be my Christmas 🎁 ~~ Happy holidays ~

I have been waiting for your video for XGBoost, hope for LightGBM next!

Thanks for making such great videos, sir! You indeed get each concepts CLEARLY EXPLAINED.

I am learning machine learning from scratch and your videos helped me a lot. Thank you very much!!!!!!!!!!!

I actually started looking for XGBoost, but every video assumes I know something. I have ended up watching more than 8 videos just to have no problems understanding and fulfilling the requirements, and find them awesome.

Thank you! Super easy to understand one of the important ml algorithm XGBoost. Visual illustrations are the best part!

For my future reference. 1) Initiate with a predicted value e.g. 0.5. 2) Get residual. Each sample vs. initial predicted value. 3) Build a mini tree, using the Residuals value of each sample. .Residuals .Different values of feature as cut off point at branches. Each value give a set of Similarity and Gain scores ..Similarity (use lambda here, the regularisation parameter) - measure how close the residual values to each other ..Gain (affected by lamda) .Pick the feature value that give highest gain - this determines how to split the data - which create the branch (and leaves) - which produce a mini tree. 4) Prune tree. Using gain threshold (aka complexity parameter), gamma. If gain>gamma, keep branch, else prune 5) Get Output Value OV for each leaf. Mini tree done. OV = sum of Residuals / (no. of Residuals + lambda) 6) Predict value for each sample using the newly created mini tree. Run each sample data through the mini tree. New Predicted value = last predicted value + eta * OV 7) Get new set of residual: New predicted value vs actual value of each sample. 8) Re do from step 3. Create more mini trees... .Each tree 'boosts' the prediction - improving the result. .Each tree creates new residual as input to creating the next new tree. ...until no more improvement or no. of tree is reached.

Thank you! I'd been waited for XGBoost explained for so long

As always, loved it! I can now wear my Double Bam t-shirt even more proudly :-)

Hey Josh, first I wanted to say thank you for your awesome content. You are the number one reason I am graduating my degree haha! I would love a behind the scenes video about how you make your videos. How you prepare for topic, how you make your animations and your fancy graphs! And some more singing ofcourse!

You are the best, Josh. Greetings from Brazil! We are looking forward you video explaining clearly the LightGBM!

You make it little bit easy to understand Josh . I am saved.

Gosh! I love your fellow-kids vibe!

Hey Josh, I really love your contents, you are the one who really explains the model details.

I've been using Random Forests with various boosting techniques for a few years. My regression (not classification) database has 500,000 - 5,000,000 data points with 50-150 variables, many of them highly correlated with some of the others. I like to "brag" that I can overfit anything. That, of course, is a problem, but I've found a tweak that is simple and fast that I haven't seen elsewhere. The basic idea is that when selecting a split point, pick a small number of data vectors randomly from the training set. Pick the variable(s) to split on randomly. (Variables plural because I usually split on 2-4 variables into 2^^n boosting regions - another useful tweak.) The thresholds are whatever the data values are for the selected vectors. Find the vector with the best "gain" and split with that. I typically use 5 - 100 tries per split and a learning rate of .5 or so. It's fast and mitigates the overfitting problem. Just thought someone might be interested...

Love StatQuest. Please cover lightGBM and CatBoost!

woah woah woah woah!... now i got the clear meaning of understanding after coming to your channel...as always i loved the xgboost series as well. thank you brother.;)

God, thank you for your "beep boop" sounds. They just made my day!

I have been super excited for this quest! Thanks as always Josh

Awesome... this vid should be a mandatory in any schools

whenever i can't understand anything, I always think of statquest...BAM!

Awesome video!!! It's the best tutorial I have ever seen about XGBoost. Thank you very much!

Wow woww wowww !! How can you explain such complex concepts so easily. I wish I can learn this art from you. Big Fan!! 🙌🙌

Thank you for posting this! I have been waiting for it for long!

Stat quest is the bestttttt!!! love it love it love it!!!!!!

pro tip: speed to 1.5x

I fell in love with XGBOOST. While Pruning every node I was like whatttt :p

This is really helpful, thanks for putting them together!

Thanks for uploading this.. i am your biggest fan!! I have noticed too many adds these days which really disturb :)

thankyou so much i watched it 3-4 times already but finally everything makes sense. thankyou so much

This has been one video I’ve been waiting for and it was well worth it. Brilliant as usual Josh. I wanted to ask about the differences between the XGBoost regression tree and the traditional regression tree with Boosting. It seems that the main difference is that the XGBoost version uses the gain measure (made of similarity) to determine the split thresholds for each feature (I presume if we had more than dosage we would consider them in the same way) and prunes according to the gamma parameter. Whereas the traditional tree uses a measure like Gini impurity to split and a method like cost complexity pruning. Is that the main difference? Or are there any more? Could you also mention why this type of tree is better than the traditional version? It seems like the algorithm has some optimisation for this type of tree than the other.

Thanks, it helped a lot! Looking forward to part 2, and if possible please make one on catboost as well!

Hi Josh, Love your videos. Currently preparing Data Science interviews based on your video. Actually, really want to hear one about LGBM !

You just amazing Josh. Xtreme Bam!!! You make our life so easy. Waiting for neural net vid and further Xgboost parts. Please plan a meetup in Mumbai. #queston

Happy holiday man! Waiting for your next episode

I'm doing a club remix of the humming during calculations. Stay tuned!

your videos have helped me a lot!! thank you so much i hope you keep on making these videos:)

You are always awesome no better explanation ever seen like this ❤️❤️ big fan 🙂🙂.. Triple bammm!!! Hope we have Lightgbm coming soon.

Thank you for this explanation. In python there is another regularization parameter, Alpha. Also, to the best of my knowledge the role of Eta is to reduce the error correction by subsequent trees in order to avoid sum explosion and in order to control the residual error correction by each tree.

Thank you for all of your videos! Super helpful and educational. I did have some questions for follow-up: - With Gamma being so important in the pruning process, how do you select gamma? I ask because aren't there situations where you could select a Gamma that would/wouldn't prune ALL branches, which would defeat the purpose of pruning right? - Is lambda a parameter that: a. Have to test multiple and tune your model to find the most suitable lambda (ie set your model to use one lambda) b. You test multiple lambdas per tree so different trees will have different lambdas

That DANG was unexpected.....you should have given a DANG alert 😋

I justo found this channel and i think it's amazing.

Can't thank you enough, MGB you 😊😊😊

Josh you are amazing!

8:45 checking my headphones - BAM; no problem with my headphones; 10:17 Double BAM; headphones are perfect

Wow! Very well explained, hats off.

Bam!!! I am totally hypnotized

Hey Josh, love your videos :) Any idea when you will make the videos for CatBoost and Light GBM ?

xgboosting! my Christmas gift!

that DANG!!! just brought my attention back😂

Thank you, Josh., I'm watching your videos every day these past couple of months. Quick question, you mentioned the initial prediction being 0.5. Some other materials I read uses "the average of all the target values in the dataset"...could you pls help me understand this a little bit more about this initial prediction? Does it really matter between 0.5 and the mean of the target values?

Gain in Similarity score for the nodes can be considered weighted reduction of variance of the nodes BTW good attempt to make this digestible to all

Best XGBoost explanation i have ever seen! This is Andrew Ng Level!

Hi.... I have 4 questions: 1) Are Gradient boosting and XGboost algorithm works in same way? Are they both using same steps like- Initial prediction, residual calculation, construct a first tree for fitting residuals etc OR there is a difference in steps (Note: I am not asking how Gradient boosting regression trees and XGBoost regreesion trees are created)? 2) Are initial prediction 0.5 is some random value? Can't we intialize initial value in a same way as we did in Gradient boosting (mean of Dependent variable)? 3) How have you decided threshold for dosage like Dosage15, and Dosage>22.5? Is there any steps or rule for that? 4) What learning parameter do and it should have small or high value? What is lamda here? Can we take any value of lambda? Sorry for many questions but answers of these queries will help me to learn better XGBoost. Thanks in advance

Life saver. Was waiting for this.

JOSH is the top data scientist in the world

I'm enjoying your videos. I'd love if you can do one on Tabnet.

Great Xmas present! Thanks Josh!

Waiting for this video since long back.

Wonderful as usual Josh

Thank you so much. You are the best

Wonderful tutorials, not only this video, but every video in StatQuest. Probably, the best videos with good explanation available in KZfaq. I was stuggling with Python until I followed your videos. Now I am very confident in analysing the big data. One question: I am looking at the recurrence of a disease following surgery and I evaluate time for recurrence and probability with CPH. But in non-life science, eg. customer churn, default of payment etc there is no censored cases and not considered in the ML models, such as XGBoost. Is it correct for me to use ML in the similar way for cutomer churn and customer default, despite the censored data? Please advice. Many thanks. I (not only me every budding data scienist) would very much appreciate if you could create a tutorial video (and upload) generating Nomogram for Time event? This will help me (and others) to analyse and publish to a peer reviewed Journal on the dataset I've collected on Cancer recurrence. Best wishes.👍

This is fantastic!

Hey Josh, the series is fantastic! I'd like to ask you to consider two more aspects of tree-based methods: 1) SHAP values (e.g., feature importance, interactions) and 2) nested data (e.g., daily measurements --> nested sampling?). I am more than happy to pay for that :-) thanks!

Xtreme Christmas gift!! :) Thanks!!

Thank you for sharing this amazing video!

Bravo! Excellent presentation. I've been through it a bunch of times trying to write my own code for my own specialized application. There's a lot of detail and nuance buried in a really short presentation (that's a compliment - congratulations!). Since you have nothing else to do (ha! ha!), would you consider writing a "StatQuest" book? I'll bid high for the first autographed copy!

XTREME BAM

please post slides, this is the best channel for ML. thank you

Hey Josh! Thanks for the video, just wanted to know when will you release part 2 and 3 of this?

Hi Josh, I have a question, in min 1 you blend in many phrases on the left side such as Approximate Greedy Algorithm, Weighted Quantile Sketch etc and you mention you will go through them one by one, but in this video only the first three phrases are covered. Are there videos on the other ones? I also wanted to thank you for all the work you put into your videos! Really helps me to understand complex concepts

Amazing explanation as usual !!! Josh, is it possible to make a StatQuest about LightGBM ? I'm sure that it will help a lot of students like me. Thank you very much !

This channel is sooo cool!! 🚀🚀🚀🔥❤️🥺