Optimizers - EXPLAINED!

Рет қаралды 112,276

Күн бұрын

From Gradient Descent to Adam. Here are some optimizers you should know. And an easy way to remember them.
SUBSCRIBE to my channel for more good stuff!
REFERENCES
[1] Have fun plotting equations : academo.org/demos/3d-surface-...
[2] Original paper on the Adam optimizer: arxiv.org/pdf/1412.6980.pdf
[3] Blog on types of optimizers: towardsdatascience.com/types-...
[4] Blog on optimizing gradient descent: ruder.io/optimizing-gradient-...
[5] Github gist of code for rending animation of a math function: gist.github.com/ajhalthor/335...
[6] Another Blog to quench your thirst for knowledge on optimizers cuz the other links weren't good enough: machinelearningmastery.com/ad...

Пікірлер: 133

@tombratfred3102 4 жыл бұрын

I like how you anthropomorphize optimizers. Makes me really empathize with their struggles.

@erich_l4644 4 жыл бұрын

with a profile pic like that- you would

@metaprog46and2 3 жыл бұрын

@@erich_l4644 LMAO - your comment just won the internet. You'll soon receive an email by a Nigerian Prince with instructions on how to claim your winnings lol.

@metaprog46and2 3 жыл бұрын

Died laughing like thrice. Witty joke.

@jonass1315 Жыл бұрын

This is how every lecture should be like. Funny but perfectly explained, and greatly visualized. Thanks!

@diaojun161 4 жыл бұрын

The best explaination of optimizers in DL I HAVE EVER WATCHED!

@CodeEmporium 4 жыл бұрын

Thank you! More of this to come!

@dude8309 4 жыл бұрын

great little overview! love how you get to the point quickly yet provide all the needed intuition

@CodeEmporium 4 жыл бұрын

Thanks! That's exactly what I was going for :)

@1harru 2 жыл бұрын

Hands down.. This is the best video on Optimizers.. !!! I've been trying to understand the complex math equations for the past few days and this one literally gave me the overall intuition in 7min 🙏🙏🙏

@CodeEmporium 2 жыл бұрын

I'm so glad this helps :)

@X_platform 4 жыл бұрын

Loving the sound effect

@carebox6187 4 жыл бұрын

This video was both informative and hilarious. I absolutely loved it!

@CodeEmporium 4 жыл бұрын

That was the objective. Glad you liked it :)

@trocketflicks 4 жыл бұрын

Man, this video is slept on. Such a good explanation!

@ErturkKadir 3 жыл бұрын

Such a clear and simple explanation of complicated things. Great job.

@kevinelkin3943 3 жыл бұрын

Such an underrated channel! Great explanations and visuals!

@CodeEmporium 3 жыл бұрын

Much appreciated :)

@ruxiz2007 3 жыл бұрын

This video is so good, and it deserves 100X more attention!

@lakshmisrinivas369 2 жыл бұрын

Absolute way of learning with lot of fun. Thanks for such a funny and insightful video

@DouweMr 2 жыл бұрын

This is one hell of a video to refresh on this stuff! kindly appreciated!!

@a.h.s.3006 2 жыл бұрын

That...... was........ one EXCELLENT VIDEO!!!!! Thank you so much, I thought I would struggle with optimizers but now it's all clear to me

@fahdciwan8709 4 жыл бұрын

thanks! one of those rare videos that explain the intuition perfectly instead of hovering around the terms

@CodeEmporium 4 жыл бұрын

I try. :)

@shubhigautam9655 3 жыл бұрын

the only video thats ever made me laugh while explaining a concept. Love it, thank you!

@CodeEmporium 3 жыл бұрын

You are very welcome :)

@carlavirhuez4785 3 жыл бұрын

Best video ever on optimizer. Thanks a lot.

@BlockDesignz 3 жыл бұрын

Absolutely love this iterative explanation.

@CodeEmporium 3 жыл бұрын

Thank you. I'm experimenting with different teaching styles :)

@ZobeirRaisi 4 жыл бұрын

Your Explanation went to the deep of my brain!

@abhikbanerjee3719 3 жыл бұрын

I am watching this at 2 am in the morning and that sudden effect 00:13 cracked me up!

@zhengyahnis848 3 жыл бұрын

Don't know why this video is under spreading, the explanation is great and the high-level summarization helps me a lot.

@CodeEmporium 3 жыл бұрын

Thank you! Mind fixing that by sharing this around? Would love to get more eyeballs here :)

@anujlahoty8022 Жыл бұрын

Very well explained and in a fun way.

@eniolaajiboye4399 2 жыл бұрын

❤️ the videos man. They're so clear

@Fransphoenix Жыл бұрын

Great explanation and fun, too. Thank you!

@CodeEmporium Жыл бұрын

You are very welcome :)

Жыл бұрын

Best explanation ever!! Thank you so much!!!

@Hariharan-yy1fu 2 жыл бұрын

Awesome work easy to get a quick review before my interview keep going

@aafaq97in 4 жыл бұрын

amazing vid you just earned a subscriber! looking forward to more content like this!

@10bokaj 3 жыл бұрын

Very clear, very well explained 10/10

@CodeEmporium 3 жыл бұрын

Thank you;

@mennoliefstingh5687 4 жыл бұрын

Thanks for the great explanation!

@Anja5233 Жыл бұрын

This is my new favorite video on the internet

@CodeEmporium Жыл бұрын

Thanks so much for the compliments:) I try

@shouravpaul3092 4 жыл бұрын

like your video, and mostly i wanted to see the graph that most people dont show, thank you

@rendevous9253 3 жыл бұрын

Man you gave the best explanation which even a noob like me in machine learning can understand . Keep it up man 👍.

@CodeEmporium 3 жыл бұрын

Awesome! Glad you like it :)

@GauravSharma-ui4yd 4 жыл бұрын

Thanks ajay for giving this a shoot. Loved it❤️

@CodeEmporium 4 жыл бұрын

Thanks for watching Gaurav (and the suggestion). Saw your comment on the last video too. And it was also in a line of videos I wanted to do. Probably not as "mathematical" as you'd like. I wanted to just explain why certain terms appear the way they do. Hopefully this helped that understanding. I might do a more mathematical video in the future though. But for now, this will do :)

@GauravSharma-ui4yd 4 жыл бұрын

@@CodeEmporium You did a pretty awesome job in just 7 minutes. Its both beginner friendly and refreshing for intermediates.

@ahmedaj2000 Жыл бұрын

love it! thank you! explained better than my professors. i finally get these now after so long

@CodeEmporium Жыл бұрын

Words that are too kind. Thank for the kind words

@artinbogdanov7229 3 жыл бұрын

Great explanation. Thank you!

@CodeEmporium 3 жыл бұрын

You are most welcome!

@TawhidShahrior 2 жыл бұрын

man you do great work!

@oskarbartosz9159 Жыл бұрын

m8, i was searching for channel like that for a really long time

@r.y.y8073 Жыл бұрын

I like how you explained this!

@CodeEmporium Жыл бұрын

Thanks so much!

@YangQuanChen 3 жыл бұрын

Nicely done! Thanks!

@CodeEmporium 3 жыл бұрын

Thank youu

@roshanid6523 3 жыл бұрын

Amazing explanation

@mahdijavadi2747 3 жыл бұрын

Thanks for the great clarification!

@CodeEmporium 3 жыл бұрын

Of course. Anytime :)

@ardhidattatreyavarma5337 Жыл бұрын

awesome explanation

@vamsikrrishrocks 2 жыл бұрын

Wow most complex topic in under 7 minutes 😊 with pretty good visualizations.

@m.a.flores7252 4 жыл бұрын

Please keep doing this kind of videos I’m in love with ML and with u

@CodeEmporium 4 жыл бұрын

Haha thank you so much for the support

@rutweeksawant6567 4 жыл бұрын

very nice explanation and visualization.

@CodeEmporium 4 жыл бұрын

Thanks homie

@chinmayeedongre5525 3 жыл бұрын

Great Video!

@youtubecommenter5122 4 жыл бұрын

What a good video!

@Simon-ed6zc 2 жыл бұрын

Hey, thank you a lot for the explanations! Do you happen to know any heuristics with which to chose a specific optimizer? Right now I have a problem where every paper uses Natural Gradient descent, but when I use it it barely ever converges, while Adam always gets it right (or at least comes close)...

@SirPlotsalot 2 жыл бұрын

Your implementation might not be ideal, I'd try to use a KFAC preconditioning term maybe?

@rakeshsinghrawat99 4 жыл бұрын

Always good

@zshahlaie4740 11 ай бұрын

this video was the bestttttt

@mridulavijendran3062 4 жыл бұрын

Hey. Great work on the video :D It was v clear and fascinating What's NAG? I wonder how come Nadam isn't popular -seems like a better choice. How would you describe RMSProp? You seem to have really great insight into DL concepts :D Also why expectation in particular for Adam parameter updates? Sorry for the questions bombardment. Jus pretty curious

@steffenmlgaardlarsen5463 3 жыл бұрын

NAG = Nested Accelerated Gradient

@crashedbboy 5 ай бұрын

Never thought I would spit out drink while watching machine learning video

@sb7048 3 жыл бұрын

What does the alpha at the SGD momentum equation do? I mean alpha is your learning rat at the first two equations but since then you use n as the learning rate, so for what is alpha since then?

@sizhuanghe1361 Жыл бұрын

That's fun!

@SujayAmberkar 3 жыл бұрын

you voice is like some cool anime main character. I wished i had a voice like yours. Anyway great explanation.

@lucha6262 4 жыл бұрын

really good overview

@CodeEmporium 4 жыл бұрын

Thanks! Making more of this stuff on the channel

@kidsfree6615 Жыл бұрын

Amazing video.

@CodeEmporium Жыл бұрын

Thanks so much!

@hariharans.j5246 4 жыл бұрын

do Neural ODEs and self-supervised learning techniques pls, great video btw

@CodeEmporium 4 жыл бұрын

Thanks. I saw your comment on another video. I'll look into this a bit

@user-wr4yl7tx3w Жыл бұрын

Well explained.

@CodeEmporium Жыл бұрын

Thank youuuu

@nikab1852 3 жыл бұрын

love this

@CodeEmporium 3 жыл бұрын

Love this too!

@RohitashChandra 3 жыл бұрын

well done!

@davisburnside9609 3 жыл бұрын

very helpful, thank you

@CodeEmporium 3 жыл бұрын

Glad it was!

@eyesyt7571 Жыл бұрын

The first scene is precisely what happened to my neural network 2 weeks ago.

@sumod12 2 жыл бұрын

Awesome ❣️❣️

@CodeEmporium 2 жыл бұрын

Thank youu

@ThamizhanDaa1 3 жыл бұрын

Nice channel! better than my professors lol

@CodeEmporium 3 жыл бұрын

Super happy this is helpful. Thanks!

@sia7001 3 жыл бұрын

So underrated 😭

@CodeEmporium 3 жыл бұрын

Right?!

@shaflyhamzah3848 4 жыл бұрын

Nice explanation

@CodeEmporium 4 жыл бұрын

Thanks!

@MrAdhito 2 жыл бұрын

0:43 this really cracks me up HAHA!

@igorg4129 9 ай бұрын

I think that a critical point missing here in the explanation: You have forgotten to mention that a loss surface is different for each sample so there DOES NOT EXIST any universal loss surface for a given dataset and this is a problem in stochastic gradient descent

@karamjeetsinghgulati6989 2 жыл бұрын

I can start laughing for intial 20 sec, I am watching in loop

@arsalan2780 3 жыл бұрын

wonderfulllllllllllllll ...........!!!!

@porimol108Tv 3 жыл бұрын

I subscribed to your channel.

@CodeEmporium 3 жыл бұрын

I appreciate that subscription!

@dan1ar 10 ай бұрын

why gradient on 1:21 would be large? Isn't it just avg for every element in dataset? Same for mini-batch, but only there gradient is avg for every element in batch

@MrSinalta 4 ай бұрын

If I understood well, Acceleration should not be called 'deceleration' in this particular case ?

@tostupidforname 3 жыл бұрын

Imo you should have gone into more detail of the math of the optimizers. I did not understand how the terms relate to the behaviour the optimizers are supposed to have.

@EdeYOlorDSZs Жыл бұрын

W explination

@negardeylami6039 Жыл бұрын

LMFAO the ontro I don't know why it's so funnyyyyyy for me

@EngRiadAlmadani 4 жыл бұрын

Good jop

@Eysh2009 13 күн бұрын

Valeu!

@CodeEmporium 13 күн бұрын

Thanks so much for the donation! Glad you liked this content!

@ArsalJalib 3 жыл бұрын

Loved the start, watched 5 times. Also my first comment on KZfaq. =)

@CodeEmporium 3 жыл бұрын

Yas! Thank for this comment! Absolutely love it

@RichardMuenzer Жыл бұрын

Question how does Newton play here?????

@theoutlet9300 3 жыл бұрын

dude where did you study this. understanding the maths make the coding so much fun

@aakarshrai5833 23 күн бұрын

Bro could you please label you equations. It'll be helpful

@yahavx Жыл бұрын

The first part is not correct: the fact that you use a mini-batch in each step, rather than the entire dataset, does not give you a higher chance to converge to the optimum. Because even when considering the entire dataset in each step, you're still taking the average gradient, so the expected magnitude of the gradient does not change. It all depends on the step size.