Backpropagation calculus | Chapter 4, Deep learning

Рет қаралды 2,730,252

Күн бұрын

Help fund future projects: / 3blue1brown
An equally valuable form of support is to simply share some of the videos.
Special thanks to these supporters: 3b1b.co/nn3-thanks
Written/interactive form of this series: www.3blue1brown.com/topics/ne...
This one is a bit more symbol-heavy, and that's actually the point. The goal here is to represent in somewhat more formal terms the intuition for how backpropagation works in part 3 of the series, hopefully providing some connection between that video and other texts/code that you come across later.
For more on backpropagation:
neuralnetworksanddeeplearning....
github.com/mnielsen/neural-ne...
colah.github.io/posts/2015-08-...
Music by Vincent Rubinetti:
vincerubinetti.bandcamp.com/a...
Thanks to these viewers for their contributions to translations
Hebrew: Omer Tuchfeld
------------------
Video timeline
0:00 - Introduction
0:38 - The Chain Rule in networks
3:56 - Computing relevant derivatives
4:45 - What do the derivatives mean?
5:39 - Sensitivity to weights/biases
6:42 - Layers with additional neurons
9:13 - Recap
------------------
3blue1brown is a channel about animating math, in all senses of the word animate. And you know the drill with KZfaq, if you want to stay posted on new videos, subscribe, and click the bell to receive notifications (if you're into that): 3b1b.co/subscribe
If you are new to this channel and want to see more, a good place to start is this playlist: 3b1b.co/recommended
Various social media stuffs:
Website: www.3blue1brown.com
Twitter: / 3blue1brown
Patreon: / 3blue1brown
Facebook: / 3blue1brown
Reddit: / 3blue1brown

Пікірлер: 2 000

@3blue1brown 6 жыл бұрын

Two things worth adding here: 1) In other resources and in implementations, you'd typically see these formulas in some more compact vectorized form, which carries with it the extra mental burden to parse the Hadamard product and to think through why the transpose of the weight matrix is used, but the underlying substance is all the same. 2) Backpropagation is really one instance of a more general technique called "reverse mode differentiation" to compute derivatives of functions represented in some kind of directed graph form.

@iamunknownperiod3355 6 жыл бұрын

You should probably change the thumbnail. The snapshot of the variables with indices (which I didn't know were indices at the time) and subscripts almost deterred me from watching this although it really wasn't that complicated.

@TalathRhunen 6 жыл бұрын

I will probably be a TA for a lecture course on Deep Neural Networks again next semester and I will recommend this series to the students (we did it in a very math-heavy way this year and it was a bit too much for some of them, even though its a lecture for master students)

@sebaitor 6 жыл бұрын

I was hoping you'd explain in either of these 2 vids on backprop why the hadamard product and transposing are used, what a waste :(

@polychats5990 6 жыл бұрын

Amazing video, I think you did a really good job of making it as easy to understand as possible while also not simplifying things too much.

@bgoggin88 6 жыл бұрын

Sirius Black what you could do is download the r package "deepnet" and poke around at the source code. Its written in base r so you can follow it around. This is how I learned, and IMHO the best way to learn.

@thomasclark8922 Жыл бұрын

This series was my first introduction to Machine Learning 3 years ago. I now work full-time as an AIML Scientist, my life is forever changed. Thank you.

@envadeh Жыл бұрын

how hard was it? I am tryna code my own neural network from scratch, there's so little resources for that it seems. and how do I even make myself unique?

@thomasclark8922 Жыл бұрын

@@envadeh Learn everything, use the feynman technique; if you can't explain how a machine learns to someone who knows nothing about it, keep filling in the gaps. Formal education is great, but honestly more of a waste of time than not. Teach yourself, learn how to learn, and then keep learning. I audited Andrew Ng's Deep Learning Specialization from Coursera, had some formal education, and self taught myself everything I could get my hands on, from theory to application, the underlying math to the practical programming. Understand the importance of data, DATA IS KING. Watch Interviews with industry leaders, understand the big turning points and continued development within the last two decades of AIML (you'll figure out what they are with time). It takes 10,000 hours to become an expert, I'm about 4,500 in, but all it took was a little bit of work every single day. Make learning a habit. Trust yourself, believe in your ability to become who you want to be. "It doesn't matter if your read two research papers in a week, what matters is if you read two research papers a week for a year, now you've read 100 papers" - Andrew Ng (Don't 'read' research papers, watch synopsis! Smarter not harder! There's so much free information, you could probably use a GPT model to teach you what you don't know!) Goodluck, and I believe in you! :)

@nczioox1116 Жыл бұрын

Did you need a CS or math degree to get into the field?

@thomasclark8922 Жыл бұрын

@@nczioox1116 "Need" is a strong word, it just depends on what kind of work you want to do/who your employer is; people used to go to college because that was the only place you could learn these difficult subjects, but now it's just an archaic way of putting you in debt since you can learn these things online for free UNLESS you want to work for an employer where you need the degree to be recognized. If you are self-motivated and can teach yourself these subjects, seriously consider your options before assuming that spending 4 years of your life and 100k+ is necessary. I have an Electrical Engineering degree, but out of the 40+ classes I had to take for it, only 2 had any sort of impact on my daily job now. It all depends on the context. Goodluck, and I believe in you! :)

@nczioox1116 Жыл бұрын

@@thomasclark8922 Thank you! I have a mechanical engineering degree. I'm in the process of self teaching myself machine learning concepts and doing some projects. Lots of job postings I've seen in the field seem to require a bachelors or masters in CS, math, or neuroscience. Of course these seem to be for larger companies so maybe smaller companies might take a more holistic approach

@kslm2687 6 жыл бұрын

“The definition of genius is taking the complex and making it simple.” - Albert Einstein You are genius.

@jean-francoiskener6036 4 жыл бұрын

I thought he said "You don't understand something well until you can explain it in a simple way"

$@fractal5764$

@fractal5764 4 жыл бұрын

That's not the definition of genius

@ericayllon7497 4 жыл бұрын

@@jean-francoiskener6036 yes, it is a quote that appeared in this youtube channel

@Djorgal 4 жыл бұрын

"More quotes are attributed to me than I could possibly have said during my entire life." - Albert Einstein

@shawnjames3242 4 жыл бұрын

@@Djorgal Did he actually say that? \

@cineblazer 2 жыл бұрын

Dear Grant, A year ago, I decided I wanted to learn Machine Learning and how to use it to make cool stuff. I was struggling with some of the concepts, so I went to KZfaq and re-discovered this series on your channel. Out of all the courses I've tried and all the hours of other content I've sat through, your videos stand out like a ray of sunshine. I just got my first full-time job as a Machine Learning Engineer, and I can confidently say it would never have happened without this series. Your channel may have affected the course of my life more than almost any other. Thanks for all your hard work!

@maruferieldelcarmen9573 2 жыл бұрын

You could say that this channel had the largest nudge to your activation value

@cineblazer 2 жыл бұрын

@@maruferieldelcarmen9573 The partial derivative of Grant's videos with respect to my career is off the charts!

@souls.7033 2 жыл бұрын

@@maruferieldelcarmen9573 get out 😂

@souls.7033 2 жыл бұрын

@@cineblazer also i just saw your comment 11months ago, it's amazing to see your development! keep it up!!!

@khai7151 2 жыл бұрын

Congrats on your job. I was wondering, when you finished Andrew Ng’s ML course, what additional steps and how long did you have to take to become a full fledge ML engineer? Thanks in advance

@noahkupinsky1418 4 жыл бұрын

Hey for all of you getting discouraged because you don’t understand this - that was me last year. I went and taught myself derivatives and came back to try again and suddenly I understand everything. It’s such an amazing feeling to see that kind of work pay off. Don’t give up kiddos

@kg3217 2 жыл бұрын

Thanks for the nice words 🙂

@angelbythewings 2 жыл бұрын

studied this 3 years ago in college and it all makes sense to me now

@xbutterguy4x 2 жыл бұрын

Yup. I tried to watch this series a year ago and make my own neural network which turned out to be disappointing. A semester into college and some passion for calculus is all it took for me to mostly understand this series!

@sukhresswarun 2 жыл бұрын

Same here man I seen this video a year ago But now only i understand fully Keep commenting

@oskarwallberg4566 Жыл бұрын

I would say it’s recommended to have read calculus 2 (for partial derivatives and the Jacobian) and linear algebra (for matrix and vector multiplication). Otherwise, just looking up mentioned things is also fine. But it might take time to build up intuition for the math.

@hiqwertyhi 6 жыл бұрын

It's not that no-one else makes top-notch math/cs videos, it's that this guy makes it CLICK.

@ravenn2631 5 жыл бұрын

hiqwertyhi It rivals even the website BetterExplained. People like this teach me how to teach.

@vgdevi5167 Жыл бұрын

Hello, I'm impressed by the way he explained this topic too, but I'm looking for more such great quality resources, youtube channels, books on deep learning, and also math and comp science in general, what do you recommend?

@shofada 6 жыл бұрын

This is how 21st teaching should look like. It feels like your work should be made a "human right". Thank you.

@fakecubed Ай бұрын

No human has the right to another human's labor. That's called slavery.

@Mrrajender2801 4 жыл бұрын

Many guys claim to know. Some guys actually know. But only one guy actually knows and can explain to his grandma as well with very beautiful animations. You are that ONE !!!

@Seff2 3 жыл бұрын

I think my grandma would understand ;D Maybe on a very very high abstract level.

@puppergump4117 2 жыл бұрын

@@Seff2 Python grandma

@SaifUlIslam-db1nu 5 жыл бұрын

It has taken me about 3-4 days worth time to understand all of these 4 lectures, lectures which are in total, no longer than 1 hour and 30 minutes. And I feel proud.

@debajyotimajumder2656 4 жыл бұрын

you should get the t-shirt-merch from 3b1b's description site, the shirt says "pause and ponder"

@danielcampelo2 3 жыл бұрын

Same here. Took my time to hear all explanations. This last video is by far more complex than the previous ones, yet still very well explained.

@polycreativity 3 жыл бұрын

I'm attempting to implement it from scratch in C# with no matrix math library or anything so I can get a feel for the nuts and bolts. This is the boss level!

@polycreativity 3 жыл бұрын

@@berkebayraktar3556 Yeah, I'd love to once I can get it to train properly! So finicky.

@vibaj16 3 жыл бұрын

Daniel McKinnon me too! I’m working on the back propagation, this math is hard

@PhilippeCarphin 6 жыл бұрын

This series is insanely good. As a teacher, I feel like Salieri watching Mozart play and being like "It's so beautiful, how is he so good!"

@stanislawgalas 5 жыл бұрын

As a former mathematician I feel the same way :).

@hansdieter9911 4 жыл бұрын

I like this analogy.

@sashwatdas5482 4 жыл бұрын

This is the first time I have ever liked a comment bc I could not agree more.

@seppl5372 3 жыл бұрын

@Stanisław Galas I don't get why we want the devirative of c in respect to w^L. can you explain pls? It isn't a division right?

@alonsorobots 3 жыл бұрын

Too many notes, I do not understand!!

@suharsh96 6 жыл бұрын

This is the longest 10 minute video I have ever watched. Literally took me half an hour, but the feeling of the idea behind this completely settling in , makes it totally worth it!

@phil.4688 4 жыл бұрын

I think it's easier if you take the time to re-write all of it yourself, on a scratchpad, work through writing the formulas etc. Then you can play with these objects in your mind more fluently. It takes a longer time initially but I feel you get more out of it. And that's a good way to begin "translating" into vector algebra by taking "simple" papers on DL (haven't go there myself yet).

@anjelpatel36 3 жыл бұрын

@@phil.4688 This, is so important. The math really doesnt click unless you write down where each derivative comes from. And the fact that you need more than one partial derivatives for each layer.

@danielcampelo2 3 жыл бұрын

@@phil.4688 Exactly what i had to do. Was getting the concepts, but only really understood once i started taking notes, and perform the calculations by myself (took even more time as had to re-learn derivatives...). The most interesting is that, now that i understood it, i'm even more appreciative the way it's explained in the video.

@arnavrawat9864 3 жыл бұрын

Some memorisation is required. The way to understanding is easily recalling the different pieces and how they fit together.

@kevinsommerfield6341 3 жыл бұрын

I hear you. I rewatched videos 1 and 2 in this series earlier, and will be rewatching videos 3 and 4 later.

@yashjindal9822 11 ай бұрын

I just started out with my ML career. This entire series made me feel as if I knew it all along. Thank you Grant I will return to this comment to share my professional progress😊

@Mayank-lf2ym 4 ай бұрын

Now it's time to return to tell your progress

@azibekk Ай бұрын

Any update?

@rmmaccount 12 күн бұрын

we are waiting

@hutc22222222 Жыл бұрын

Your work of making high levels of math accessible to anyone wishing to learn a variety of new topics is not obvious to me. You succeed to explain everything so clearly, making me want to start learning maths again, reminding me of and introducing me to beautiful aspects of math, and you deserve more than a 'thank you' :)

@SaintKhaled Жыл бұрын

The quality of this education is top-tier. I absolutely am speechless that you make it freely accessible. Thank you so much!

@thiyagutenysen8058 4 жыл бұрын

I came here after Andrew Ng's week 5 in coursera and you blew my mind

@whereismymind1841 4 жыл бұрын

Oh my god, me too

@tuxfighter 4 жыл бұрын

here too :D

@DMDRPBHU 4 жыл бұрын

yeah bcz of corona..bw i am doing it for free

@bryancheong924 4 жыл бұрын

same bro

@masrursakib6528 4 жыл бұрын

Same bruh

@snf303 2 жыл бұрын

At time when I just finished my university - I could not imagine that at one chilly Sunday evening, in almost 15 years after the graduation, I will sit with a bottle of beer, watch math videos, and have so much fun! Thank you!

@antoniobernardo9884 6 жыл бұрын

this is easily the best channel in youtube today! once I get a job i will more than glad to support you!

@utsavprabhakar5072 6 жыл бұрын

Exactly what i was thinking!

@chinmayrath8494 4 жыл бұрын

It has been two years. Have you supported yet??

@pranaysingh3702 4 жыл бұрын

Did you get a job ?

@fitokay 4 жыл бұрын

Two years ago, could you get AI job?

@hozelda 4 жыл бұрын

@@fitokay I think AI won and got his job.

@vedant7090 3 жыл бұрын

Man u deserve a Nobel Prize for teaching Machine Learning with this simplicity.

@vladimirfokow6420 Жыл бұрын

Thank you a lot for this series! It has really helped me get into this topic, and changed my life. Your intuitions have been immensely helpful in my efforts to understand backpropagation. I just can't overestimate, how great your channel is!

@vgdevi5167 Жыл бұрын

Hello, I'm greatly impressed by the way he explained this topic too, but I'm looking for more such great quality resources, youtube channels, books on deep learning, and also math and comp science in general, what do you recommend?

@cineblazer 3 жыл бұрын

I'm taking Machine Learning by Andrew Ng on Coursera right now, and just got stuck on backpropagation. Thank you thank you thank you thank you Grant, you have no idea how incredibly helpful your videos are and how much your channel has inspired me through the years.

@rembautimes8808 2 жыл бұрын

I was in the same position 2 years back . This video does clarify the topic - tremendously

@mrflyswat 2 жыл бұрын

Here I am. Same situation. Andrew Ng course and backpropagation is rough. This video in particular really helped to clear things up. Breaking it down to a single neuron is enormously helpful.

@barditheweird6062 2 жыл бұрын

Same here!) I was somewhat disappointed when Andrew Ng course just through the formulas at me, so I tried to derive backpropagation myself and got stuck in all the little details. Thankfully, 3b1b rode in like a knight in shining armor and now I am really damn happy))))

@mo_l9993 Жыл бұрын

I think the wheel gets repeated with every new comer !

@vgdevi5167 Жыл бұрын

@vectozavr 5 жыл бұрын

That is the reason for learning the math! To understand such a beautiful things! That is awesome! Thank's a lot!!!

@anthead7405 3 жыл бұрын

Math on his own is also the reason for learning math.

@vvii3250 3 жыл бұрын

Интересно повстречать тебя тут. :)

@owaisfarooqui6485 3 жыл бұрын

and when I asked my math teacher, that person told me you need this to pass the test. that didn't make a lot of sense back then

@krenciak 3 жыл бұрын

@@vvii3250 Ага, чувствую себя как в своеобразном мини-клубе, где собралась небольшая компашка и тусуется))

@bocik2854 3 жыл бұрын

@@krenciak Ыыыыыы

@borg286 6 жыл бұрын

The point where you addressed the concern that the example you were using was too simple, having only 1 edge, was spot on as you were leading me down this merry garden path. I appreciate how much you watch your own videos and predict where the watcher would mentally say, "but what about..."

@altaypbank9036 3 жыл бұрын

0;090505452 0145

@alexdebate7081 8 ай бұрын

Grant, I've come back to this series many times over the last five years. Every time I do, I pick up more and more pieces of the puzzle. I think I've finally got it now, but we shall see! Thank you!

@matthewhaythornthwaite754 3 жыл бұрын

If anyone is interested, I worked through the chain rule for the differential of the cost function w.r.t the weight in the second layer down. Two additional terms are added to make everything cancel as they should. It shows how as you progress down the layers, more partial differentials are added to the equation from all the variables above, making it more unstable and hence more susceptible to the exploding or vanishing gradient problem. dC/dw(L-1) = dz(L-1)/dw(L-1) * da(L-1)/dz(L-1) * dz(L)/da(L-1) * da(L)/dz(L) * dC/da(L)

@galileofrank5779 3 жыл бұрын

what's the vanishing gradient problem?

@MrJmayz 3 жыл бұрын

i'm looking for this in the video. appreciate if you could share your work

@sergiokorochinsky49 6 жыл бұрын

I just unsubscribed to this channel, so I can have the pleasure of subscribing again.

@pranavnigam11 3 жыл бұрын

I just unliked your comment... So I can have the pleasure of liking it again

@ifusubtomepewdiepiewillgiv1569 3 жыл бұрын

i just liked and then unliked ur comment bc i realized it was at 69

@etsagcoenagpur1200 3 жыл бұрын

@@ifusubtomepewdiepiewillgiv1569 nice

@neillunavat 3 жыл бұрын

"Not gonna lie, you got me in the first half"

@neillunavat 3 жыл бұрын

@@ifusubtomepewdiepiewillgiv1569 .

@Ensorcle 6 жыл бұрын

I cannot tell you how much I appreciate these videos. I don't have a strong math background (english undergrad) but I'm teaching myself data science. It REALLY helps to have the equations explained rather than just presented and to tie the components of the equation back to intuitions. Thank you thank you thank you.

@nairanvac79 5 жыл бұрын

Thank you for starting your indices at 0.

@elizfikret7489 Жыл бұрын

Thank you so much! I have understood more math from this channel than from all teachers I have had in high school or university in total.

@vgdevi5167 Жыл бұрын

@jaysoaring6318 6 жыл бұрын

If there is an award for educational video series on advanced scientific matters. Please give this award to 3b1b. Love it!

@saptarshimitra1267 6 жыл бұрын

Amazing man..... I say 3gold1platinum

@General12th 6 жыл бұрын

More like 3plutonium1antimatter

@TwentySeventhLetter 6 жыл бұрын

Yeah, cause he nukes the brains of laymen like myself.

@General12th 5 жыл бұрын

@@ishwarkarthik327 It's not.

@bradleydennis210 4 жыл бұрын

I just finished up calc iii this semester and I have never felt happier with myself for being able to apply my new knowledge than this episode. I also don't think I have ever been more excited to hear calc iii topics being brought up in a field I am trying to teach myself currently. Thank you for making such a simple to understand series!

@mooglefan 4 жыл бұрын

I've worked with AI for 2 years now. I have never seen anyone explain this as succinctly and aptly as you have. This video is legitimate gold. Going to show this to anyone who needs an explanation in future!

@kangChihLun 6 жыл бұрын

This is the best and clearest explanation in all BP course I could find ! 沒有之一！

@songlinyang8614 5 жыл бұрын

没有之一，haha

@OGLOCK69 5 жыл бұрын

Are you going to use this knowledge to help develop the social credit system?

@dysonsun3306 5 жыл бұрын

kang Chih Lun 哈哈哈哈哈

@shengchuangfeng227 5 жыл бұрын

Good! 沒有之一！

@user-jf5bw7mf9y 5 жыл бұрын

you are right! 没有之一，哈哈哈

@AnshulKanakia 6 жыл бұрын

I can't tell you how long I've been trying to visualize all this in my head to get a solid mental picture of backpropagation... Well, I guess I can - it was the duration of a flight from Frankfurt to Seattle (about 9 hours) and it involved one terribly lit backside of an airplane menu and a shitty pencil. I am so grateful for the work you put into animating this algorithm. It has literally brought tears to my eyes and a smile on my face. Thank you.

@meeradad 10 ай бұрын

These videos are the best ways to make a high schooler fall in love with calculus instead of hating it or fearing it. And open his/her mind to the joy of creativity rooted in mathematical insights.

@sainandandesetti3268 4 жыл бұрын

Stunningly beautiful... The best part of the series (for me, obviously) is that the beauty of this series does NOT make it very easy to understand. No. Each video may need multiple views. But these videos are so beautifully made that you'd want to watch them again and again, not with the frustration of getting your head over a concept but with the thrill of unravelling a mystery... So for creating such excitement in me, thank you.

@Kevin-cy2dr 4 жыл бұрын

Honestly this channel doesn't deserve a dislike button. It took me days to figure out one video(at the beginning),but the concepts remain still in my head. This channel taught us that maths is not just changing numbers, but its conceptual and intuitive just like science. Grant if you are ever read this, please know that you are one of the very few people that change the world. I just dont have words for you man, great job is an understatement for you. I promise once i earn enough i will contribute to your channel

@thfreakinacage 6 жыл бұрын

My god! A basic machine learning video series that actually makes sense to completely beginners! Subscribed, and waiting in great anticipation for the next one! :D

@rohitdatla724 4 жыл бұрын

u r not just teaching NN concept but how to think, break down and understand any complex problem and digest, U R AWESOME!!!!!

@homieboi5352 Ай бұрын

I’m gonna need to rewatch this a few times to grasp it all, but wow, what a thorough explanation of back propagation! I adore how you referenced the entire equation earlier in the series and it made no sense, but now you’ve broken it down entirely. Phenomenal work!

@samuelreed5481 5 жыл бұрын

These videos are unbelievably well produced. Thank you so much for your effort. You've made this topic incredibly clear and I cannot understate how much I appreciate the amount of effort you put into these. You have incredible talent as a teacher.

@Jabrils 6 жыл бұрын

youre a deity Grant

@Jabrils 6 жыл бұрын

Haha, why hello Bill. Nice to find you on 3B1B's channel :D

@bevel1702 5 жыл бұрын

wut

@anjelpatel36 3 жыл бұрын

wut

@gumbo64 3 жыл бұрын

wut

@idr7789 3 жыл бұрын

wut

@grigorioschatziandreou2558 5 ай бұрын

THANK YOU - MSc student here. Taken a module on machine learning. My university is world class and so the module was very well taught, so I had already good knowledge of neural networks. But now, I am doing research and need to dive deep into them and realised how much I lack deep understanding. This has helped A LOT.

@drstrangeluv1680 2 ай бұрын

Grant I was able to go through my entire PhD at University of Maryland by watching and learning Math from your videos. Now I am trying to get a full time position as an ML engineer and I come back to your channel again for understanding. I hope you know you are changing lives- you certainly changed mine. Thank you!

@Redrumy0 5 жыл бұрын

Literally the only youtube channel, that makes studying 2 hours of math, go by in a blink of an eye

@LimitedWard 4 жыл бұрын

Absolutely brilliant explanation! I took a course on deep learning in college, but ended up auditing it in the end because I couldn't grasp the concepts well enough to pass the tests. You just took the entire first unit of the course, which took several weeks, and condensed it into 4 easily digestible videos that anyone can understand!

@jamesjin1668 3 жыл бұрын

All of the animation you used are so simple yet at the same time so illuminating. I bet people would appreciate your videos even more when at the end of the video you zoom out to show all of your visual aid in a nicely summarized flow chart / spatial diagram.

@colorlace 3 жыл бұрын

Best explanation I've ever seen of the chain rule in a video that isn't about the chain rule

@cineblazer 3 жыл бұрын

Ok but SAME like I genuinely didn't get the chain rule until this video and now it's like "wait -- that's it? It's that... simple?!" THIS IS WHY GRANT IS THE ABSOLUTE GOAT.

@patelnirmal4726 6 жыл бұрын

Awesome channel

@samarthsingla1082 4 жыл бұрын

The amount of help you are providing is nothing short of amazing.

@tekashisun585 25 күн бұрын

Learned ML in an intro to AI course offered in my university, it’s the content of the last couple of weeks. Lots of details are left out, so this series has been putting things into perspective for me. Thanks

@bean217 2 жыл бұрын

I am currently going through Michael Nielson's "Neural Networks and Deep Learning" book. This video helps to clear up and visualize the chapter on back propagation a lot. Thank you for making this video series.

@giron716 6 жыл бұрын

I seriously have a hard time explaining how much I appreciate this video. I am far and away a symbolic thinker, as opposed to a geometric one, and while I love all of your videos and how intuitive you make the concepts, it's sometimes hard for me to think about the geometry. I am much more comfortable working with symbols and that's why I treasure videos like this. Thank you :)

@Erioch 6 жыл бұрын

Honestly, this is one of the best (If not the best) channel on Mathematics/Science education I have seen. Intuitive but not oversimplified. Thank you so much that for offering your spectacular work and you help so many people understand these concepts.

@steelcitysi 3 жыл бұрын

This was the best animated description I've come across. I hope you can continue on this topic more. Especially interested in the jump to CNNs, and the intuition for the effects of changing the number of layers and number of nodes in the hidden layers.

@catchingphotons 3 жыл бұрын

Unarguably one of the best "tutorial" videos of all times! The carefully taken logical steps of understanding, the animations, the visualizations, the tempo, the examples... boggles my mind! This is a masterpiece! Greetings -Chris

@micahsheller101 6 жыл бұрын

Beautiful work! Reminds me of my late father who was a math professor: he had the same gentle, happy style, and believed heartily in making math a safe place for everyone to learn and have fun. Gonna make me tear up :)

@zilongzhao3274 3 жыл бұрын

your video should be shown in every university's lesson, the animation makes the calculation just so easy to understand.

@bigbluetunafish4997 6 ай бұрын

Finally I finished these 4 chapters of neural networks, and some of your linear algebra and calculus stuff. I feel much better that now I have deeper understanding of how neural network works and have built up that base for further exploration of machine learning. Thanks very much for your effort creating all these great videos together.

@michaelarmstrong2251 9 ай бұрын

Just watch this series of videos. I'm a mechanical engineer with no prior experience of machine learning - now I feel like I understand quite a few concepts that were hard to wrap my head around when learning from other sources. Absolutely awesome videos - well done!

@sjgmc 6 жыл бұрын

As an hobbyist programmer, i can't thank you enough! Once i finish my studies i will donate to you. :)

@13thxenos 6 жыл бұрын

Nicely done video. I knew I learned backpropagation before, but it was hard, and I didn't use it manually ( I used frameworks like TensorFlow which uses computational graphs and backpropagate automatically) so I've forgotten how it actually worked. But this video is a great resource for newcomers to ANNs and people like me that have forgotten the theory behind it all. Thank you.

@ganjarulez009 2 жыл бұрын

I don't understand why the books cant explain this stuff this easy yet precise, great job!

@andrewvalenzuela1790 Жыл бұрын

I despised math books. It always felt as if the examples were not "showing their work"

@ashkankiafard4493 2 жыл бұрын

The fact that I can understand what you're talking about shows that your teaching is flawless!

@GaborGyebnar 6 жыл бұрын

Awesome material. :) Please make a video about convolutional neural networks, too.

@MeriaDuck 6 жыл бұрын

After seeing a few pieces of books, descriptions on the internet about back propagation, with this video I finally reached some kind of enlightenment (especially at about 4:50 into this video). Thank you so much for that! Just as a hobby, I was trying to implement a neural network from scratch in java: plain objects for neurons, connections and layers. I wanted to really visualize how a neural network WORKS. (Visualize either as computer code, but maybe I even want to create some visual for it...) This will certainly help me on my way!

@austinellis-mohr2189 4 жыл бұрын

I am continually impressed by your videos. Your essence of linear algebra lit up my imagination which such a geometric interpretation of the subject, and I have used your general methods many times in teaching others (both calculus and linear algebra). This is, in my opinion, the most beautiful series yet as it concisely describes a sort of mystified topic. The analogies you draw and even the notation you use is clear, informative, and friendly. I just want to say thank you and let you know that even outside of people watching your videos, your imagination and passion for teaching affects a lot of people’s learning in a variety of subjects. Again, just thank you and keep being a great teacher for literally millions.

@PatryczakPL Жыл бұрын

I really can't believe how well and simply this is explained here by you, I still keep coming back to this exact video even after few years of being in AI to refresh the fundamentals and to this day this is the best resource for this.

@kimtheass1 6 жыл бұрын

You videos have such informative and well explained content with an amazingly calm and including tone! I'm a fan

@prashamsht 4 жыл бұрын

One of the best lectures I have ever heard. Great explanation of NN, cost functions, activation functions etc. Now I understand NN far far better...(P.S. I saw previous videos Part 1, 2,3 as well)

@n9537 3 жыл бұрын

This 10 min video is pure gold. Lays down the math in an easy to understand, intuitive manner.

@Arthur-fz5dw 3 жыл бұрын

This NN series was amazing, thank you so much! I've watched a lot of these kinds of videos, read several online articles, and am following a popular online course. This series is the best resource by far, so helpful.

@sacation6057 5 жыл бұрын

Awesome series! Even though i already had quite a intuitive feeling about the concepts of Deep learning, your videos just always make complex subjects click in my mind, it sort of forms the right connections between the neurons in my mind i suppose so ;) Even without any advanced math knowledge i was able to follow your math, so thanks for choosing to keep your examples as simple as possible! I'm gonna make my own network from scratch in code some time, to see if i truly understand it throughly.

@ehsanmon 6 жыл бұрын

Thank you so much, Grant. I finally learned back prop, and I have become a patron. I wish I could do more.

@joecjaffe Жыл бұрын

LOVED these four introductory videos. My brain was so exhausted processing them--lots of pausing--but I feel like I get something conceptually that I knew near nothing about a couple days ago. Great foundation to scaffold the programming and math onto. Thank you! will definitely be watching more of your videos. Looking forward

@ciherrera 4 жыл бұрын

Wow, this took a long time to get my head around fully, but I was finally able to understand it enough to implement my own version of backpropagation from scratch thanks to this video! Neural networks are something I've wanted to get into for a while and I'm really grateful for these wonderful in-depth explanations!

@notbobbobby 6 жыл бұрын

Right now, I am so thankful for having taken vector calculus and several numerical approximation courses. This was an AWESOME video to watch. Thanks! :)

@lagduck2209 6 жыл бұрын

*looking at thumbnail oh sh~, Im never going to understand that complex stuff. probably should watch it anyway *ten minutes later whoa! that's actually quite clear now!

@kairostimeYT 6 жыл бұрын

same for me.

@alex-gb9iy 6 жыл бұрын

Тоже учишь нейросети?

@lagduck2209 6 жыл бұрын

просто люблю математику и познавательные видео на эту тему) особенно нейросети

@sage5296 6 жыл бұрын

I think that’s kinda the point of much of this channel lol. Same experience though! It’s amazing how a KZfaq video can do so much teaching in like 10 minutes

@tommysawyer9680 6 жыл бұрын

I had the same reaction. Beautifully broken down into calculus us mere mortals can understand!

@vivekpujaravp Жыл бұрын

Your insight and eloquence are exquisite. You are quite possibly the greatest educator to ever exist. Thank you for everything, please keep making more. I will continue studying and appreciating your brilliant work.

@gibemass6578 5 жыл бұрын

You are some sort of sorcerer. The chain rule was finally elucidated for me in about 20 seconds. So concise and intuitive indeed. That was one thing, for whatever reason, I couldn't get my head around as a younger student. Big props man. Love this channel.

@kirilllosik7054 Жыл бұрын

Thanks a lot for creating such a fantastic content! Anticipating to see more videos about AI, ML, Deep Learning!

@tcc1234 5 жыл бұрын

I saw a 6.5 minute ad (related to AI) in order to support you. Keep up!

@hunteranderson5573 10 ай бұрын

This is hands down the best resource I have seen on the calculus of backprop. I have called back on my knowledge gained from this video many times, it has been invaluable in my understanding of the current machine learning landscape. Excellent work with the notation too. Incredibly valuable video put together perfectly, good work 3b1b!

@JinruWu 6 ай бұрын

This is back propagation so brilliantly explained, I can now feel how efficient this way of calculation is and how it enabled computation not in the past possible. Teaching a Deep Learning course next week and have been scratching my head understanding this concept - now I'm confident I will be able to explain to my students with clarity. Thank you so much! You have touched my life as well as theirs :)

@qwert-cj4ld 4 жыл бұрын

9:28 sums up the whole thing

@Abstruct 6 жыл бұрын

This stuff is an amazing supplication to Andrew Ng's courses, it gives a lot more intuition and visual understandings of the formulas.

@claytonharting9899 6 жыл бұрын

It certainly is a huge help for backprop. Just the tree visual is a huge help. Btw, what do you think of 3b1b’s use of a bias value vs Ng’s use of a weighted bias node? I think 3b1b’s may be more clear, but the node version is more computationally efficient.

@Viplexify 6 жыл бұрын

... in which Ng mentioned that he still doesn't fully understands backprop. I wondered if it was true or just a consolation for beginners.

@ab452 6 жыл бұрын

Consolation, it is just to sooth your frustration. But he can also be referring that you can understand how to compute it for a simple case ,but it in a large instance you simple lose track of it. Without a computer is would be a hopeless task.

@tanmaybhayani 4 жыл бұрын

andrew should link this series in his course, cos this is just beautiful!

@hayden.A0 4 жыл бұрын

I'm actually here in between Andrew Ng's course on machine learning. there were a few concepts I didn't completely understand but they are quite clear now.

@zafar0132 2 жыл бұрын

I don't know how you do it 3Blue1Brown.... but give YOUR self a pat on the back... with only a single view of the 1st 3 and two views of the 4th. I actually understand how neural nets work. congratulations to you for making a complex subject seem so easy!

@Mizar88 3 жыл бұрын

I am speechless. This channel undoubtedly contains the best pedagogical scientific material on KZfaq, and possibly in the world. Thanks for making these videos, your skills and passion are unreachable!

@aravindkannan9490 6 жыл бұрын

This is by far the best video I have ever seen in Neural Networks. Thanks for this! :)

@tisajokt7676 6 жыл бұрын

I also suggest the video series on neural networks by Welch Labs, or if you've seen it already, I'd be interested to hear your comparison between it and 3Blue1Brown's series.

@aravindkannan9490 6 жыл бұрын

Just completed their playlist! equally good :) I like the application-oriented explaination However, I would still recommend 3B1B for an absolute beginner because of the in-depth explanation and for the help in visualizing the math behind it

@thomasschwarz1973 Жыл бұрын

This is truly awesome, as pedegogy and as math and as programming and as machine learning. Thank you! ...one comment about layers, key in your presentation is the one-neuron-per-layer, four layers. And key in the whole idea of the greater description of the ratio of cost-to-weight/cost-to-bias analysis, is your L notation (layer) and L - 1 notation. Problem, your right most neuron is output layer or "y" in your notation. So one clean up in the desction is to make some decisions: the right most layer is Y the output (no L value), because C[0]/A[L] equals 2(A[L] - y). So the right most three neurons, from right to left, should be Y (output), then L then L minus one, then all the math works. Yes?

@kazenokize2 3 жыл бұрын

I just watched this video going back and forth several times and I am at a point where I am not sure if it made sense or not, but I still feel like it helped. Thank you so much for this series!

@ssachdev1 2 жыл бұрын

"So pat yourself on the back! If all of this makes sense, you have now looked deep into the heart of backpropagation, the work horse behind how neural networks learn." felt soooo goooooood

@cowcannon8883 6 жыл бұрын

Neural networks have layers, ogres have layers Shrek is an AI confirmed

@icegod4849 5 жыл бұрын

Goddamn nice reference would like it a thousand times over if I could

@minecraftermad 5 жыл бұрын

Shrek is our AI overlord

@Ammothief41 5 жыл бұрын

My AI overlord has decided they're both onions.

@inlandish 4 жыл бұрын

~onions have layers too~

@shiveshramlall2809 4 жыл бұрын

Big. Brain.

@bilalsedef9545 2 жыл бұрын

This is a great and very educational video. But I think it needs one more part to show how the weights are updated.

@yonitlev5720 3 жыл бұрын

thank you SO MUCH for publishing this video! The illustrations, explanation and your calm voice made it so easy to understand! I first hear of backprop yesterday and the professor didn't go into the math of the cost function derivative one step back from the output layer and your explanation really helped me get a better sense of understaning what's happening here. I can not thank you enough - bless you!

@dteja92 5 жыл бұрын

God Bless You man! I have tried watching many videos about backpropagation but this series made my conceptual understanding and intuition super clear. Thanks a lot. You have no idea how happy I am right now to have understood the concept of backpropagation.

@securemax 9 ай бұрын

For everyone wanting to implement backprop from scratch: don't use dC/da = 2*(a-y). Instead use dC/da = a-y. This is because the cost function would actually be defined with a factor 1/2 in front which is missing here. Hence, the derivative changes. All other derivatives are good :)

@carloscortes2391 5 ай бұрын

Why would there be a 1/2 factor?, to average the square error since there are 2 outputs?

@AbhishekKumar-bo1yi 6 жыл бұрын

I always feel, if u have a mentor who can break complex things into simple stuff so beautifully, even a dumb guy can grasp the concept. Keep doing the good stuff. Admirer++

@classicalfandom8219 Жыл бұрын

Dear Grant, Thank you for all the hard word and effort you put in these videos and not to mention the resources you link us with while also making these lectures available in written form on your website. I have completed this series and looking forward to apply the knowledge gained from you and the resources you linked to, to the betterment as much as I can. Regards, Your Subscriber

@martinmartin6300 4 жыл бұрын

Great explanation! I like how you really focus on giving why the formulas actually help and to make this stuff as intuitive as possible. This is the best inteoduction on backpropagation that I have seen so far!