Your saddle point animation took two seconds to illustrate why SGD might outperform vanilla GD. Amazing

very nice animations, and well explained. But just to be a bit technical isn't what you described called "mini-batch gradient descent". Because for stochastic gradient descent don't we just use one training example per iteration?? 😅😅

2:22 and after is just magical. Thanks for the amazing video.

Awesome explanation, better than many books

Didnt expect to see Dota gameplay lol. Very useful video btw

I love watching these videos when I just need a short refresher. Great content.

Brilliantly Genius!

I enjoy slow and sloppy the most.

Thabks a lot, this great short video closed that corner for me

These are very nice visualizations, and a great explanation of the fundamental idea of SGD. But I'm very skeptical of some of the intuitive seeming explanations of why it's better than regular gradient descent. In particular, the saddle point example seems extremely constructed. It works with simple R²->R functions like the one we see, but even there only if the starting point (=model weights) are placed perfectly on the line, the probability of which is basically 0. Given model weights are usually initialized randomly, and we're in R^n space with n >> 2, I doubt that such cases ever happen in actual deep learning. Secondly, of course you can argue that SGD due to its noisiness may better escape local minima. But 1) do local minima actually exist in these extremely high dimensional spaces? If you have a billion dimensions, it's exceedingly unlikely that the derivative of all of them is 0 at the same time, and it may be almost impossible to run into them with a discrete approach. I think all these R²->R visualizations build some very strong yet incorrect intuitions about what high-dimensoinal gradient descent actually looks like. And 2) it could via the same process also _miss_ a _global_ minimum that regular GD would find (or avoids a local minimum but never makes it to any point that's better than the local minimum that GD would have found - so avoiding it then was a _bad_ thing). Noise is not inherently good. We can construct specific examples where it happens to help, but we could just as well find many examples where GD would win against SGD, and in the end the reality is probably that noise in SGD _hurts_ less than the gained performance helps, meaning overall it's the better option. It kind of makes sense: evaluating all your training samples to compute the gradient has diminishing returns. So the first, say, 10% of samples are much more important than the last 10%. But if you always used the same 10% of samples, you of course lose a lot of information overall. And if you choose some systematic process of how to select them, you might get strange biases. So naturally you pick them randomly. And hence, SGD.

If the outcome of the SGD step is random, do you think it could be done multiple times and we could chose the best step?

Watching these videos gives Way better enjoyment than memes.

TinyGPT uses adam (an SGD variant) with a small batchsize for pre-training warmup, and large batchsize for fine-tuning

Such a great work. May i ask which software did you use for animation?

It's like being less restrictive keeps you from optimizing the wrong thing and getting stuck in the wrong valley (or hill for evolution). Feels a lot like how i kept trying to optimize being a nice guy bc there was some positive responses and without some chaos i never would have seen another valley of being a bad boy which has much less cost and better results

Finds better solutions? Isn't it just offering faster convergence?

Great channel. Thanks!

amazing explaination

nice explanation

I'm trying to program my own neural network, but there training algorithms can't get in my head. Thanx.

good one

You're awesome, subbed!

Sorry, I do not get something What do you randomly take random obsevations? Or random set of features? Or random amount of weights (=random amount of neurons)

Perfect

stochastic gradient descent doesn't take some constant number of terms, it takes one training example at random, then performs the feed forward and back propagation with that one training example.

You are insane with those animations, please tell me which software you use for doing that

how to use emoji in manim

Hello, I don't understand the meaning of 2:38 (the formula can be understood). Can you explain more?

持續支持行政程序中

summizaitok

hdj😂