Shouldn't be Gamma should approximate to the true variance of the neuron activation and beta should approximate to the true mean of the neuron activation? I am just confused...

Exactly what I was looking for. Very well researched and explained in a simply way with visualizations. Thank you very much!

Just found your channel and binged through all your videos so heres a general review. As a student i assure you your content is on point and goes in depth unlike other channels that just skim the surface. Keep it up and dont be afraid to go more in depth on concepts. We love it. Keep it up brother you have earned a supporter till your channels end

This is good! I think that giving an example as well as the use cases (advantages) before diving into the details alwayd gets the job done

Love your videos. They help me alot understanding machine learning more and more

The loss is not a function of the features but a function of the weights

I clearly understand the need of batch normalization and its advantages! Thanks !!

Hey man, thank you. I really appreciate this quality content!

thats actually a very very good and intuitive video. Honestly Thank you

You are really and also really good because you reference paper and introduce the idea

Awesome video. I really like, how you go into the actual papers behind it.

why did you plot the cost against height and the age isnt it supposed to be a function of weights in a neural network

This is beautiful. Thank you

Amazing work really liked it

Great explanation, thanks!

Just what I was looking for, thanks.

Great explanation thank you!

thanks, this video helped me understand BN better. and I liked your sense of humor. made watching is more fun.:)

Can you please make a video on how batch normalization and l1/l2 regularization interact with each other ?

Awesome explanation.

Quick note: at 6:50 there should be brackets after 1/3 (see below) Yours: 1/3 (4 - 5.33)^2 + (5 - 5.33)^2 + (7 - 5.33)^2

Hey there . Im on my way to become data scientist , and your videos help me a lot ! Keep going Im sure I am not the only one you inspired :) thank you !!

at 3:27, I think your axises should be the *weight* for the height feature and the *weight* for the age feature if that is a contour plot of the cost function

Awesome thank you!

Hey! Just cited you in my undergrad project because your example finally made me understand batch norm. Thanks a lot!

Subtracting the mean and dividing by std is standardization. Normalization is when you change the range of the dataset to be [0,1].

The best I have seen so far

Thanks for awesome video Get many idea about Batch Norm

Thank you!

Thank you bro.

Good video, could you please make a video on capsule network.

Thanks, Man!

I see you are checking all these comments - so will try to comment on all the videos I see going forward and how I’m using these videos. Currently using this video as supplement to Andrej Karpathy’s makemore series pt 3. The other video has a more detailed implementation of batch normalization but you do a great job of summarizing the key concepts. I hope one day you and Andrej can create a video together 😊.

I found this a really good and intuitive explanation, thanks for that. But there was one thing that confused me: isn't the effect of batch normalization the smoothing of the loss function? I found it difficult to associate the loss function directly to the graph shown at 2:50.

Bro you have made the perfect video

Do we calculate the mean and SD across a mini-batch for a given neutron or across all the neurone in a layer? Andrew NG says it's across each layer. Thanks.

I wonder is it suitable to use population estimator? I think nowadays most of the machine learning learner/student/fans spent very less time on statistics after several year study I find that The model selection and the statistical theory take the most important part especially the Bayesian learning the most underrated topic today

Is not Batch Normalization also used to counter the exploding gradient problem? Would have loved some explanation on that too..

Fantastic explanation using contour plots.

great explanation

this is a great video

wow, easy to understand , and clear accent. Thank you, sir. u done a great job

This was so well put together- why less than 10k views? Oh... it's batch normalization

Hello, thank you for your video. I am wondering regarding the batch normalisation speeding up the training: you showed at 2:42 the contour plot of the loss as a function of height and age. However, the loss function contours should be plotted against the weights (the optimization is performed in the weights' space, and not the input space). In other words, why did you base your argument on the loss function with weight and and height being the variable (they should be held constant during optimization)? Thank you! Lana

thanks

Very helpfull!!

Thanks for the video. I wanted to add that there's debate in the community over whether to normalize pre vs. post non-linearity within the layers, i.e., for a given neuron in some layer, do you normalize the result of the linear function that gets piped through non-linearity or do you pipe the linear combination through non-linearity and then apply normalization, in both cases, over the mini-batch.

the real magic starts at 6.07, this video was exactly what I needed

wow, you explained pretty clearly

How can we use batch normalization in the test set?

Thanks!

Hello, I wonder what if we don't normalize the image input data (RGB 0-255) and then we use batch normalization? Is it going to work smoothly? or is it going to mess up with the learning?

Thanks buddy

@CodeEmporium , could you please tell me that why do we need to normalize the outputs of activation function whe they are already within a small range(example sigmoid ranges from 0 to 1)? and if we do normalize them, then how do we compute and updates of its parameters during backpropgation? please answer.

grate video bro keep it up

can I add a Batch Normalization layer after an LSTM layer in pytorch?

Sorry, but where did gamma and beta come from and how is it used?

Nice video, but i guess there is a little error at 6:57? I guess you have to multiply the whole with 1/3 not only the first term

Also if someone can say how to make gamma and beta learnable? gamma can be thought as an additional weight attached to the activation but how about beta? how to train that?

what about layer norm ?

thansk

Great explanation, very helpful!

Nice content

Wasn't there an episode where Peter was playing against Larry Bird?

I do not understand property how this Is going to be implemented. At the end of an epoch actually we perform those operations right? At the end of that epoch, at this point the layer where i have applied It Is normalized right?

Why to use beta and gamma?

Nice explanation :+1

Very good explanation. However, there is something I didn't understand. Doesn't batch normalisation modify the inout data so that m=0 and v=1 as explained in the beginning ?? So how the heck we moved from normalisation being applied on inputs, to normalisation affecting activation function ? 😅😅

Nice animations

can I get these slides?

Can u make a video for all types pooling layers

Hey brother can you please tell me how on fly data augmentation increase the image data set every on blogs and vedios they said it increase the data size but hiw

This is good for ghost box

Awesome, keep going like this

Great explanation, but missing parantheses at 6:52 :p

Good video here's a comment for the algorithm

The only difficult part of batch norm, namely the back prop isn't explained.

Good explanation until 7:17 after which, I think, you miss the point which makes the whole thing very confusing. You say: "Gamma should approximate to the true mean of the neuron activation and beta should approximate to the true variance of the neuron activation." Apart from the fact that this should be the other way around, as you acknowledge in the comments, you don't say what you mean by "true mean" and "true variance". I learned from Andrew Ng's video (kzfaq.info/get/bejne/qrR5o6iLsdzDlZs.html) that the actual reason for introducing two learnable parameters is that you actually don't necessarily want all batch data to be normalized to mean 0 and variance 1. Instead, shifting and scaling all normalized data at one neuron to obtain a different mean (beta) and variance (gamma) might be advantageous in order to exploit the non-linearity of your activation functions. Please don't skip over important parts like this one with sloppy explanations in future videos. This gives people the impression that they understand what's going on, when they actually don't.

BatchNorm reparametrizes the underlying optimization problem to make it more stable (in the sense of loss Lipschitzness) and smooth (in the sense of “effective” β-smoothness of the loss). Not my words

good visualization

And why not just normalizing the whole training set instead of batch normalization?

fuck me, you explained way better than my prof did

Hey 🅱eter, did you make it to the NBA?

🤯

hey can u link the graph that you used please?

For RNN's Batch Normalisation should be avoided, use Layer Normalisation instead

why didnt peter make it..

Your visualizations are misleading. Normalization doesn't turn the shape on the left into the circle seen on the right. It will be less elongated but still keep a diagonal ellipse shape.

Peter is most definitely not 1.9m That’s 6’3

Sorry to say but very poor video. Intro was way too long and explaining more the math and why BN works was left for 1-2mins.

55555 you get it :) HaHa