I cannot express the amount of appreciation enough of your videos, especially NLP deep learning related topics! They are extremely helpful and so easy to understand from scratch! Thank you very much!

@ 40:00 why we consider 64? - It is based on the how many multi head attention you want to apply. We used embedding size for each word = 512 and want to apply 8 multi head self attention; there fore for each attention we are using (512/8 =) 64 dimensional Q, K, V vector. So that, when we concatenate all the multi attention heads afterward, we can achieve the same 512 dimensional word embeddings which will be the input to the feed forward layer. Now, for instance, if you want 16 multi attention head, in that case you can use 32 dimensional Q, K, and V vector. My opinion is that, the initial word embedding size and the number of multi attention head are the hyperparameters.

I am very new to the world of AI. I was looking for easy videos to teach me about the different models. I cannot imagine that I was totally enthralled by this video as long as you taught. You are a very good teacher. Thank you for publishing this video free. Thanks to Jay as well for simplifying such complex topic.

Thanks for your fantastic LLM/Transformer series content, and I admire your positive attitude and support for the authors of these wonderful articles! 👏

You are a really good teacher that always check your audiences weather they get the concept or not. Also, I appreciate your patience and the way you try to rephrase to have a better explanations.

Krish, I really see the honesty in you man, lot of humility, very humble person. In the beginning of this video, you gave credit to Jay several times who created amazing blog for Transformers. I really liked that. Be like that.

For anyone having a doubt at 40:00 as to why we have taken a square root of 64 is because, as per the research it was mathematically proven to be the best method to keep the gradients stable! Also, note that the value 64, which is the size of the Query, Keys and Values vectors, is in itself a hyperparameter which was found to be working the best. Hope this helps.

Krish is a hard working person, not for himself but for our country in the best way he could...We need more persons like him in our country

Million tons appreciation for making this video. Thank you soo much for your amazing work.

You can skim through all the youtube videos explaining transformers, but nobody comes close to this video. Thank you Sir🙏🙏🙏

I really admire you now. Just because you give the credit to the deserving at the beginning of the video. That attitude will make you a great leader. All the best!!

How did I miss the subscription to your channel? Thank you so much for this thorough explanation, and hats off to Jay Alammar.

thank you, appreciate your time going through this material

Very well covered GPT-3 topic. Very important from NLP point of view. Thank you for your efforts.

Sir, Please release the video of Bert. Eagerly waiting for it.

This might help the guy who asked why we take the square root and also for other aspirants : The scores get scaled down by getting divided by the square root of the dimension of query and key. This is to allow for more stable gradients, as multiplying values can have exploding effects.

Yes this is the best video explaining these Models so far even non computer science people can understand what is happening , great work

Excellent blog from Jay, Thanks Krish for introducing this blog on ur channel !!

Thanks for explaining Jay's blog. To add to the explanation at 39:30, the reason for using sqrt(dk) is to prevent the problem of vanishing gradient as mentioned in the paper. Since we are applying softmax on Q*K and if we consider a high dimension of these matrices, it will produce a high value which will get transformed close to 1 after softmax and hence leads to a small update in gradient.

Thank you. Your teaching and jay's blog combination pull this topic. I like the way you are teaching. Keep going.

Great Session Krish. Because of Research paper I understand things very easily and clearly.

Wonderful explanation of blog, thanks for introducing with jay. Your teaching style is awesome.

Very helpful! Thank you all contributors!

Jay alammar blog is of course awesome. But you made it even more simpler while explaining. Thanks a lot

I love your patience how many times you go around explaining things until they get clear even for such dumb guys as me. BTW residual connection are not due some layers are not important and we have to skip them, it is for to solve the vanishing gradients problem.

Really nice sir, looking forward to Bert Implementation 😊

Very huge and tremendous effort, million thanks for your dedication

sir thanks a alot , mza agia sir , your way of teaching with so humble and honest and most important patience , awesome video sir, too gud

Thanks Krish, Awesome session, keep doing the great work!

Wow what an explanation of transformers.. perfect for us.. it aligns with the way we r taught at school…

Very underrated video... this is super awesome explanation. I m watching and commenting 2nd time after a month.

For those getting confused with 8 heads, all the words would be going to all the heads. It's not one word per head. The X matrix remains the same only the W matrix would change in case of multi-head attention.

Every time I get confused or distracted while listening to the Transformers, I have to watch the video again; this is my third time watching it, and now I understand it better.

Very clear explanation. And Jay's blog is also amazing!!

After watching your lecture it's more clear to me Thanks Krish

Thank you Krish. I learned so many things from your video.

Great effort Krish, Thanks

Superb. Well done and thank you for this.

Great presentation! I understand it fully now I think.

Great Effort. Very well explained

Took More than 5 hours to understand this. Thanks Krish wonderful explanation.

Video was so good, i understand each and every thing just except only decoder side .

THank you Krish and Jay for this work.

@31:45 If my understanding is correct, reson why we have 64, is because we we divide 512 into 8 equal heads. As we are computing the dot products to get the attention vaue, if we do the dot product of 512 embedding dimension length it will not only be computationally expensive but also the fact that we will get only one relation between the words . Taking advantage of parallel computation we divide 512 into 8 equal parts. this is why we call it as multi head attention. This way its computationally fast and we also get 8 different relation between the words. (FIY Attention is basically a relation between the words ). Any way Good work on explaining the architecture krish.

Very well explained Sir! Thank you.

Thanks to jay alamaar sir and you for the great explanation.

Hey Krish, thanks for the session. Great explanation! Could you please suggest if you have already uploaded session on Bert? And if not do you have still on plans? Would be very interesting to deep dive into practical application of Transformers.

Awesome explanation.. when will you post BERT video ? waiting for it and if possible please cover GPT-2 as well.. Thanks a lot for this amazing playlist.

Great Session!....looking forward to Transformer Based recommender system

Great to overcome confusions. I hope next to get hands on Bert.

sir the way you explained the topics is ultimate sir

Very well explained. Thank you sir.

superby you made the things look so easy

great explanation.. I understood Transformers now..

Thanks a lot for detailed explaination. Really appreciate your effort for creating these videos

Thank you, sir, that's a nice explanation. also thanks to Jay Alammar sir.

Thank you so much sir for this superb session.

Hi Krish, great session. I have a question - the Z we get after the self-attention block of the encoder, is it interpretable? that means if we could figure out by just looking at Z what results does the multi-head self-attention block gives? Kindly help me out with this.

krish sir, it's amazing!!!!

I don't know how to thank you and jay enough!

Thank you sir. It was awsome

Why they multiply each value vector by the softmax score because they want to keep intact the values of the all word(s) and they want to focus on, and drown-out irrelevant words (by multiplying them by tiny numbers like 0.001, for example) ... they wanted to immerse whatever that sentence has irrelevant words ...

thanks for such free contents!!...u r awesome sir!

Thank you Krish for making such a great video. Really appreciate your hard work. One thing I have not understood here is that where is the loss getting calculated? Is it happening on the multiple heads or at the encoder decoder attention layer. What I am assuming is that while we are training the model, the translations will not be accurate and we should get some loss which we will try to minimize but I am not understanding where is that comparison is happening?

Thanks for the wonderful explanation .. For the decoder in the 2nd time instance we passed word/letter 'I', then in 3rd time instance do we pass both the words 'I' and 'Am' or only the word 'Am' is passed? Similarly for the 3rd time instance do we pass the words 'I', 'am' and 'a' or just the word/letter 'a' is passed?

superbly explained

Always helpful Sir!

Good job Krish.

Great Explanation

You are amazing as always !

i really want to thank you for your nice explanation actually i could not be able to understsnd it befor watchining this video

In my opinion, At 40:00 the under root is taken for the purpose of scaling to normalize the value from larger value to be transformed to smaller value so that SoftMax function of these values can also be calculated easily. Dk is the dimension whose under root is taken to scale the values.

More this kind of videos on Research paper explanations and advanced concepts of deep learning and reinforcement learning sir.

Thanks a lot!

Hi Krish, When you gonna make a video on "Bert" with practical implementation ??

Thank you.

Wooowoooooooowwwwwwwwwwww At last I got clarity. Thanks so much for wonderful explanation

My MS SE thesis completion totally depends on your videos. Just AWESOME!!!

great!!!!!!! Krish

thank you🙏

Thanks krish don!!!

The reason to divide by sq of k is to prevent a constant value of x. That x = 1/2 for values near x = 0 from the left or right f(x) approaches y = 1/2. Look at the shape of the sigmoid function.

Well Explained Sir

Pretty good Explanation Mate

Thank you so much ..

Superb explanation

awesome bro

Thanks!

Thankyou ❤️

thanks, Question: in step 1 (30:52), what if the randomly initialized weights have the same value during the start? then all resulting vectors will have same values.

You are the best 😇

The layer normalization does (X + Z) here X is input Z is result of self attention calculation. You mentioned when the Self attention doesn't perform well, the self attention calculation will be skipped and jumps to Layer Normalization, hence the Z value will be 'EMPTY' (Please correct me here, if I'm wrong). In this case the layer normalization happens only on X (the imput). Am I correct?

watching through this video, I can only conclude that the whole process is more of a Art than it is a science

Clear explaining

at 58.49 it is told that if we increase no of heads it will give more importance to different words. so 'it' can give more importance to 'street' also. so between 'The animal' and 'street' which word will be more prioritized?

After the encoder. Is there any repository like which store all the output of encoder and then one by one it will pas to decoder to get one on one decoded output!

This video describes the inference of the Transformer. Can you do a video on training Architecture? I suppose we would need to give both languages datasets for training.

AFAIK Resnet is not like dropout, instead it brings information from the previous layer to the n_th layer by doing this, vanishing gradients are less likely to occur.

How do we get K and V vectors from encoding output. It would be helpful if you can explain ot with dimensions

Hey Krish, Had a quick question related to the explanation at 1:01:07 about positional encodings. How do we exactly create those embeddings, as in the paper the authors have used sine and cosine waves to produce these embeddings, I could not understand the intuition behind this, could you please help me understand this part, Thanks in advance.