Personally I find it more clear to link embedding to hidden class of words, i use character sheets as a netaphor, because what attention does is not looking at word but the description in its sheet, with each attention head focusing on different part of the description, which mean a word representation have multiple attention on different hidden class. Then at the end we look at the sheets transformed at each layer to find the next word. That also allows to explain multimodality, ie make sure image input and text input share the same description sheet.

@@NeoShameMan Interesting.

Transformers needs more than one video, each part(multi H attention, word embeding(sin&cosine similarity),training &…) I was waiting for long to reach stat of art.

@@MrMehrd I thought about doing it that way - and that was the original plan. But my video on Attention convinced me that most people would rather have a single video that has everything in it all at once. However, I've provided links in this video's description to full length videos on each topic you are interested in.

@@statquest oh you mentioned that you don't know why this number of head, that's hardware optimization, ie they can be split into gpu or memory pool or reduce bandwidth, such that they can parralelized or compute sequentially on resource starved machine.

Thank you! BAM! :)

Me too!

@@statquest our data analysis professor also uses your videos as references and recommends you almost every session haha. i learned about this amazing channel from him.

@@Daigandar That's awesome! BAM! :)

Well said. I do the same!

Thank you! :)

Much like Richard Feynman.

BAM!

Wow, thank you!

Did you just tokenize your comment?

@@lumiey I'm not sure I understand.

@@statquest He just separated words like complet, est, precious, est, pur, est... like tokenizer does (e.g. following -> follow, ing)

@@lumieyHaha

Awesome!

Thank you

Does the second part mean that each head only gets a portion of the embeddings?

@@jakob2946 curious aswell, I looked it up and it seems that its not true, every head is applied to all dimensions of the embedding. This also makes more sense to me since the word embeddings should be looked at as a whole. please correct me if I'm wrong

@@oliviervangoethem9365 I don't know about more recent transformers, but in the initial architecture each attention head is applied to a projection of input embeddings, with reduced dimensionality (in the original "Attention is all you need" paper: embeddings have a dimension of 512, and each of the 8 attention heads has a dimension of 64). The reason for this is spelled out in the original paper: "Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this."

Thank you so much 😀!

Thanks! It's a little easier to understand how this model was created in the first place if you follow the whole Neural Networks playlist. You'll see how things changed, one step at a time, to eventually end up with a transformer: kzfaq.info/get/bejne/edd_mcxllrLKdKs.html

@@statquest Thanks Josh! I’ll check out that playlist. What you’re doing is so special to the world, and humanity is so indebted.

Glad it's helpful! :)

Congratulations! TRIPLE BAM! :)

Thank you so much!!! I really appreciate your support (TRIPLE BAM!!!). I hope to do the GPT video soon, but we'll see - the timeline is a little out of my control right now.

Thank you very much! :)

Thank you very much! :)

Thanks, will do!

Hooray! :)

Triple bam.

2017...

Wow, thanks!

Which book?

@@meirgoldenberg5638 I think he is referring to my book, The StatQuest Illustrated Guide to Machine Learning at statquest.org/statquest-store/

It can be made much simpler if you learn about neural networks first. You can do that with this play list: kzfaq.info/get/bejne/edd_mcxllrLKdKs.html

Glad it was helpful!

Hooray!!! Thank you very much!

Thank you very much! I'm glad my videos are helpful.

Thank you very much!

BAM! :)

Thank you from a fellow slowpoke

Glad it was helpful!

Thank you very much!!! I really appreciate it.

Wow, thank you!

Thank you!

Thank you!

bam! :)

Glad it helped!

Thank you!

agreed!

bam!

You are so welcome!

Glad you enjoyed it!

Hooray! :)

Yep - I'd like to make a GPT video just to highlight the explicit use of masking (the self attention in the decoder in this video used masking implicitly).

@@statquest Please do that video soon :) BAM

Happy to help!

Thanks! I've also got a video on Decoder-Only Transformers: kzfaq.info/get/bejne/mLdlddKg0b6dcZs.html and I'm working on one that shows the matrix algebra (color coded) of how these things are computed.

@@statquest are all these topics covered in your book ? Would love to read them in printed format

@@shaktisd They'll be in my next book.

@@statquest looking forward to the next edition.

Thanks!

Thank you very much!

TRIPLE BAM!!! Thank you so much for supporting StatQuest and I wish you the best as you continue to learn about ML and Data Science! :)

Glad you liked it!

In visual transformers, we do understand what each head does. I guess heads trained on language are more difficult to interpret for us.

@@GreenCowsGames no, we don’t know that they do.

Thanks!

Thank you! Will do!

I'm starting a book on neural networks every soon.

Thanks! - Yes, this video packs in a ton of information, but I couldn't figure out any other way to make it work.

That's the plan!

Thank you very much!!! BAM! :)

Glad you like them!

I have a couple of thoughts on this. Maybe I should make a short video called "some thoughts about positional encoding". Anyway, here they are... Thought #1: Remember the positional encoding is fixed, so the word embedding values have to take them into account when training. For example, since all of the positional encoding value are between -1 and 1, it is possible that the word embedding values will have larger magnitudes and thus, not move around a lot when position is added to them. Thought #2: Because the periods of the squiggles get larger for larger embedding positions, after about the 20th position, the position encoding values end up alternating 1 and 0 (in other words, after the 20th position, the position encoding values are 1010101....) and it is in that space, from the 20th position to the 512th position (usually word embeddings have 512 or more positions) that the word embeddings are really learned, and that the first 20 positions are mostly just for position encoding.

@@statquest Ah ok yeh that makes a lot of sense, thanks so much for taking the time to reply!

I’ve been having some additional thoughts on this and think I may have another reason (or rather an example) why adding positional encoding to the word embedding vectors makes sense, Josh if you read this, feel free to shoot it down! Take the following sentence: “The weather is bad, but my mood is good”. In this sentence the first “is” refers to the weather, whereas the second "is" refers to my mood. Without positional encoding and only word embedding, the vector for “is” being passed into the attention unit will be the same for the two instances of the word in the sentence. If we don’t use masked self-attention and compare the word “is” to every word including itself in the sentence, then the output of the word “is” in the self-attention unit I believe should be the same for both instances. Therefore, the unit will struggle to successfully differentiate the relative meaning of the two words. By adding in positional encoding prior to the self-attention unit, we’re suddenly adding context to the word. The second “is” comes straight after the word “mood”, therefore the position vector we’re adding to each of the two words should be similar. However, because the word “weather” comes 6 words before the second “is”, the positional vector we add will be quite different. Presumably this difference helps a self-attention unit to differentiate the relative meanings of the two instances of the word “is”.

@@matthewhaythornthwaite9910 That all sounds reasonable to me! BAM! :)

@@matthewhaythornthwaite9910 Another reason why you would want to add positional encoding instead of doing something else is that it preserves the dimensionality of the encoding. Imagine a theoretical encoding which is not added (like a one-hot encoding for each sequence location), and some linear (or non-linear for that matter) transform to combine word embedding and positinonal encoding. This is great in the sense that we do not polute the embedding space with "arbitrary" offsets, but now our input sequence has to be of a fixed shape. Addition of orthogonal sinusoids guarrantees a non-parametric, dimensionality preserving encoding which does not fix the number of inputs we can give to the network. By the way, I think there is an analogy between adding positional encoding to embeddings and adding residual/skip connections to network outputs. Imagine that we have a network that is represented by the function f(x) and we have some target function F(x) which we want the network to learn. Imagine now that we modify our network to compute the function f(x) = h(x) + x (where "h(x)" is the network in front of the skip connection "h(x) + x"). Here too we polute the output space of h(x) with the values of x. However the network f can still learn F, so long as the network h(x) learns the function h(x) = F(x) - x (such that f(x) = h(x) + x = F(x) - x + x = F(x)). I suppose for positional encoding something similar holds (altough it probably has to learn a much more difficult internal pattern), where the network is f(E(x)+q) learns to associate word embedding values E(x) which are "convolved" by some known offsets q and probably learns to deconvolve E(x) and q (into some abstract representation).Given that E(x) + q may in theory be (nearly) non-unique (i.e. E(x_1) + q_1 = approx E(x_2) + q_2) it might still be possible for the network to deconvolve the values into the correct inputs based on the context vector C which is calculable from the rest of the input sequence. I suppose one can't exclude that the network may never get this wrong, but in practical terms, it seems to work well enough.

I'll keep that in mind.

Thank you so much 😀!

Thank you!

Thanks!

Thanks!

Glad you enjoyed it!

That's a great idea!

So kind of like a unit circle?

:)

Thank you!

BAM! :)

THAT IS AWESOME!!! (what are the chords?)

@@statquest I V vi IV

Thanks!

Great to hear!

Thank you very much! :)

Thanks and welcome!

Glad it helped!

@@statquest I actually had a doubt as I was going through, about the decoder part. In the masked multi head attention part of the typical transformer, what inputs do we provide? And is this part only used during training?

@@pratyushrao7979 I actually talk about masking in my video on decoder-only transformers here: kzfaq.info/get/bejne/mLdlddKg0b6dcZs.html

:)

Thank you!

Wow! Thank you!

Wow, thank you!

Thank you very much!

Thanks! :)

Wow, thank you!

Thanks!

Hooray! :)

Glad you like them!

Wow, thanks!

Glad it was helpful!

bam!

Glad I could help!

Thank you!

Thank you!

Glad you think so!

Thanks!

bam!

Awesome, thank you!

Thank you!

Glad it was helpful!

Bam! :)

BAM! :)

Glad you like them!

Triple Bam!!! Thank you so much for supporting StatQuest!!!

Thank you!

Glad you liked it!

Thank you very much! :)

Thanks!

For an encoder-decoder transformer, masked self-attention is only used during training, which this video doesn't cover. However, I cover it in my video on Decoder-Only Transformers here: kzfaq.info/get/bejne/mLdlddKg0b6dcZs.html

BAM! :)

Thank you!

Yep - this is a big one! :)