Thank you so much!

Was coming to comment this, thanks

Please make video

Please!

Would you share the source code for the animations?

amen. the power of out of the box teachers is infinite.

Damn FNN's and CNN's are basic stuff we were taught in our 4semester of our undergrad. Adam and RNNs were in the "additional resources" section for an Introdcutory course for Deep Learning I took in the same semester. Encountered LSTMs through personal projects lol Still haven't used GANs and Autoencoders, but it they were talk of the town back then due to the diffusion models.

@@newbie8051 yea I did FNN from scratch in high school, I was really hopeful for getting into Ai Research and then the transformers arrived in my college year…

My one-sentence summary of why transformers matter would be "they are standard CNNs, except the words are re-ordered in a way that makes the CNN's job easier first before being fed ". Also, a single NN layer IS a linear layer; I'm not sure what you mean by saying you don't know how important the distinction between the two is.

thanks@@maxkho00

"they were using a convolutional neural network in the transformer" No no, Transformers do not have any convolutional layers, the author of the video just chose CNN as a starting point in the process "Let's start with the solution that doesn't work well, understand why it doesn't work well and try to improve it, changing the solution completely along the way". The main architecture in natural language processing before transformers was RNN, recurrent neural network. Then in 2014 researchers improved it with attention mechanism. However, RNNs do not scale well, because they are inherently sequential, and scale is very important for accuracy. So, researchers tried to get rid of RNNs and succeded in 2017. CNNs were also tried, but, to my not-very-deep knowledge, were less succesful. Interesting that the author of the video chose CNN as a starting point.

@@Hexanitrobenzene, I suppose I’ll have to watch this video again. I’ll look for what you mentioned.

@@terjeoseberg990 A little off topic, but... Not long ago I noticed that KZfaq deletes comments with links. Ok, automatic spam protection. (Still, the thing that it does this silently frustrates a lot...) But, does it also delete comments where links are separated into words with "dot" between them ? I tried to give you a resource I learned this from, but my comment got dropped two times...

...Silly me, I figured I could just give you the title you can search for: "Dive into deep learning". It's an open textbook with code included.

@@Hexanitrobenzene, The best thing to do when KZfaq deletes comments is to provide a title or something so I can find it. A lot of words are banned too.

Thanks for the tip.

Thanks for the tip! I'm always happy to answer questions.

Thanks, any title in particular you'd recommend?

Thanks so much!

the first so far this year

this is a good question and it's also a GOFAI type approach where we make the mistake thinking we can inject some human semantic idea to improve a network. but the reality is it will do this automatically without our help. For example papers back in 1986 show tiny networks automatically grouping words into nouns or verbs, it's amazing. let me know if you want more details

I"ll be doing this, but in short, the little networks he showed connected to each pair are KQ (word pair representation) and the V is the value network., all of this can be done in the decoder only model as well. and cross attention is the same thing but you are using two separate sequences looking at each other (such as two sentences in a translation network). it's nice to know that GPT for example is decorder only, and so doesn't even need this

Yep all my videos so far have been done in Manim.

Yes that's correct, the transformer just adds x without passing it through an additional linear layer. Including the additional linear layer doesn't actually change the model at all, because when the result of self attention is run through the MLP in the next layer, the first thing the MLP does is apply a linear transform to the input. Composition of 2 linear transforms is a linear transform, so we may as well save computation and just let the MLP's linear transform handle it.

The entire model is trained end-to-end to solve the training task. What this means is you have some training dataset consisting of a bunch of input/label pairs. For each input, you run the model on that input, then you change the parameters in the model a bit, evaluate it again and check if the new output is closer to the training label, if it is you keep the changes. You do this process for every parameter in all layers and in all value and score networks, at the same time. By doing this process, the importance score generating networks will change over time so that they produce scores which cause the model's outputs to be closer to the training dataset labels. For standard training tasks, such as predicting the next word in a piece of text, it turns out that the best way for the score generating networks to influence the model's output is by generating 'correct' scores which roughly correspond to how related 2 words are, so this is what they end up learning to do.

The entire model is trained end-to-end to solve the training task. What this means is you have some training dataset consisting of a bunch of input/label pairs. For each input, you run the model on that input, then you change the parameters in the model a bit, evaluate it again and check if the new output is closer to the training label, if it is you keep the changes. By doing this process, the score generating networks will change over time so that they produce scores which cause the model's outputs to be closer to the training dataset labels. It turns out that the best way for the score generating networks to influence the model's output is by generating 'correct' scores which roughly correspond to how related 2 words are, so this is what they end up learning.

Yeah it's a term I made up so you won't find it in any sources, sorry about that. Usually sources will just talk about self attention in terms of key, query and value lookups, so you can look at those to get a more detailed understanding of the transformer. The value transform is equivalent to the linear representation function I use in the pairwise convolution, the key and query attention scores are equivalent to the bi-linear form scoring function I use (with the bi-linear form weight matrix given by Q^TK). I chose to use this unusual terminology because, personally, I feel the key, query and value terminology comes out of nowhere, and I wanted to connect the transformer more directly to its predecessor (the CNN).

@algorithmicsimplicity, this is a surprising connection. Thanks a lot for the explanation.

@@algorithmicsimplicityit would be great if you could make this connection between terminology in video form. maybe next time?

Thank you for your support!

The importance-scoring neural network is trained in exactly the same way that the representation neural network is. Roughly speaking, for every weight in the importance-scoring neural network you increase the value of that weight slightly and then re-evaluate the entire transformer on a training example. If the new output is closer to the training label, then that was a good change so the weight stays at its new value. If the new output is further away, then you reverse the change to that weight. Repeat this over and over again on billions of training examples and the importance-scoring neural network weights will end up set to values so that that the produced scores are useful.

The pair vectors are first concatenated together into one vector e.g. [00010100], and this vector is then run through the neural network which produces the output vector. The output is the result of the weights in the neural network. Initially, those weights are completely random (usually sampled from a normal distribution centred at 0), and then they are updated during training. The neural network is trained on a labelled training dataset of input and output pairs. For example, ChatGPT was trained to do next word prediction on billions of passages of text scraped from the internet. In this case, each training example is a random part of a text passage (e.g. "the cat sat on the") and the output is the next word that occurs in the text (e.g. "mat"). For every training example an update step is performed on the neural network to update all of the weights of all of the layers. The update step works as follows: 1) Evaluate the neural network on the input. 2) For every weight in every layer, increase the value of that weight by a small amount (e.g. 0.001) and then re-evaluate the entire neural network on the input. If the new output is closer to the target (e.g. the vector output is closer to the one-hot encoding of "mat") then it was good to change that weights value, so it keeps the new value. If the new output is further away from the target, then it was a bad change, so reverse it. And that's it. Just keep repeating that update step for billions of different inputs and all of the weights in all layers will eventually be set to values which allow the transformer as a whole to map inputs to outputs correctly. Also I should point out that in practice there is a faster way to do the update step which is called backprop. Backprop computes exactly the same result as the update process I described, it is just faster computationally (you only need to evaluate the model twice instead of once for every weight), but it is also more difficult to understand.

i'm working on a video now which is attempting to do this, been on it for months. one key thing here I notice where you get fuzzy, is thinking CNN's 'only do classification', and that this is different than nex word prediction. Because next word prediction is a type of classification (the output class is the next letter or word). and so you could have just a plain old fully connected network do "next word prediction" based on training it that way. please let me know what else you are thinking as it might help me with my script in my video I will open with RNN's applied to next work prediction (starts in 1986) then I will explain where they break and why we need parallel approaches why simple brute force doesn't work (too many parameters, and hard to train) why transformers helped (compressed many layers into fewer adaptive layers)

@@ArtOfTheProblem uff, I'm not sure my comment made any sense at all. I'll reply more on your latest micro-vid, which considers an Attention block as a dynamic-routing layer.

The animations were made with the Manim Python library (www.manim.community/ ) and edited with KDenLive.