I watched this video at the beginning of my ML Masters journey 2 years ago and it all felt like a foreign language. Today I'm watching it again and I’m proud to say that I understand a lot of the things discussed. Whoever is reading this, keep grinding!

Holy cow, these videos are so valuable. This view that Transformers are generalization of MLPs, while CNNs and LSTMs are specializations of it is brilliant. Thank you.

Even the number of TPUs that they have used is enough to guess where this paper comes from.

Transformers still have inductive biases ( no, I am not talking about the residual connections ). Like for example the softmaxes used to determine similarities between the elements or the fact that we still use multiple layers which still talks about hierarchical structure of sorts. But they have weaker bias in the sense that their biases look much more abstract at the world. A CNN bias says this is how images looks like. Transformers say this is how you can describe how a large set of unique stuff interact. And before CNNs, Wavelet transformers were saying this is how edges in an image look like. The biases are still there they have just become much more abstract as we learned that we can learn anything if we applied the right training tricks and datasets. That said residual connections are very abstract concept and for that I do not think we are touching them for a while. They essentially say look at the big picture first and give me your decision, then look again but this time closer and try revising your decision. That is a very efficient computational model we as human beings employ on daily basis. So are transformers by the way.

I'm slowly starting to grasp more and more what the heck is going on in these videos!! I've been watching your paper reviews for some time now, and I think this is the first time I actually get everything you're saying in a video. That feeling is great. Thank you for your work!

You have a talent of making a complicated idea simple ans digestable yet keeping its technical fundamentals understandable.

The Transformer architecture is a nice example of Sutton's "Bitter Lesson".

I love the insights you give into the concepts presented in the paper. I very much like the last 2-3 minutes ranking NN architectures by bias

This is so helpful for an undergrad like me. Yannic explains topics in easy understandable terms. Thank you. Love from INDIA.

Double blind review as it is currently done is based on a purely voluntary work done by reviewers, ACs, etc and is based on the mutual trust that i) reviewers respect the idea of double blind that is necessary to reduce the bias of paper evaluations, as well as ii) authors write their papers in an anonymous way. There are of course exceptions when this is broken, intentional or unintentional, but it would be very hard and resource expensive for any conference to impose 100% accuracy of this process.

This was the best video on transformers I've seen on YT, still in 2022. I think Yannic may be one of the only people that is actually able to productively reason about the core dynamics, effectiveness and utility of ML model designs.

At first I thought you were very happy with the double blind review process, but I soon found out that you weren't. You have encouraged me to arxiv my work and continue to work hard to do what I think is important, rather than wait for top journals/conferences to accept. Thank you.

Excellent explanation for visual transformers. The only thing that can be concluded is that most large companies will offer their large models as a service through the API. It is simply very difficult to train such a large model with small capacities.

It has been difficult for me to understand the transformer until you mentioned "sets" at 7:06 halleluah. small vocabulary precision can make a huge change!

The comments about the biases of these different type of architecture really helps me gain better understanding of these things, thank you!

Not sure if I buy your argument that transformers are somehow more general and less biased than MLPs. They implement the prior of repeated computation and spatial invariance. In addition to those priors, i think the real reason transformers win over MLPs is because we given them a multiplication operation in representation space (attention). MLPs only get to perform addition of their hidden representations

The discussion (and insight) that we've introduced these biases to simplify the Model's job and make up for inadequate training data, but in doing so have limited overall performance is incredible.

'Double blind' - but only in one eye!

22:46 I think, that while you use the terms inductive bias and inducive prior interchangeably in this passage - you rather describe an indictive BIAS than an inductive prior. The difference would be, that an inductive prior will set a starting point until your model gets informed by your data, while the inductive BIAS inevitability limits the space of possible model-solutions. Both occur together, like the case when you bias your DL-model through its architecture while giving it a prior initialization of parameters. Have I misunderstood something? Either way: I love your videos, thanks!

I wonder what would be more general than a transformer. Maybe something like: - dynamic #convergence iterations (currently transformers do only a single projection step between layers) - dynamic #layers - which layers to skip to (potentially giving something like type determination where you can feed in data and if it is text it gets routed through one set of layers, if it is images, it gets routed another way or something like that) - what amounts to wider-label attention, like, not only care about what the current and next layer of this specific neuron output/is looking for, but also their same-layer neighbours, or their equivalents from earlier/later neighbours. Not sure if that makes sense, but basically just give the soft routing mechanism more inputs to arrive at its decisions.

Getting rid of back propagation altogether could be a "next thing". Maybe by randomly approximating the weights, maybe using some sort of very-fast-very-parallel-hardware that would update all weights at once with random values. And maybe afterwards make this process be learnable somehow :/ (maybe learnable as in disconnecting neurons that are not needed) Maybe also having a generative neural network synthesizing datasets from a small amount of samples, so that if the model sees an input that was not in the dataset, it may have "dreamed" about it during training and be more robust.

I am enrolled in a masters course in AI and i have to read lot of research papers like these as new ones come out and this channel has the best simplest paper explanation videos out there. Also i completely disregarded all the hints about the authors of the paper i dont know who wrote it. 🤫

It's impressive that you predicted that residual connections are going down next. I've seen multiple papers where people start improving on their complexity

Never got into your videos before but your perspective is super interesting, thanks for making these videos :) I think your perspective on how we bias the models and how you view transformers to be the most general even more so than MLPs) was insightful and something I had to reflect on

Yannic, I absolutely love your reviews :D so energetic! thanks for your work man

I figured it out! Google must have submitted that article.

As humans when we think of attention in vision. It is not that our eyes shut down when we are not paying attention to our surroundings. Our eyes keep running object detection/motion detection tasks with dedicated neurons. It is just the higher order functions decide not to pay attention to that already processed information and use that bandwidth on something else. 19:30 Attention is a generalization of convolutions. 10:50 I think this is equivalent to running the image through a 16x16x3x16 conv filter with stride 16. (i.e. it takes a 256x256x3 tensor and gives a 16x16x16 tensor). Then pass it through a few layers of 16x16x16x16 conv with stride 0. (i.e. a conv block which takes a 16x16x16 tensor and spits out a 16x16x16 tensor). Then a dense at the end for classification. MHA is equivalent to a full nxn conv. 20:40 A 16x16 conv2d could do the same. Seriously though. I dont see the benefit of double blind in non-medical papers.

I appreciate the jests :D great to have you pumping out videos again. Thanks for another banger :)

I like the idea to see transformer as a general version of mlp with even less inductive bias. very insightful

I wonder if the initial “down sample” to a low resolution image could be avoided (without insane compute) using a transformer of transformers. One spatial one temporal. The inner transformer (spatial) would be just like this paper on a downsampled image, but instead of downsampling the whole image you downsample a window (still to a fixed low res size). You then do this on N windows to form a sequence (sort of like your eye temporally moving around focusing on parts of the image), and transform that.

"Look, we know you can learn anything from data" - Yannic to MLP.

Thank you so much for the video! It's a pleasure as always. Should we just stop ai research for 20 years and than train a Transformer net without skip connections on quantum computers with yottabytes of image data? Would save a lot of work 😜

I want to read papers but reading paper is hard and costs so much time your videos are so fast and clear thank you my friend

Even famous group of authors can be identified easily in open review, it probably still benefits less known authors.

The whisper part in Experimental Results got me real good )))

I don't think skip-connections are as biased as you claim. The visual intuition is: you might recognize an alligator by the texture of its skin combined with higher order features like the ensemble of shapes that make up its body. But the intuition can be generalized beyond just the visual domain. Sometimes a combination of low-level and high-level features is more informative than just high-level features on their own. If your language model is reading some Eminem lyrics and it outputs only high-level features like the subject-matter, sentiment analysis, theme, etc. It might classify it as Country Western if it abstracts away the low-level features like syllables and rhyme.

I think transformers are more general simply because they take into account the lateral connections between the neurons. This is what self-attention does in principle. The human brain has lateral connections among the neurons too.

Yannick is at his best when he goes hard with the shade :>

Great video Yannic, explaining transformers. You were on the money about who the anonymous authors were. No surprise there. I found your video because of another paper from Google, in this case, DeepMind that also uses transformers, but in this case applied to 3D meshes. The name of the paper is Polygen: An Autoregressive Generative Model of 3D Meshes. I will be looking forward to see you reviewing this paper, if you ever do. Is it the case that we are facing a transformer revolution in deep learning?

Authors: Let's be anonymous and make the readers go nuts. Kilcher: 🤭

Could you explain a little more about how Transformers are a generalized version of MLPs? Aren't the weights of MLPs also found on the fly?

Hi Yannick, I love your videos because you always ask a lot of questions and not just read the paper. If I understood correctly, when you say, transformers are more general because other than feed forward connection, we have self-attention where we calculate the node/neuron values each time before feed-forward. Could you please elaborate a little why this is more general than MLP? Shouldn't MLPs be able to figure this out because of their complete connection given enough data and self-attention is another inductive bias in a way?

This is an awesome explanation. We can learn everything from data if we have an infinite amount of data, and we can have never have infinite amount of data, we must introduce inductive bias or strong priors and try to learn the universe from limited samples.

This video and especially the explanation about inductive biases are pure gold!

Having a bias but a small variance while making use of the Markov property at the same time might be the closest to human intelligence for learning the real world model. Is this what the Temporal Difference Learning discloses?

24:18 "Bias -- in the statistical sense". I see someone has learned from Yann's twitter feud 😬

Nice explanation, thanks! For me it was even super obvious just mentioning Kolesnikov so many times, you only do that if you're Kolesnikov and you're building off of that work! It's obviously not anonymous.

2D slices of high-dimensional loss landscapes show that skip connections make the landscape much smoother/convex. My intuition is that they will remain, but who knows?! :)

Great video. I love your intro about the anonymous paper. Thank you for sharing. Well done!

18:14 so fun :D

27:28 "There can be something even more general than the transformer" - Yes and that something is called a GNN! Check out this blog Yannic: thegradient.pub/transformers-are-graph-neural-networks/ And thanks for the video! I saw the paper the same day that Karpathy tweeted it. Edit: forgot you made this one back in the day when GNNs were not as popular.

Good point about convolution being an inductive bias

Nice review...especially the sections after 21:00.

The generalization/specialization discussion was extremely helpful. Amazing content

That intro was hilarious

MLP bias/prior is the connections themselves. Very Interesting. So the most general system would have the entire architecture and structure be computed on the fly.

"we know this, you know this" me: nods head as a C+ student in my ML course

Great explanation, But I'm still confused about 1 thing. Tried reading the paper but didn't really click still. Question is regarding "extra learnable [class] embedding". Can someone give some examples as to what the input is to this embedding is? Is the input going to be the same special token (*) for all input samples or is the class label going to be added here? (The latter seems not very sensible), but whats the point of having an learnable embedding if the input is always static?

So... how does this compare to adding 'zoom'/'pan' functions, adding xyz+self embeddings, and letting a transformer tell a single 16x16 fully connected layer where to look? Did I beat google?

Amazing explanation especially about the comparison between transformer, mlp and cnn. Thank you so much!

Awesome, I have been wondering a while ago if the network that a network performs at each layer is not "too static". In a normal CNN the operations are completely defined by the programmer und only the weights are learned. But maybe the operations themselves can be learned as well. Seems like Transformers are going a bit in that direction if I understood everything correctly.

Thank you for your sharing, it's very useful and interesting! But I have an issue about it. My issue is, with the constant number of patches, why don't we use the naive positional encoding (0/n, 1/n, 2/n, ..., n/n)? I think if we use it, the distance between 2 different patches will not be effected by n, because n is the constant.

I still like Double-Blind reviews. The problem with open review is that the bias is higher. Even if the paper gives a lot of clues about who the author is in this case, you still have to do some digging to reach this conclusion. Not everybody will have this time. In addition, double-blind reviews will benefit unknown researchers that have not made a name for themselves yet by giving them a more fair chance to be evaluated. It's impossible to be perfect but I do strongly believe that double-blind is way better than open review or single blind review processes.

Having peer-reviewed under the traditional "blind" review system, remember that "if the reviewer is well-acquainted with the field and most aware of the nuances involved with the discipline and the domain in specific, then it's often trivial--especially if the topic was arrived at scientifically, whose paper it is." Of course, for people who don't know what I meant by properly arrived at the topic scientifically, as only one logical, syllogistic extension from a previous work, then you're contributing at least as much noise as signal, but unfortunately what I found is that traditional researchy universities do not give formal structured introductions to philosophy of science flowing through to refining research topics, which implies an appropriate research design, which implies an appropriate methodology, analysis, interpretation, and so on: So the fact of the matter is placing too much emphasis on the blind part will convey exactly how uninformed about a specific topic area one is (by virtue of their own knowing what researcher is active in what direction, with finer and finer breadth to deeper and deeper extent across the "knowledge shore", which becomes so familiar that one soon drops any such value in that pretense. Of course, I can only speak from a neurodivergent perspective; I hear that it is quite common for neurotypical types to not notice glaringly obvious facts, or ignore them if they're not already important to them--in which case, I can totally see why the crutch of anonymity is needed...

I'm naming my next network NiT-L/16.

Video starts at 5:22

Hey Yannic. You ever seen this paper arxiv.org/pdf/2004.02016v4.pdf ? The topic is pretty niche (meeting summarization) so I understand that it didn't make a huge splash. However the model they use I find interesting. They essentially take a transformer but apply a convolutional philosophy to it, by which I mean multiple layers, each abstracting the previous a little more. In this case they had two encoders, one which operates on word-level (same as normal transformer encoder) but then they have another encoder which takes in the output of the previous. Thus they have an encoder which operates at word level, feeding each sentence into another encoder which operates at sentence level. Both of these feed into a decoder. I personally believe this might be the future of transformers. Early layers encoding a large amount of input to more abstract and condensed information, then feeding that into more encoders. Have you seen other research doing this also? What do you think of this?

Good explanation. Disagree with the opinion on residual connections being an inductive bias. Without residual connection, a layer might not be able to learn an identical mapping because of non linear transformations in the layers.

Great video. Have a quick question though, when you mention the difference between MLP and transformer, you said transformers compute w onnthe fly12:02. Doesn't w in transformer also fixed?

I really enjoying your humor! Totally random, absolutely not related to all between papers similarities comment.

How do we ensure that learnable parameters of each head is different from each other? It might happen, all the heads end up learning same thing

Aren't we also adding a similar bias to CNNs by feeding in image patches to the transformer? This still forces a relationship between neighboring pixels.

Yannic! your channel is awesome, thank you for covering so many interesting things in a nice digestable way! Stay cool

Hey, what's the app you use to anotate your papers? Thank you!

Hi, in the video you claim that you can not get the relation objects far away with CNN (in 21st minute). But when we put fully connected layers after flattened the CNN layers we should be getting all combinations of those features regardless how far they are, am I wrong?

do you guys think that this method so similar to the NON LOCAL

I would actually consider the transformer as a regularized MLP (i.e., a special case of MLP, not the opposite way around). The weights that connect nodes between layers are regularized by the similarities of the connected nodes.

Wouldn't the most general learning system be a fully connected graph with something like an FIR filter for each connection instead of weights?

Great video, but when we read we actually don't just read one word after the other. We are more like transformers with a certain attention span.

Your insights really helped me understand this paper. Thank you!

skip to 5:08 for start of explanation

very interesting and informative explanation, and also a very well complement of my own understanding, thanks.

18:14 When you're destiny to be an ASMR artist but your parents want you to do science

Great great video! As a PhD student, thank you for making this valuable video

Liked in just the first 3 minutes for the laughs on your very accurate jokes. Then I saw the entire video, I would have liked it anyway ;) thanks great work

We are so close to this being a reality friends. Hyperealistic images from a simple definition. GptI was really promising, imagine the quality if a model the size of gpt3 was trained to do it.

Thanks tons for sharing this ! you are really good at explaining papers :) I'm new to transformers so please bare w my potential noobie question. Would you say that one disadvantage of transformers (probably stemming from their general design) is that, in comparison to CNNs, larger amounts of data is required in practice to train good models? Thanks for your feedback

Excellent video summary! Can you explain why you think Transformers is a 'more general' architecture? It seems to me that it has more 'inductive biases', but would that not imply less general? What is it about transformers that make it more general than MLP?

love your insight on LSTM/CNNs are more 'biased' architectures and transformers are less biased, and more general architecture thus, it seems to work in the paper. one question tho. then if there are not enough data, does that mean it would be better to use LSTM/CNN based architectures instead of transformer based? wonder if there is anyone who did this comparison

At 08:00 you saying that there are N^2 connections. Isn't it "n choose 2", C(N, 2) = N!/(2*(N-2)!) ?

The question here is how to allow some translational invariance. You could do a random projection before a cnn and it would lose any of that ability. For example a fixed pattern of random sign flips followed by the fast Walsh Hadamard transform gives a quick random projection. However using a less random pattern of sign flips with some spacial structure and maybe you can even do away with convolution since it results in more or less the same thing.

Not sure why the review is termed “double blind’... ...double blind in medical/psychological studies are needed so that: - Blind 1: subject’s condition doesn't change due to knowledge of which test condition they're in (enabling placebo control + reduced variation from demand characteristics) - Blind 2: investigator bias is not introduced, which can corrupt the test conditions, leading to different outcomes Analogously, the reviewer is the investigator, the paper is the participant. Blins 2- I totally see the point of Blind 2 - potential conditions are e.g. "research by renound group", "don't like these people", "never heard of them" (though as Yannic points out, there might be some holes in the blindfold). Blind 1 - didn't make sense to me - would imply the paper can change it's outcome based on what condition a specific researcher believes it belong to... (an author could tailor the paper and bias resuults to a specific reviewer, but they'd need to know the reviewer during the study/writeup - knowing the author during review doesn't change the content of paper or the outcome & arguably any revisions submitted should be taken on by a different reviewer) Anonymous reviewer if fine, has the benefit of no backlash, and reduces perception bias once published (later readers not knowing who reviewed means they're not dismissing or overemphasizing papers based on reviewers). But that's not double-blinding , nor is it part of the review.

great video please keep on reading papers and explaining them!

People are training on 100s# of TPUs and I am suffering with my 250Gb SSD and 1080ti GPU, then people say why don't you beat SOTA

Now what about the attention of the pixels in the same patch between eachother?

"Rethinking Attention with Performers" coming up next?

Even stranger how our brain can learn with less data. and yet generalize so well.

arxiv version released: arxiv.org/pdf/2010.11929.pdf

That intro was Epic :)

Aside from the unnecessary 5 minutes long rant, this is a good video.