As many of you have asked: LLaMA 2's architecture is made up of the ENCODER side of the Transformer plus a Linear Layer and a Softmax. It can also be thought of as the DECODER of the Transformer, minus the Cross-Attention. Generally speaking, people call a model like LLaMA a Decoder-only model, while a model like BERT an Encoder-only model. From now on I will also stick to this terminology for my future videos.

Umar, Andrew Ng, 3Blue1Brown and Andrej are all you need. You are one of the best educators of deep learning. Thank you.

The best 1 hour I spent! I had so many questions exactly on all these topics and this video does an outstanding job at explaining enough details in an easy way!

The best machine learning videos I've ever watched. Thanks Umar!

As always, the PDF slides are freely available on GitHub: github.com/hkproj/pytorch-llama-notes/

Amazing video ! Thanks for taking the time to explain core new concepts in language models

I became your fan in 55:00 when you explain how GPU capability drives the development. 🙂

This video can be like official textbook of llama architecture. Amazing.

Fantastic explanation on LLaMA model. Please keep making this kind of videos.

The network came in Feb 2023! This is a youtube channel worth subscribing. Thanks man

Really glad I found your channel you create some of the most in depth and easy to follow explanations I've been able to find.

Such an amazing step by step breakdown of concepts involved! Thank you so much.

Very underrated video. Thanks for providing such a good lecture to the community

Thank you very much for your work. The community is blessed with such high quality presentations about difficult topics.

This video, along with the previous ones about coding up transformers from scratch, are really outstanding. Thank you so much for taking such a tremendous amount of your free time to put all of this together!

AMAZING! AMAZING AMAZING! Great work Umar .. Thanks a ton

Thanks for your free time and offering this valuable tutorial👏👏👏 Hope you keep going to do this, thanks again

This intro to llama is awesome ❤, thank you for making such a great video.

Thank you very much! I once read the paper, but I think watching your video provided me with more insights about this paper than reading it many more times would have.

Your videos are the best man !! Please keep releasing as much content as possible, on the famous papers

Fantastic video! Your explanations are very clear, thank you!

One of the best explanations on youtube right now !!

Please keep doing content like this. Thank you very much. I learnt alot

Thank you for creating such quality content!

I didn't even see the time passe ! Great work your are a future rock star at teaching complex thing in ML

Great content ,deep understanding after watching video

My TLDR for the video (please point out the mistakes): - LLaMA uses RMS normalization instead of LayerNorm because it provides the same benefits with less computation. - LLaMA uses rotary embeddings. These act as a distance-based scaling to the original dot product scalar value coming out of queries and keys. In other words, two tokens, X and Y, will have a larger scalar value versus two tokens X and Y that are far apart. This makes sense from the point of view that closer tokens should have a bigger say in the final representation of a given token than the ones far away. This is not the case for vanilla transformer. - LLaMA uses Grouped Query Attention as an alternative to vanilla attention mostly to optimize GPU Flops (and its much slow memory access). Key slide on 1:03:00. In vanilla attention, each token (within each head) has its own key, query and value vector. In multi-query attention (MQA), there is only one key and value vector for all query vectors. In between lies the MQA where a few query vectors (say 2-4) may be mapped to one key and value vector. - LLaMA uses SwiGLU activation function since it works better - LLaMA uses 3 layers instead of 2 for the FFNN part of the encoder block but keeps the number of parameters same.

Incredible explanation!> You have really predisposition to explaining materials. Keep going!

Amazing. Just wow. I cannot find this stuff on whole internet

Very good explanation! Keep up the good work!

Thanks a lot, Great explanation of KV cache and Multi Query Attention.

You actually explained attention here better than the previous presentation!

It's surprising that content of this quality is free.

very informative video, details are clearly explained. thanks a lot

Exceptional Video on LLaMA

Fantastic video, thanks Umar!

Thanks a lot. Looking forward to your next video!!!

Great video! Worth spending the time going over and over again. I actually saw your video from a Chinese site (the video probably has been forwarded by many other people already), then I come here for the author. 讲得超棒，谢谢分享！

You are doing god's work.Keep it up and Thank you.

great illustration!

Great explanation! 👌🏻

Great video. Thanks, 小乌!

Amazing explanation

Thank you for the amazing explanation

Great work man🙌

Thank you for posting Thank you existing Thank you

Thanks for the very resourceful video

u are doing great job, ty for tutoring me part by part! it helps a lot

really awesome work!

It's really useful. Thank you.

Brilliant, thank you.

Your video are great. Please keep going.

Great channel and content Umar

Thank you!

This is the way!

Despite your name deceived me at first, I had no doubt you were a fellow Italian!

Beautiful, thx.

Love your videos, and I also love the meowing cat in background😂

great explanation bro!!!

This video is great！

good content,nice explanation, THANKSssss.

another amazing video Umar! you do know how to teach for sure, it would be nice if you put into a repo very influential papers to read! Did I hear a baby in the background? Also given you are from Italy, there is a lovely video worth watching by Asianometry on 'Olivetti & the Italian Computer: What Could Have Been'. thank you again for the hard work you put on this video

Amazing video

Thanks!

Thanks for sharing

love it

thank you a lot

This is great. Thank you. It will be very helpful you could also create a video for hands on coding a Llama model, the way you did for vanilla transformer. Thanks in advance

amazing

Great video!!! Thank you very much for enriching the community with such great explanations! Can you please share your slides?

Genio!

nice，man

Hello , Many thanks for such a great content. Really enjoyed your work. Sorry for being greedy, may I request to create a video showing show llama model can be run effectively on local machines(with or without gpu) for inference(say with a custom flask api)

Thanks for the videos. There are a few errors in the video which I mentioned below: 1- Llama is an decoder-only model 2- The size of Q and K are the same. However, they are not the "same" tensor.

hi Umar! Your explanation is really excellent! By the way, llama3 has been released, will you continue to explain llama3 in a video?

Hi Umar, I recently started studying LLMs and I loved your explanations on transformers and the Llama architecture. I wanted to know that is there any way to look at the attention weights of a model and gain insights on which specific portions of the text influenced the output prediction? Is there any way to do this which is beginner friendly?

thanks! super helpful. Seems your channel didn't tell the GPT model architecture. would you be open to introduce that?

Great explanation for all levels. Que: did you say around 2:55 that its encoder only arch, i read that its decoder only?

❤

Many thanks for the great video. From your explanation Llama uses Rotary Positional Embedding (RPE) as positional encoding. It is applied to *each* Q and K vector just after transformation by their respective W. In this case, I don't get what the Relative Positional Encoding has to do with it (and why it was explained before Rotary PE). It is because Rotary PE has connections with the relative positional method, or both are applied in the case of Llama?

Hi, great explanations. btw, is there a chance you explain separately this torch.einsum operations shown in the code at time 58:41?

Great video +1. I have a few queries about the LLaMA model. 1. In the architecture diagram, does one NX represent a single layer for the LLaMA? 2. Could you also please clarify how many NX are utilized for the LLAMA-2 13B model and any variants? 3. Finally, what are the potential for distributed computation in LLaMA model inference? What are the possible breakpoints in the model from an architectural standpoint?

Valeu!

hey! it there any way you can avoid using background bangs when slide changes, please?

Thanks for the great video. at 48:56, you mentioned that we don't care about previous attentions. Does that mean we will trim the attention tensor from (SeqLen x d_model) to (1 x d_model) ? If so, does GPT do the trim as well? I thought GPT uses the whole attention tensor (including the previous attention "vectors") to predict the next token. That seems redundant but I wonder what they did exactly here. If they did use the whole attention tensor, does it mean that GPT inferencing needs to cache QKV instead of just KV? Thank you.

What is the size of the key and value matrices in grouped query multihead attention. In vanilla attention we were dividing it equally?

I'am wondering why multi-head attention with KV cache uses operations on O(bnd^2) in 1:00:01, isn't it on O(bnd + bd^2)? Could you please explain this or give some references to refer?

What I still didn't understand how grouping is happening in Grouped Multi Query attention? I didn't understand Rotary Positional Encoding concept, but will re-watch or read more?

Wowwwwwwwwwwwwww

I could not understand something: is token2 that we append to the Q after first inference equal to attention1 ???????

Can you explain where did you come up with time complexity of O(bnd^2) ? I personally think it's wrong, since performing just one scaled dot product attention should already make the quadratic term n^2 appear in the complexity formula, which is not your case. To give you more details into my calculations: Considering we have a batch size of 1, with one scaled dot product attention we are already making O(n^2d+nd^2) computations with d_k=d_q=d_v, and since it's perfomed h times we can say it's O(h*(n^2d+nd^2)). Am I wrong somewhere ?

49:10 "Since the model is causal, we don't care about the attention of a token with its successors" ~ I mean, a simpler explanation is also that the matrix is symmetric anyway, right? Like regardless of whether or not we care, it would be duplicated values.

1:01:42 great video! what is this beam 1 beam 4

Is the Rotary Positional Embedding just applied in the first decoder or in every decoder?

Hi Umar, can I ask you how do you get the LlaMa architecture, which paper is it from ?

How do i give you most of my money to make sure you keep making videos like this?

Isn' t the embedding size 768, 512 --the max seq length.

Llama is Decoder only plz check 2:50

Llama is only decoder model right ?

What does d model of 512 mean