Implementation video: kzfaq.info/get/bejne/n7-iZNuL05iagmw.html

This was beautifully explained. As someone who relies exclusively on self learning from online materials, the mathematical barriers of differig notations and overly complex wording is the most time consuming challenge. Thank you for such a distilled explanation, with only the notation and wording that makes sense for an intuitive and initial dive understanding. Subscribed!

I came across this video a month back, at that time, I didn't understand your excitement, though I understood the technique. Presently, I have better understanding of LLMs, and how they are trained and fine tuned for down stream tasks, and now I share your excitement.

The high-level intuition you gave at the end of the video was great. As a mathematician I'm aware of the theory behind low-rank decomposition and the classic applications, but the way it is applied in the context of LLMs is interesting.

The AutoGPT team is learning about LoRA and I recommended this because it is such a clear explanation. Thanks for the awesome resource!

This is the first video I watched from your channel, and I loved it! now I have 12 tabs open, to watch rest of your videos. Simply Amazing! (and WOW, I could not imagine LoRA can be so good. it will save me a ton of resources in developing the product I am working on.

I particularly appreciate the depth of research and preparation that clearly goes into this video. It's evident that you're passionate about the topics you cover, and your enthusiasm is contagious. Your dedication to providing accurate information while maintaining an accessible and entertaining format is commendable.

Thank you for thus nice, deep yet easy to follow explanation of LoRA. Nice job

Thank you so much Chris! This was an awesome video and has stopped me from going down the fine-tuning rabbit hole! Just dipping my toe into AI so it’s really great to find an informative channel like yours!

Thanks for the great intro to LoRA. I liked your graphics and your take-aways, also you energetic presentation :)

Best explanation of the whole KZfaq so far. Great job!

Woow the best explanation I've found so far.Coming from mathematics background it's really amazed me. Thank you

Really simple explanation without skimming through the intuition and other important details. Subscribed for more content like this :)!

Great explanation and first video I've seen from your channel. Somehow, I had the impression that you have 624K subscribers. Was shocked to see there's no actual "K" in there, when browsing your whole video history and seeing your channel is only ~2 months old. You'd deserve that "K" from my pov. Looking forward to the implementation video! (new subscriber)

Fantastic video and a really concise and well-considered explanation. Thanks!!!

Thanks a lot for this amazing explanation. I am fine tuning Mixtral 8x7B and using QLoRA have been able to perform test runs on Colab Pro using A100 machine.

Love your enthusiasm and great explanation!

Thank you very much. I didn't understand the paper so this helped me a lot. I'd be curious to hear about some of the limitations people are experiencing. I know it's not all completely characterized yet because this is new. I've noticed that Lora can redirect the type of output that I get but it's not very good at adding information to the model. This makes sense

Love your clear and passionate explanations!!!

Man that video is fireee! Thank you for your work!

Thanks a lot for this video! This is the first time I see a good explanation on this LoRA thing! 14:45 One minor note, is that it would indicate that the model has a low intrinsic info only if you could get rid of the original weights and just stick to the lora. That is, during the lora finetune training, if you could get away with while decaying the original (non-lora) weights down to zero. So I think that what has a low intrinsic info is "what you have to change from the base model" for your finetuning needs - but not the base model itself.

Great explanation! Keep up the great work!!

Beautifully explained my friend thabk you very much!

bears repeating that you should continue doing these videos, wish I had your communication skillz

Really a wonderful explanation. Best I've seen so far

Thank you for the explanation, It helps me a lot.

Love you Chris! It really helps me a lot!

Amazing explanation, Thank you!

One of the BEST explanations!!

You explained this very well. thanks!

Incredible video, just subscribed to your channel!

Dude, you're great at explaining stuff. 👍

Thank you for the good explanation of LoRA! Super!

Amazing explanation! Thankyou😊

I got here thinking this would be explaining how my waifu appears in the pretty image but instead I got a precise mathematical description of how this system works. Very good!

the best LoRA explanation I ever get. Better than ChatGPT.

great work dude please do some more videos god bless you

Chris, actually, what you are describing when you relate to this kind of application on the attention matrix, is specifically at inference, because the attention matrix is constantly being fine tuned (the only part of the GPT model to be constantly fine tuned at inference). This will enable to actually construct an AI that will enjoy the ability to think long term, since this will make use of so much less memory, so no need to reset the attention matrix so often. This, however, could be dangerous🙂.

Great explanation! I have gained an intuitive insight from your video. Thank you. Looking forward to your new videos! It seems that there is a missing x in the formula on the slide LoRA Explained (Cont.), which h should equal to w0x + wAwBx, if I understand your explanation correctly.

You are really good at explaining it. Keep it up

Amazing video! Thanks

Thank you so much! I'm working on a project to develop a chatbot for student advisory services and i am contemplating between two approaches: Fine-Tuning and Retrieval-Augmented Generation (RAG). Here are the key tasks the chatbot needs to address: Answering general queries about university courses and requirements. Providing personalized advice on study plans and schedules. Assisting with administrative processes (enrollment, documentation, etc.). Offering support for common academic and personal challenges faced by students. Given these responsibilities, which approach would be more suitable? Fine-Tuning could offer more precise, tailored responses, but RAG might better handle diverse, real-time queries with its information retrieval capabilities. Any insights or experiences with similar projects would be greatly appreciated!

Thanks. This was very useful

Absolutely brillant video!

this was explained so well. why can't the authors write their paper like the way you explained it.

Couple of basic questions: Is the 'h' or embedding layer an implementation detail related to LoRA? I thought the embedding layer is only used for the initial conversion of tokens into vectors during input feeding in the beginning Does LoRA only apply to the finetuning or training process? To prepare for inference, I'd imagine we'd combine the pretrained weights with the delta matrix and store that resulting matrix right? I wasn't really sure what you meant by the whole swapping out of weights for different problem types, because I assumed we just store the resulting weight matrix after training is complete.

Thank you Chris .

Hello ... new to the field of llm training .. thanks for putting up these two videos .. However , I am a bit confused by the comment ' ...you can do it during inference time' ? As per my understanding the weight updates are done during fine tuning ... and they are later used during inference ... if the task is changed we just revert back to the pertained weight by getting rid off the weight update for current task .. and fine tune the model based on the new task to get the new weight update ... the new update are then again used later during inference .. so the weight updates are during fine tuning only ... which I think why the authors mentioned that batch processing is not obtained by LoRA (base) ... though possible and difficult ... may be there is some future version where its implemented ? I am not sure .... but please correct me if I am conceptually wrong ...

amazing, thank you so much

Oh my God, thank you very much for this.

this saved me a ton of time. thank you!

thanks man, love the genuine!

love the enthusiasm!

Wow, what a great video!

@AlexG4MES 8 months ago This was beautifully explained. As someone who relies exclusively on self learning from online materials, the mathematical barriers of differig notations and overly complex wording is the most time consuming challenge. Thank you for such a distilled explanation, with only the notation and wording that makes sense for an intuitive and initial dive understanding. Subscribed!

Thank you so much for this video. The explanation is super clear. I do have a question though. When we reduce the WA and WB, does that wind up causing a lower fidelity in terms of the quality of inference results? Of course the difference is negligible, otherwise Lora wouldn't be useful, but I wonder if there is a loss, to what degree that might have a cumulative effect across many operations over time? In other words, yes, it's super that we are saving time and money, but is there a sense of at what cost overall? Thanks again for a great video!

great explanation!!

Dude. This video is amazing

how is Lora fine-tuning track changes from creating two decomposition matrix? How the ΔW is determined?

bless up nice work thank you

Could you explain why this saves memory? Don't you need the pre-trained weights in backprop to calculate the difference matrixes and during the forward pass?

Please make a playlist on llm and different architecture and there implementation..🙂

The one issue I have had is that this causes memory footprint to grow. But it sounds like you should be able to merge it into the base model at the end to keep the same footprint. Maybe that is something for me to try. I wonder if this low rank decomposition can be used for model distillation. Instead of just quantizing weights.

Hi a question, can we use lora to just reduce the size of a model and run inference, or we have to train it always?

Thank you

Hi Chris, just trying to understand. Assume that I have two downstream tasks, and I have two adapters trained for these tasks. During inferencing and I want to use both of them simultaneously. Will LoRA allow me to do this with one base model along with the two different adapters ? Basically, can I use multiple adapters simultaneously ?

So it's basically a linear algebra trick. Kinda sorta like pricipal component analysis. Very interesting. I can see this is essentially lossless compression post-training, but I have to wonder about subtle effects on the quality of the model during training as it's left with fewer parameters to discover features.

thank You

First of all, thanks for the video. I know its just a minor thing but still for the mathematical correctness, the equation in the LoRA explained slide should be updated as: h = W_o x + W_A W_B x Annoying reviewer1 out ! :)

I wonder why this is such a recent and new concept. Matrix decomposition applied to machine learning models. Like, matrix decomposition has been known to work really well for so many different applications and tasks. How is it possible that we only thought about applying it to machine learning recently? Also, the authors talk about fine-tuning with low rank adaptation, but why are we not assuming low rank for the whole training process as well (meaning NOT from pretrained). If it's worth it to decompose a model before starting to train it, then the implications are huge for machine learning as a whole.

cool video tahnk you!

Follow up video on LyCORIS locon?

this is a masterpiece

But dW matrix changes with each propagation. When do we determine how many dimension is needed to represent dW?

LoRA is a game changer

Nice!

what does the author mean by "parameter budget"? which in this case is 18M.

Hmmm.. Isn't it also extremely faster inference, due to that little thing called "the associative law of matrix multiplication"?

Great!

Why does deltaW need to be represented by both WA x WB? Why couldn't it be represented using just smaller matrix?

love your explanation it helped me a lot

how is Lora different from PEFT? or are they related?

Отличный туториал)

What's the difference between less parameters, and low intrinsic weights? because weights are parameters of Neural Net isn't it?

@15:28, there is nothing great about adding the extra LoRA parameters to the weights that makes it easier to swap the behaviour of the model at inference time because the difference between adding the matrices and loading the entirely new weight matrix from different finetuned models to the model architecture is negligible.

What’s with the latency, I didn’t get it. Can’t we swap with traditional fine tuning, we store delta_W but it’s just way bigger than A & B.

what is "prefix-based approaches"?

If the key and query matrices are of dimensions m×n but have rank r=max(rank_k, rank_q) in the pretrained net then why don't we transform these matrices into m×r matrices?

sub’d and even hit the spam bell

It doesn't train the entire Lora model, so I came up with an idea to divide the model and train each part on Lora in each epoch. Wouldn't this approach, which requires less RAM like Lora and involves full fine-tuning, yield the same results?

Is this the basis of why we think model distillation works?

Subscribed ;-)

good

What about smaller AIs networked together

Holy shit this is huge

Is it feasible to exchange the pre-trained model as well, as you explained for LoRa in 16:40? By the way, amazing content! Subscribed!

So I don’t get why, if the weight matrix ranks is much less than the original matrix. Why doesn’t the larger mega model just use the low rank matrix instead of their monster weight matrix?

Yes yes yes - D, K, matrix, 100 etc etc etc - but what does actually LoRa do?

as clear as mud

doing a f'cking god's work