Low-Rank Adaptation - LoRA explained

Рет қаралды 9,320

Күн бұрын

RELATED LINKS
Paper Title: LoRA: Low-Rank Adaptation of Large Language Models
LoRA Paper: arxiv.org/abs/2106.09685
QLoRA Paper: arxiv.org/abs/2305.14314
LoRA official code: github.com/microsoft/LoRA
Parameter-Efficient Fine-Tuning (PEFT) Adapters paper: arxiv.org/abs/1902.00751
Parameter-Efficient Fine-Tuning (PEFT) library: github.com/huggingface/peft
HuggingFace LoRA training: huggingface.co/docs/diffusers...
HuggingFace LoRA notes: huggingface.co/docs/peft/conc...
⌚️ ⌚️ ⌚️ TIMESTAMPS ⌚️ ⌚️ ⌚️
0:00 - Intro
0:58 - Adapters
1:48 - Twitter ( / ai_bites )
2:13 - What is LoRA
3:17 - Rank Decomposition
4:28 - Motivation Paper
5:02 - LoRA Training
6:53 - LoRA Inference
8:24 - LoRA in Transformers
9:20 - Choosing the rank
9:50 - Implementations
MY KEY LINKS
KZfaq: / @aibites
Twitter: / ai_bites
Patreon: / ai_bites
Github: github.com/ai-bites

Пікірлер: 18

@gelly127 3 ай бұрын

Underrated channel, keep making videos and itll eventually blow up

@AIBites 3 ай бұрын

Sure. Thanks for the encouraging words 👍

@benkim2498 Ай бұрын

Super in depth and specific, thank you!!!

@mr.daniish 3 ай бұрын

Amazing video

@AIBites 3 ай бұрын

Glad you think so! 😊

@dileepvijayakumar2998 Ай бұрын

this is better explained than what the inventor of Lora itself explained in his video.

@talmaimon4534 5 ай бұрын

Thanks for the video! I loved that you added some libraries we can use for this.

@AIBites 5 ай бұрын

do you want me to do more videos on hands-on? Or should I continue on the theory and papers? your inputs will be quite valuable :)

@jacobyoung2045 5 ай бұрын

@@AIBites Hands on videos will be great too

@unclecode 7 ай бұрын

Good job on the clear explanation of the method and simplification. At 3:40, when you showed the matrix decomposition, the result on the left side does not match the result on the right side. Is this a mistake in the video editing, or is there a point to this? [1 2 3] x [2 20 30[ should be [[2. 4 6], [20 40 60], [30 60 90]]

@AIBites 7 ай бұрын

ah yeah! super spot! I got that wrong while editing. Sorry... 🙂

@ananthvankipuram4012 5 ай бұрын

@@AIBites Yup the Matrix should be [1/2/3] * [ 2 20 1]

@AIBites 4 ай бұрын

Thanks again :)

@abdelmananabdelrahman4099 7 ай бұрын

wow u r great 😄

@AIBites 7 ай бұрын

Thank you! I am chuffed :)

@pshivaramakrishna 4 ай бұрын

Very Well Explained! If ΔW's dimensions is 10 x 10 , A and B dimensions are 10x2 and 2x10 respectively. So, instead of training 100 params we only train 40 params (10x2 + 2x10). Am I correct ?

@AIBites 4 ай бұрын

yup you got it right. And based on the compute available, we can adjust the rank ranging from say from as low as 2.

@pshivaramakrishna 4 ай бұрын

@@AIBites Thanks for the confirmation.