When in doubt, set alpha = rank for the effective scale factor to be 1. There are better ways to have a larger impact on training than bluntly multiplying the change in weights, like improving your dataset or dialing in the learning rate.

@@EntryPointAI this does make sense they way you put it. Thanks so much for your reply!

I couldn't say for sure. Binary classification is a fairly simple task, so I would start with supervised fine-tuning.

Depends on the situation and how you want to apply the company information. If you mean to prevent it leaking in chats, I would suggest cleaning your data from anything sensitive before training an LLM on it or performing an embedding.

Thank you!

Not sure if that's a rhetorical question, but I'll give it a go. You can call it just "training," but that might imply that it's ready to do something useful when you're done. If you call it "pre-training" it implies that you'll train it more afterward, which is generally true. So it may be useful in being a little more specific.

Besides including the big text in a model's pretraining, you can fine-tune on it using empty prompts, which will make the model more likely to respond in a style similar to the writing. That doesn't necessarily make it an expert on the contents. In order to answer questions about a corpus, the typical approach is to chunk it up and use RAG. I have another video on the difference between RAG and fine-tuning.

Thank you for the info, it was very good explained for an introduction

I'm not familiar with voiceflow, so I'm not sure.

Multi-modal fine-tuning is definitely going to be a thing, but it's not available right now (at least, readily). For the time-being you would run an image through some OCR model to extract the text and then you could use that in a fine-tuned language model

Thank you very much!

Thanks and agreed. It shouldn't run the code just from opening a worksheet, but it would be worth looking into more. I added a version to the code repository that uses Google Sheets' built-in caching, but it has some of its own limitations. I was thinking that going through a proxy with caching would be beneficial. Open to any ideas to reduce API calls and token usage.

Your videos are fantastic. Because of the amazing value you deliver on KZfaq, I discovered Entry Point and became a customer.

Awesome! Thanks for sharing - see you on the Masterclass today 😉

The matrices are multiplied together and the result is the changes to the LLM's weights. It should be explained clearly in the video, it may help to rewatch.

@EntryPointAI My understanding: Orignal weight = 10 * 10 to form a two decomposed matrices A and B let's take the rank as 1 so, The A is 10 * 1 and B is 1 * 10 total trainable parameters is A + B = 20 In Lora even without any dataset training if we simply add the A and B matrices with original matric we can improve the accuracy slighty And if we use custom dataset in Lora the custom dataset matrices will captured by A and B matrices Am I right @EntryPointAI?

@@ArunkumarMTamil Trainable parameters math looks right. But these decomposed matrices will be initialized as all zeroes so adding them without any custom training dataset will have no effect.