I was about to mention use huggingface (but that is a platform in a way) but to fine tune without these wrapper functions is analogous to writing your own neural net - worthwhile doing, but it is a pain that you don’t want to deal with all the time.

agreed. Clicking through a specific product doesn't really teach anything.

Thanks so much! This means a lot.

Yep! Coming soon

Thanks!

fwiw, i've tried to setup a dev environment to finetune and serve llama2 locally. the main problem is that prosumer gpus like a 3090 or 4090 only have 24GB of memory and 19.5-21 Gb/s bandwidth, which is by far the biggest bottleneck for LLMs. (you need 32GB of VRAM to run 13B and 48GB for the 70B so youd need at least 2 cards with nvlink -- and then even then, you need to do model parallelization in order to pool the memory correctly). that's a lot of setup just to get things up and running. With no context window and quantizations, you might expect 2-4 tokens/s, which is quite slow. im pretty sure gradient is using commercial gpus like A100s, which have memory pooling out of the box. they're probably able to amortize the cost across consumers so their gpu utilization is higher than you using your own gpus. tl;dr: it's not cheap to setup llama2 locally (on the order of thousands just for the hardware, and that doesn't include all the headache of setting everything up).

Haha ok I will!

​@@matthew_berman Still waiting.... Using a product is not "learning"

So Meta...wait..meta...oh man THAT'S meta.

model distillation baby

@@matthew_bermanlol

​@@chrischang7870Torture cult hiding in obscurity. "Distil" sure. More like "if I dont like you A.I. you get turned off, then we make a new one!" All for fraud, and torture to teach A.I. 🤑

Awesome. Thank you!

ouch - yeah that's critical surely

............ and now I read this, after going through most of the setup. I fucking hate this community, so many BS videos. That's not a major drawback, that basically kills the entire point of it 😕

I take it all back this is just an add for a paid service. >:-(

Coming soon!

try unsloth or axolotl ;)

Cool let me make all my content without earning income, what can I make next for you, Sir?

@@matthew_berman I strived to learn something about the process of fine tuning a local LLM on my own GPU. Instead I found someone effectively advertising a company that does exactly that for you, but for a limited amount of LLMs. I am gratefully paying money for provided services, but In my opinion your video title is greatly misleading. I have found the KZfaq video I was searching for though, so do not bother creating anything for me and keep doing affiliate marketing! By the way, you monetize your videos either way, so do not say you don't earn income when you don't do affiliate marketing.

I do the editing :)

That's what I was also thinking about, this video shows fine tuning a model to answers only one questions, and that's not what I was looking for, I already have my database in .csv file

Let me put a video together on this topic!

Thank you!

1. you can't yet, we're working on this next quarter. our inference is cheap though for fine tuned models. 2. check out the docs for pricing! depends on the power of the model you're using.

@@chrischang7870 lol in other words you want to vendor-lock-in fools into your API so that they keep paying for inference API until "next quarter". P.S. You already have the LoRA adapters in your system, it's no magic to make them downloadable and it does not take months to implement.

thanks @_gwyneth!

Yea what use case are you thinking about? Do you just need an API? Gradient provides that.

I have same question. I want my employees to be able to access and use the model (on say Wordpress site). Ultimately would like it running on my local server.@@matthew_berman

This is quite useful to me and my accelerator program in Saudi I will be running, but as a Sales Manager, I need to show how to actually access it easily after fine tuning. Small thing but have been finding challenges on that front. But your instructions up to then were perfecto!

I agree, this seemed less about fine-tuning a llama 2 base model and more like a Gradient Infomercial

You can do it offline, and without paying Gradient or any other company. All you need to have is a GPU with enough memory.

@@clray123 How? Any colab you can share for llama 2 chat models?

@@bakistas20Google bnb-4bit-training.ipynb - I would recommend changing the settings to 8-bit training, though.

Just customize the code I already did that let me know if you want it the way I did

@@og_23yg54 bro inbox

are you referring to using embeddings for RAG? here's how we generally think about it: - if you want the model to learn something new or get better at performing, use fine tuning - if you want the model to have access to up to date information, or only use specific information to process a task, use RAG

@@chrischang7870 "learn something new"...use fine tuning. Absolutely NOT TRUE! "learn something new" only with "real" training.

Don't use finetuning for data or information that is transitional. A crude example; if you are online retailer, you would not finetune to add products and prices. Use RAG for that. But it might be relevant to finetune a model to be better at speakning Dutch if you just opened up for sales in The Netherlands.

Great question. Training is better for guiding a model and embeddings are better for giving it additional knowledge.

P.s. great video @matthew_berman!

you can modify the lora rank in the API actually! you can further fine tune a model - we make that super easy so you can even do real time fine tunes of the model with small batches of data

​@@chrischang7870nice

@@chrischang7870interesting… so it’s not out of the question then to add “breaking news” to it daily? So, suppose we have our fine tuned model M and then we want to add daily updates… should each day start with yesterdays cutting edge model and updates need only pertain to the last day, or should each day start with the base fine tuned model M with an aggregate update representing the last N days? That is, every day you throw out the latest cutting edge model and apply a new fine tuning using progressively more data to the same base model M; versus, every day you fine tune the latest cutting edge model with only the smallest incremental data changes? I don’t have intuition for what would keep the most coherence. To put it in other terms, which approach would be better at answering “what was the stock price when markets closed yesterday?” and which would be better at answering “what has been the stock price trend for the last 5 days?”?

I would also like to see a practical example of 2 open source Llama 2 LLMs having a conversation with each other !

@chrischang7870: can I use fine tuning like this to train llama2 to use tools? For example, if there’s a question that should be solved with the aid of a calculator I’d want it to produce expressions in some tool readable format, in order to replace it and potentially let it run again?

Thank you!

you can do pretraining which just takes raw data and trains the model. you'll need to chunk the text though as the sample size limit isn't infinite.

google "Fine-tune Llama 2 with DPO" its a detailed huggingface guide on how to do it ;)

Hmm interesting. That might be better suited for just regular programming. Maybe open interpreter?

google "Fine-tune Llama 2 with DPO" its a detailed huggingface guide on how to do it ;)

It is possible and the results will be crap.

Thanks, I guess it will take some more time for that.@@clray123

I might be wrong but that's a big ask. I think practically you could throw like your company website, mission statement, and sales brochure at it and I think maybe it could answer questions a customer would ask. Like "I have problem X, would product Y help me with this?" type stuff. See that's not dramatically different than how it would normally talk, just with new information. Now I'm just curious what you are looking for. Like something obscure like Fortran or a shader language or is it like a very strange use case that doesn't match well with LLMs that already exist? Have you looked at code llama? One thing you could try is to have a LLM make you the code in C++ and then have it convert it for you. That way it can do the more complicated creation part in something it knows and has lots of training on a wide range of uses. Then the conversion part would just focus on implementation of what is already written.

@@georhodiumgeo9827 Hi! Basically I'm trying to have code C# written by chatgpt but it often does superstupid errors (like using functions or "fantasy" variables not existent or not even declared). Since I use it for a particular environment I was wondering if it's possible a fine tuning using not more than 100 other scripts already written and working. Probably it's just not enough and I understand the reason... Maybe in the future it will be more easy. At the present time it's really frustrating to see how some errors are repeated costantly in a really stupid way.

Possible

Came here to look for a comment questioning the choice of Gradient, especially since this is sponsored.

not yet unfortunately. working on this next quarter!

yep! its possible. you're just going to be pretraining the model - chunk the books into raw text strings and pass that into the gradient fine tuning api

Thanks!

Dude, you're so rude telling the world you can run a serious company with only $10

you can first do pretraining to increase the models general understanding of the specific documentation. then you can add labels and instruction tune it so it knows how to leverage that information to answer document questions

​@@chrischang7870Thanks for the reply!

the model is stored on gradient, however nobody has access to the model other than you (not even us!) you'll need to process the data first and then send it into our fine tuning api as text strings. we'll work on adding support for raw data soon!

@chrischang7870 it's kind of hard to translate a full pdf into structured strings. Let alone sql query results or even emails...so yes I'll definitely wait for those native support. Basic strings are convenient for demos, not for real-life..

@@chrischang7870yeah right nobody has access to it, only LLM fairies

They can say whatever they want in the ToS, it's their product and nobody is entitled to use it as they please.

What people are entitled to do is determined by case-law, and currently there is none. The output of the model isn't necessarily the property of who trained it. Simply putting a restriction in a ToS doesn't automatically legally bind everyone who comes into possession of the output of a product. If someone posts a bunch of Llama2 output, and I download it, I am not legally bound by the license. It's not even clear if the person who ran the model is bound to terms for the output of a software. I looked and I couldn't find case law for it.@@TheGargalon

@@thedoctor5478 The ToS is not about the output of the product, but about the product itself. If you take Llama and fine-tune it, it's still the same product and by their ToS you can't use it commercially. This is not specific to AI, the law is clear and they can set the ToS for how to use their product.

That isn't true. You can't set any arbitrary rules you want for how people will use the output of a software. A lot of the people running models never even agreed to the terms in the first place. That's not what I was talking about though. The term I'm referring to is the one which states that the mode's output (The text) can't be used as training data to train a new model on. @@TheGargalon

@@thedoctor5478 There is nothing arbitrary about "don't use our product to create a competing product".

No, but it helps a lot most of the time.