All You Need To Know About Running LLMs Locally

  Рет қаралды 129,848

bycloud

bycloud

Күн бұрын

RTX4080 SUPER giveaway!
Sign-up for NVIDIA's GTC2024: nvda.ws/48s4tmc
Giveaway participation link: forms.gle/2w5fQoMjjNfXSRqf7
Please read all the rules & steps carefully!!
1. Sign-up for NVIDIA's Virtual GTC2024 session between Mar 18 - 21st
2. Participate the giveaway DURING Mar 18 - 21st
3. ???
4. Profit
TensorRT LLM
[Code] github.com/NVIDIA/TensorRT-LLM
[Getting Started Blog] nvda.ws/3O7f8up
[Dev Blog] nvda.ws/490uadi
Chat with RTX
[Download] nvda.ws/3OHPRHE
[Blog] nvda.ws/3whKZTb
Links:
[Oobabooga] github.com/oobabooga/text-gen...
[SillyTavern] github.com/SillyTavern/SillyT...
[LM Studio] lmstudio.ai/
[Axolotl] github.com/OpenAccess-AI-Coll...
[Llama Factory] github.com/hiyouga/LLaMA-Factory
[HuggingFace] huggingface.co/models
[AWQ] github.com/mit-han-lab/llm-awq
[ExLlamav2] github.com/turboderp/exllamav2
[GGUF] github.com/ggerganov/ggml/blo...
[GPTQ] github.com/IST-DASLab/gptq
[LlamaCpp] github.com/ggerganov/llama.cpp
[vllm] github.com/vllm-project/vllm
[TensorRT LLM] github.com/NVIDIA/TensorRT-LLM
[Chat with RTX] www.nvidia.com/en-us/ai-on-rt...
[LlamaIndex] github.com/run-llama/llama_index
[Continue.dev] continue.dev/
Model recommendations:
[Nous-Hermes-llama-2-7b] huggingface.co/NousResearch/N...
[Openchat-3.5-0106] huggingface.co/openchat/openc...
[SOLAR-10.7B-Instruct-v1.0] huggingface.co/upstage/SOLAR-...
[Google Gemma] huggingface.co/google/gemma-7b
[Mixtral-8x7B-Instruct-v0.1] huggingface.co/mistralai/Mixt...
[Deepseek-coder-33b-instruct] huggingface.co/deepseek-ai/de...
[Madlad-400] huggingface.co/jbochi/madlad4...
[Colbertv2.0] huggingface.co/colbert-ir/col...
This video is supported by the kind Patrons & KZfaq Members:
🙏Andrew Lescelius, alex j, Chris LeDoux, Alex Maurice, Miguilim, Deagan, FiFaŁ, Daddy Wen, Tony Jimenez, Panther Modern, Jake Disco, Demilson Quintao, Shuhong Chen, Hongbo Men, happi nyuu nyaa, Carol Lo, Mose Sakashita, Miguel, Bandera, Gennaro Schiano, gunwoo, Ravid Freedman, Mert Seftali, Mrityunjay, Richárd Nagyfi, Timo Steiner, Henrik G Sundt, projectAnthony, Brigham Hall, Kyle Hudson, Kalila, Jef Come, Jvari Williams, Tien Tien, BIll Mangrum, owned, Janne Kytölä, SO, Richárd Nagyfi, Hector, Drexon
[Discord] / discord
[Twitter] / bycloudai
[Patreon] / bycloud
[Music] massobeats - magic carousel
[Profile & Banner Art] / pygm7
[Video Editor] maikadihaika

Пікірлер: 245
@danielmadstv
@danielmadstv 5 ай бұрын
Thanks for the video! Minor correction: GGUF is not the predecessor to GGML, GGUF is the successor to GGML.
@ambinintsoahasina
@ambinintsoahasina 5 ай бұрын
The amount of infos you give both in the videos and the descriptions is insane dude! Keep up the good work!
@lunadelinte
@lunadelinte 4 ай бұрын
that was awesome, thanks for the concise information bycloud! 🔥
@papakamirneron2514
@papakamirneron2514 5 ай бұрын
Immensely helpful video. I hope the future has tonnes of user controlled locally ran llms for us in store!
@flexoo7
@flexoo7 5 ай бұрын
Poor Faraday nearly always gets overlooked when people talk about local LLMs, but it is without a doubt the most easy to use "install and run" solution. Unlike nearly all other options it's near-impossible to mess something up and default settings out of the box are not sub-par.
@hablalabiblia
@hablalabiblia 4 ай бұрын
How much is Faraday?
@flexoo7
@flexoo7 4 ай бұрын
@@hablalabibliaLike all the best things in life - it's free.
@joure.v
@joure.v 4 ай бұрын
@@hablalabiblia It's free and very easy to use! It's really meant just for chatting, it's basically a Silly Tavern kind of app, just not with that many options but it has its own back end with a focus on GGML models. If you're looking to just run models through character cards I'd say, give it a go!
@Elegant-Capybara
@Elegant-Capybara Ай бұрын
Faraday has outdated models and whenever you download models, you have to fumble with model cards and directory structures, plus it's not as fast as other options. LM Studio is better than Faraday.
@RetroPolly
@RetroPolly 4 ай бұрын
A thousand thanks! Finding a good LLM model was a complete nightmare for me + it is difficult to figure out which formats is outdated and which - new hot stuff.
@bycloudAI
@bycloudAI 2 ай бұрын
stay up-to-date on the latest AI research with my newsletter! → mail.bycloud.ai/ Minor correction: GGUF is not the predecessor to GGML, GGUF is the successor to GGML. (thanks to danielmadstv)
@Sebastian-oz1lj
@Sebastian-oz1lj 2 ай бұрын
please make step by step guide how to install locally and private for example Mistral-7B. im trying to do this with multple guides and all time im stuck at something
@bossdaily5575
@bossdaily5575 5 ай бұрын
Nice video! Can you do a video about fine tuning a model?
@johnsarvosky533
@johnsarvosky533 4 ай бұрын
Thanks, this is great. Please make a comprehensive video on Fine-tuning locally 101..Cheers
@robertmazurowski5974
@robertmazurowski5974 3 ай бұрын
I was pretty sure this was a fireship video, but the video is great and informative. Exacly what I was looking for.
@vladislava5237
@vladislava5237 5 ай бұрын
Very nice, tons of useful info Thank you!
@H1kari_1
@H1kari_1 5 ай бұрын
I love your adhd-friendly edits cloudy.
@Leo_Aqua
@Leo_Aqua 5 ай бұрын
You can also use ollama. It even runs on a raspberry pi 5 (although slow)
@siddhubhai2508
@siddhubhai2508 16 күн бұрын
Yeah you're right OLLAMA can be run on the raspberry pi 5 even, but don't forget that ollama is made for using local llms, and if you try to run local llms like llama 3 or deepseek, ready for FBI at your home catching you for building a unknown b*mb. Important life lesson - FIRST TRY THEN CRY. GOOD LUCK! 💣
@juanantonionieblafigueroa377
@juanantonionieblafigueroa377 4 ай бұрын
Your videos are way more fun than my algebra homework
@lintalyor6535
@lintalyor6535 Ай бұрын
I like this simple explanation with the video editing thanks!
@Kevin.Kawchak
@Kevin.Kawchak 2 ай бұрын
Thank you for the discussion.
@trolik9113
@trolik9113 3 ай бұрын
Absolutely fantastic and informative video. Well done! I will say I feel like the information certainly speaks to the grip that OpenAI has, especially from a development standpoint, despite the whole video being about open-source models. The procedures, time, research, and money required for any rando or small (even mid size) business owners to integrate open-source and local AI without any practical knowledge about it is near impossible. OpenAI wraps up RAG, "fine-tuning", and memory nice and neat into Assistants which can be easily called via the API. It would be amazing to have a completely standardized system that allows for the same type of application, but geared towards the variety of open-source models out there. Some platforms like NatDev let you compare multiple models based on the same input. Being able to see how RAG and fine tuning affects different models, both open-source and non, from the same platform would be unreal.
@NeostormXLMAX
@NeostormXLMAX 4 ай бұрын
but anyways this video was very helpful because no one made it very clear on what are the best front end interfaces to install, I kept trying to make one myself to no avail and give up after a while after testing stuff in the command prompt
@noobicorn_gamer
@noobicorn_gamer 5 ай бұрын
I hoooonestly don't know how to feel about the thumbnails looking too similar to you-know-who that got me accidentally clicking this video but meh... One's gotta do what one's gotta do I guess.
@EdissonReinozo
@EdissonReinozo 5 ай бұрын
Same
@Deathington.
@Deathington. 5 ай бұрын
I don't know who, who?
@nathanfrandon2798
@nathanfrandon2798 5 ай бұрын
@@Deathington. Fireship
@NIkolla13
@NIkolla13 5 ай бұрын
Bycloud removed the frame and the grid background on his thumbnails, I think those work great as his signature style. I hope he keeps them
@seanrodrigues8184
@seanrodrigues8184 5 ай бұрын
Let's just hope he doesn't get _burned~_
@aketo8082
@aketo8082 3 ай бұрын
Thank you. Very interessting. Is it possible in LM Studio to work with own files? Or create own LLM or extend LLM for own cases?
@Veptis
@Veptis 5 ай бұрын
Now we just need a cheap inference card with 128GB memory to run 70B models locally... Maybe we can hope for Qualcomm
@cbuchner1
@cbuchner1 5 ай бұрын
I’d love to see AI inference accelerator cards with dual or quad channel DIMM slots.
@Veptis
@Veptis 5 ай бұрын
@@cbuchner1 Qualcomm AI 100 Ultra is using LPDDR5
@nyxilos9167
@nyxilos9167 5 ай бұрын
groq is using something of the sort, an LPU. although only usable through an api. no consumer cards yet that i know of, but it shows the trend towards it
@Veptis
@Veptis 5 ай бұрын
@@nyxilos9167 you can buy a single groq card right now. it costs 21k and has 230MB on board. So to run 70B models at fp16 you need like 572 cards.... which is several racks. 14+ million to buy and 30kW to power. It will run the model at 400 tok/s easily. You can buy a ready made 8x H100 box for maybe 350k and run that with like 8kW and it might be slower than the groq card. none of that are consumer solutions. The one I am hoping for is Qualcomm AI 100 Ultra. Which comes with 128GB LPDDR5 and 150W. They say it's for edge inference, but it would be perfect for workstation.
@Vifnis
@Vifnis 4 ай бұрын
idk Qualcomm SoCs are for phones mostly... maybe iPhone 30 will have it XD
@kernsanders3973
@kernsanders3973 4 ай бұрын
In regards to context, would LLM Lora's help with that? Lets say im busy with story writer LLM and the fantasy world I'm working with would be as big as something like Middle Earth from LOTRs. Would a Lora help with that? Like if I train a Lora on all our past chat history about the story etc. Also more text regarding the lore of places and history of characters and family trees. So taking that into consideration, would that assist in keeping the context low? So I don't need to keep a detailed summerized chat history etc. What would the requirement be for training such a Lora and what would the minimum text dataset require for a coherent training?
@joseph-ianex
@joseph-ianex 4 ай бұрын
with local models are you able to make much longer responses given that you have enough ram and vram?
@frazuppi4897
@frazuppi4897 5 ай бұрын
what about ollama as a backend, what is your take on that? Thank you so much for the video, sending love from switzerland
@christianremboldt1557
@christianremboldt1557 5 ай бұрын
Stup osing Fireship thumbnails😭
@idk-dk7bq
@idk-dk7bq Ай бұрын
Y
@aouyiu
@aouyiu 17 күн бұрын
Stop neglecting proofreading comments 😭
@christianremboldt1557
@christianremboldt1557 17 күн бұрын
@@aouyiu I apologize. I normally proofread all my comments, but I suspect that I was drunk while writing this one. As I don’t like editing comments afterwards, I didn’t change the spelling mistakes.
@rougeseventeen
@rougeseventeen 2 ай бұрын
thanks, this videos is very funny and helpful!
@a.........1._.2..__..._.....__
@a.........1._.2..__..._.....__ 4 ай бұрын
Ive been hamfisting my way through llms for over year. Just ramming squares into circles till it worked since informations so sporadic. 100% checking out your other videos. Learned more in 5 min then 4 hours reading github docs
@Paulo-ut1li
@Paulo-ut1li 5 ай бұрын
Boy, Chat with RTX is my personnel oracle for now on. Its RAG really indexes local documents without that whole hallucination from previous tools.
@the_gobbo
@the_gobbo 5 ай бұрын
I can finally start my side project to take over the world, thanks!
@magfal
@magfal 5 ай бұрын
The one thing I hope to see soon is offloading different layers to different GPUs I have a 4090 mobile in my laptop and an RX6800 in my eGPU. I do have 96GB of system memory in addition to these two 16GB cards so I can do some fun stuff already.
@shoddits2156
@shoddits2156 5 ай бұрын
does the a Giveaway has country restriction?? I mean maybe you can't send it overseas due to shipping cost or something else.
@krzysztofmaliszewski2589
@krzysztofmaliszewski2589 5 ай бұрын
That's a great question.
@dustindustir521
@dustindustir521 5 ай бұрын
Step 4 is Clear, but How can I unlock step 3? I only see questionmarks. Do I have to do step 1 and 2 to unlock what I have to do at step 3, Or do I just need to gain more XP for the unlock. Maybe I just have to do step 4 twice to make up for the missing third step...
@christopheralvarez1090
@christopheralvarez1090 5 ай бұрын
Where do I upload the photo once GTC comes around ?
@WINTERMUTE_AI
@WINTERMUTE_AI 5 ай бұрын
LM STUDIO and TRINITY 1.2 is my favorite non-GPT entities!
@RedOneM
@RedOneM 5 ай бұрын
What 3 models do you recommend with 24 GB VRAM? Preferably 21-22GB / 24GB in practical usage.
@nyxilos9167
@nyxilos9167 5 ай бұрын
huggingface lists models with their respective memory requirements. any 7b model will likely work very well and be under 21gb. you could also go with a bigger model but at a lower quantization. mistral models are among the most popular, open source, and very competitive.
@fxstation1329
@fxstation1329 4 ай бұрын
I'm a noob when it comes to this. I've come across Ollama, and started using it. Can I upload multiple things, texts, and possibly images, to chat with RTX and create my own data? And will it be uncensored? what are some other good options to 'Chat with RTX'
@bigglyguy8429
@bigglyguy8429 5 ай бұрын
How did you miss Faraday? Very easy to use and runs faster than LM Studio
@squfucs
@squfucs 4 ай бұрын
i run LM Studio and i think its great, good video my dude
@valeriapadilla5860
@valeriapadilla5860 4 ай бұрын
Hope this works better than the time I tried to download more RAM
@user-js4mt5cd2t
@user-js4mt5cd2t 4 ай бұрын
Good to know.
@itisallaboutspeed
@itisallaboutspeed 5 ай бұрын
as a car content creator i approve this video
@user-uo1fj8mo1l
@user-uo1fj8mo1l 4 ай бұрын
Finally!!!!
@Paulo-ut1li
@Paulo-ut1li 5 ай бұрын
Please make a video on how to fine tune a model using local documents.
@user-ud9su7qn1x
@user-ud9su7qn1x 3 ай бұрын
I don't have strong GPU , do you reccomend any sevices that i can run models on .
@pedrogorilla483
@pedrogorilla483 5 ай бұрын
Where ollama?
@sZenji
@sZenji 5 ай бұрын
agree, with the new windows installer its so easy for everyone to get local models
@4.0.4
@4.0.4 5 ай бұрын
For a while it was only Mac-based, so it saw limited use with most AI folks who have Nvidia cards. If you're stuck on a Mac I hear it's really the better one for that.
@zikwin
@zikwin 5 ай бұрын
wow now on support windows too ?@@sZenji
@babbagebrassworks4278
@babbagebrassworks4278 4 ай бұрын
I use it on my Raspberry Pi5 to run LMM's, which is seriously cool, er hot when working.
@u13e12
@u13e12 2 ай бұрын
Just to clarify then. For inference speed is more important GDDR6 will be GDDR5, but for fine tuning more more having 2x the amount of GDDR5 will be the GDDR6?
@Anthonyg5005
@Anthonyg5005 5 ай бұрын
EXL2 does support AMD GPUs. Turbo bought a couple just to make sure it runs with rocm
@narpwa
@narpwa 5 ай бұрын
what happened to the newsletter ????
@harrychaklader
@harrychaklader 4 ай бұрын
We love LM Studio 😫
@samuelpeery
@samuelpeery Ай бұрын
Total newbie with running an LLM locally. What is the best llm for summarizing books and being able to ask questions about the books?
@XdekHckr
@XdekHckr Ай бұрын
What would be the best llm for math?
@WW-ir7sm
@WW-ir7sm 5 ай бұрын
How hard is it to run LLM with AMD GPU? Is it still Linux only hell bc no driver support?
@diewindows5628
@diewindows5628 4 ай бұрын
rocm works for some stuff on windows, just don't expect to be on the bleeding edge with new features
@shakta108
@shakta108 2 ай бұрын
What do you think of phi model ?
@keffbarn
@keffbarn 3 ай бұрын
OOOGABOOOOGAAAAH 💪😎🍺
@cristianionascu
@cristianionascu 4 ай бұрын
I guess my machine is not good enough, 2019 intel imac, because running any model locally is usually lagging way behind ChatGPT 3, Gemini, Perplexity, etc.
@plagiats
@plagiats 5 ай бұрын
Ollama + openwebui is the way to go. Same ui as ChatGPT, plenty of convenient functions. It's a no brainer.
@NoidoDev
@NoidoDev 24 күн бұрын
The important part for me is accessing it from CLI or Python. Ideally, doing the whole configuration in there. Because I need it to be automatized (no NodeJS of course).
@mrrespected5948
@mrrespected5948 5 ай бұрын
Nice
@Lakosta826
@Lakosta826 3 ай бұрын
How can I unistall text generation web ui? Anyone know this??
@siddhubhai2508
@siddhubhai2508 16 күн бұрын
Oh Fireship's second hidden channel! 😂😂
@dungeon4971
@dungeon4971 5 ай бұрын
what about ollama
@harrychaklader
@harrychaklader 4 ай бұрын
you're awesome
@exaltedjoseph7963
@exaltedjoseph7963 4 ай бұрын
Your thumbnail reminds me of fireship
@D0J0Master
@D0J0Master 5 ай бұрын
How do local models compare to cloud ones like openai? Wouldnt a local pc have way worse results? A server farm can have way more vram and hence is better?
@joseph-ianex
@joseph-ianex 4 ай бұрын
I'm getting ~gpt 3.5 performance on my laptop with 16gb ram and rtx 3060. I'm primarily using it because I feel like commerical ai chatbots are getting more and more censored
@MrBoxerbone
@MrBoxerbone 4 ай бұрын
@@joseph-ianex Can you share which model are you using?, I have a laptop with those exact specs
@joseph-ianex
@joseph-ianex 4 ай бұрын
@@MrBoxerbone *rtx 3050 ti. Most 7B models run fine, you can try Mistral, Gemma, or Llama 2. Get either ollama (command line) or llm studio (ui) to run the model. If you are new to running models I would recommend llm studio. The models are a bit slow and the context window is pretty small but they run. Pinokio is another cool ai if you want to test out open-source AI art tools 👍
@ericcadena2030
@ericcadena2030 3 ай бұрын
Where is the diagram at 8:50 from?
@Rookie_AI
@Rookie_AI 4 ай бұрын
is it possible though to run LLM on iOS?
@mzafarr
@mzafarr 2 ай бұрын
Please make a video about making our locally running LLMs available for others to use maybe like our own API which people can use or a webUI interface to use our local LLM.
@Necessarius
@Necessarius 5 ай бұрын
Fireship thumbnail is working for me
@hyposlasher
@hyposlasher 5 ай бұрын
2:17 Bro lives in the future where M4 is already released
@wevii9043
@wevii9043 4 ай бұрын
Make a video about stable diffusion like this
@WINTERMUTE_AI
@WINTERMUTE_AI 5 ай бұрын
I keep canceling my GPT4 subscription and then renewing it... 'Just when I thought I was out, they pull me back in.' GPT4 reminded me of that phrase from The Godfather. :)
@rumali_roti7406
@rumali_roti7406 19 сағат бұрын
You added models in the description but specify their usage. Can you add more details, please?
@NeostormXLMAX
@NeostormXLMAX 4 ай бұрын
I spent so much time trying to get something like this set up, but ended up back to gpt, most of these models are also censored just like gpt, and unlike gpt they are much slower AND on top of that they canot use plugins or special api's that let you access the internet or generate images etc. its sad but currently gpt has no peer
@つロつ
@つロつ 5 ай бұрын
Besides saving money, are there any other reasons to do it locally vs spending $20 a month for chatGPT?
@nyxilos9167
@nyxilos9167 5 ай бұрын
privacy mainly
@lodyllog
@lodyllog 4 ай бұрын
privacy and reliability, as with local LLM you don't depend on anyone's else infrastructure
@voidsofold
@voidsofold 4 ай бұрын
Privacy, it's not filtered so you can do more things with it, won't see random dips in quality based on the whims of investors.
@AshishKumar-kv4hr
@AshishKumar-kv4hr 5 ай бұрын
Are you the same as fireship?
@swaggitypigfig8413
@swaggitypigfig8413 5 ай бұрын
Different human being
@I_SEE_RED
@I_SEE_RED 5 ай бұрын
it’s fireship experimenting with 100% channel automation
@Frab1985
@Frab1985 5 ай бұрын
The best RP model atm is Kunoichi v2
@Dilfin90
@Dilfin90 5 ай бұрын
I am from Russia, can I participate in the contest?
@remi.scarlet.
@remi.scarlet. 5 ай бұрын
oobaGooba, OOGABOOBA
@elasdebastos235
@elasdebastos235 2 ай бұрын
My brain melted
5 ай бұрын
You did not name countries you are able to ship for the giveaway. Is it worldwide?
@bycloudAI
@bycloudAI 5 ай бұрын
i’ll pay for whatever shipping it costs unless the country is unshippable like north korea
5 ай бұрын
@@bycloudAI Thank you for this information, and also for the amazing content that you are putting out ♥
@twelvecatsinatrenchcoat
@twelvecatsinatrenchcoat 4 ай бұрын
Which model is best for uh... y'know... stuff...
@user-yj2tc8xu1f
@user-yj2tc8xu1f 3 ай бұрын
idk if you still need this, but one of the most "fun" models is MLewd
@twelvecatsinatrenchcoat
@twelvecatsinatrenchcoat 3 ай бұрын
@@user-yj2tc8xu1fI don't know what you're talking about but thank you. This conversation didn't happen.
@knoopx
@knoopx 5 ай бұрын
lm studio/ollama are probably the simplest ways to get started, not sure why you picked these ones
@kingki1953
@kingki1953 2 ай бұрын
I thought it was a video from Fireship 😂
@fennecthechoosenone5189
@fennecthechoosenone5189 5 ай бұрын
Koboldcpp crying in the corner
@FlafyDev
@FlafyDev 4 ай бұрын
this isn't fireship.. where am I?
@myname-mz3lo
@myname-mz3lo 4 ай бұрын
same . the thumbnail got me and then i realised this guy took fireship's entire style
@abdelkaioumbouaicha
@abdelkaioumbouaicha 5 ай бұрын
📝 Summary of Key Points: 📌 The video discusses the landscape of AI services in 2024, highlighting the abundance of hiring freezes and the prevalence of subscription-based AI services. 🧐 Various user interfaces for running AI chatbots and language models locally are explored, including UABA, Silly Tarvin, LM Studio, and Axel AO. 🚀 The importance of choosing the right model format, understanding context length, and utilizing CPU offloading for running local language models efficiently is emphasized. 💡 Additional Insights and Observations: 💬 "Garbage in, garbage out" is a crucial principle highlighted when fine-tuning AI models, emphasizing the significance of quality training data. 📊 Different model formats like GGF, AWQ, and EXL 2 are explained, showcasing how they optimize model size and performance. 📣 Concluding Remarks: The video provides a comprehensive guide on running AI chatbots and language models locally, emphasizing the importance of model selection, context length, and fine-tuning techniques. Understanding these key aspects can help individuals navigate the AI landscape effectively and optimize performance while saving costs. Generated using TalkBud
@zyxwvutsrqponmlkh
@zyxwvutsrqponmlkh 5 ай бұрын
I just really really like how many serious people have to say ooobabooga. It's like, almost as good of a joke on science as when that guy named the seventh planet.
@adamofigueroa
@adamofigueroa 5 ай бұрын
I just have a question, why this channel is so similar to fireship? are you the same person? : )
@donciutino7490
@donciutino7490 5 ай бұрын
easy oogabooga Apple M4 cpu support before release 😂
@AleNovelasLigeras
@AleNovelasLigeras 5 ай бұрын
We're gonna need a bigger GPU
@alan_yong
@alan_yong 5 ай бұрын
🎯 Key Takeaways for quick navigation: 00:28 *🤖 Running AI chatbots and LM models locally provides flexibility and avoids subscription costs.* 00:43 *📊 Choosing the right user interface (UI) for local AI model usage is crucial, depending on individual needs.* 02:05 *🖥️ UABA is a versatile UI choice for running AI models locally, supported across various operating systems and hardware.* 02:33 *💡 Installing UABA enables access to free and open-source models on Hugging Face, simplifying the model selection process.* 05:18 *🤔 Context length is crucial for AI models' effectiveness, affecting their ability to process prompts accurately.* 06:12 *⚙️ CPU offloading allows running large models even with limited VRAM, leveraging CPU and system RAM resources.* 06:52 *🚀 Hardware acceleration frameworks like VM inference engine and TensorRTLM enhance model inference speed significantly.* 07:36 *🎓 Fine-tuning models with tools like Kora enables customization for specific tasks, enhancing AI capabilities.* 08:47 *💰 Running local LM models offers cost-saving benefits and customization options, making it an attractive option in the AI landscape.* Made with HARPA AI
@Good.Idea.Zlovakia
@Good.Idea.Zlovakia 17 күн бұрын
So is LM Studio trusted?
@rotors_taker_0h
@rotors_taker_0h 5 ай бұрын
Basically to understand this video one should already know everything mentioned in this video by heart.
@Trahloc
@Trahloc 5 ай бұрын
Eh, it provides terms to hunt for and sometimes that's all someone needs, a starting point. The video is short and covers a lot of ground.
@MonkeeGeenyuss
@MonkeeGeenyuss 5 ай бұрын
Dude wants a 16 part lecture to explain it all😂
@rotors_taker_0h
@rotors_taker_0h 5 ай бұрын
@@MonkeeGeenyuss I mean, I can only follow because I know it all and cannot imagine someone unfamiliar to understand anything from this firehose, lol.
@TiagoTiagoT
@TiagoTiagoT 5 ай бұрын
06:14 How do you have Mixtral 8x7b setup to use just that much VRAM and run that fast? On Oobabooga just 9-10 layers and I'm already risking running out of VRAM on my 16GB GPU, and the thing still takes enough time to finish writing moderate length replies that often my ADHD kicks in and I go do something else while it finishes writing.... Which of the GGUF quantization are you using? Is the issue I'm using the Dolphin version instead of the raw Mixtral? Should I ditch the Dolphin variants? It's been getting so hard to keep up with which models are the current most well regarded by the community with so many models coming out all the time...
@voidsofold
@voidsofold 4 ай бұрын
For mixtral 8x7B you're going to want 2 3090s.
@TiagoTiagoT
@TiagoTiagoT 4 ай бұрын
@@voidsofold16GB is not enough?
@voidsofold
@voidsofold 4 ай бұрын
@@TiagoTiagoTNot enough for 8x7B
@TiagoTiagoT
@TiagoTiagoT 4 ай бұрын
@@voidsofoldThe example he shows after mentioning using 10GB VRAM for quantized Mixtral was not actually that being done in practice but just some unrelated LLM output clip?
@voidsofold
@voidsofold 4 ай бұрын
@@TiagoTiagoTOh, if you offload it sure you can run 8x7B, it'll just be very slow and have barely any token context
@jawbone1218
@jawbone1218 5 ай бұрын
Curious headcount? 🙋How many of us watching these type videos are not developers?
@Afro__Joe
@Afro__Joe 23 күн бұрын
Lol nobody reads anymore by these comments. Ooh shiny picture, click! Thanks for the info, I was looking for a video like this yesterday.
4 ай бұрын
So, AI is the new computer, everyone will have one? Seems good to me. I wonder how the job market will be, hardware will be on top for sure and Open AI will still being a giant. But the thing is how other industries will be affected.
@gregNFL
@gregNFL 3 ай бұрын
Stanford open source LLama model is free. 🎉
@MoisesMendes-hb1ee
@MoisesMendes-hb1ee 4 ай бұрын
Hope it works on my toaster too
@Cergorach
@Cergorach 4 ай бұрын
I was highly disappointed, Shōji Kawamori and Kazutaka Miyatake are not on the panel about Transformers... ;)
host ALL your AI locally
24:20
NetworkChuck
Рет қаралды 895 М.
[Autonomous Agent Workshop Preview][Swarms]
0:41
Kye Gomez
Рет қаралды 55
Nastya and SeanDoesMagic
00:16
Nastya
Рет қаралды 19 МЛН
DAD LEFT HIS OLD SOCKS ON THE COUCH…😱😂
00:24
JULI_PROETO
Рет қаралды 15 МЛН
ЧУТЬ НЕ УТОНУЛ #shorts
00:27
Паша Осадчий
Рет қаралды 9 МЛН
The Petabyte Pi Project
22:27
Jeff Geerling
Рет қаралды 2,2 МЛН
Prompt Engineering Jobs: Career Outlook 2024
5:41
Arnold Trinh
Рет қаралды 124
What's next for AI agentic workflows ft. Andrew Ng of AI Fund
13:40
Sequoia Capital
Рет қаралды 277 М.
PC Hardware Upgrade For Running AI Tools Locally
9:12
Ai Voice Tutor
Рет қаралды 10 М.
BERT Neural Network - EXPLAINED!
11:37
CodeEmporium
Рет қаралды 385 М.
What is RAG? (Retrieval Augmented Generation)
11:37
Don Woodlock
Рет қаралды 117 М.
NVIDIA Unveils "NIMS" Digital Humans, Robots, Earth 2.0, and AI Factories
1:13:59
ОБСЛУЖИЛИ САМЫЙ ГРЯЗНЫЙ ПК
1:00
VA-PC
Рет қаралды 2,2 МЛН
S24 Ultra and IPhone 14 Pro Max telephoto shooting comparison #shorts
0:15
Photographer Army
Рет қаралды 9 МЛН
Лазер против камеры смартфона
1:01
NEWTONLABS
Рет қаралды 654 М.