Links gamefromscratch.com/nvidia-chatrtx-easy-local-custom-llm-ai-chat/ ----------------------------------------------------------------------------------------------------------- *Support* : www.patreon.com/gamefromscratch *GameDev News* : gamefromscratch.com *GameDev Tutorials* : devga.me *Discord* : discord.com/invite/R7tUVbD *Twitter* : twitter.com/gamefromscratch -----------------------------------------------------------------------------------------------------------

I have used LM Studio, it will run on lower end hardware, but expect very bad performance. You can try using models that have been quantized, which will perform better, but will be less precise (they can degenerate into random text). And I do not remember it having an easy way to reference files. About RAG, be aware that you want a model that has been trained on the general subject. That is: if a model is specialized in poetry, it probably won't do well with code even if you give it all the text books. Why? Because it is trying to rhyme, that is the pattern it learned. On the other hand, with convinience of ChatRTX, you should be able to give it your project files - those that you would not dare upload to an online AI - and have it give you results based on that, which would be specific to what you are doing. And let that be another reason to put comments and choose good variable names: the better the context you can give the AI the better. Finally, do not forget: Garbage in, garbage out.

RAG does not go to the web. With retrieval augmented generation, the local data you provide is embedded (converted to numerical form) and store in a vector database of some sort..then when you make a request to the chat bot, your query is also embedded using the same method as the data you previously embedded. A semantic search is performed (generally with cosine distance from the vectors) and the relevant data is sent to the LLM in its context window so it can base its response on the content in your data. This is done specifically to prevent hallucinations by the LLM since it has never seen the data in your documents.

As a man who has countless useful epubs and pdfs, this looks very useful. I especially like that it will give you a list of its sources; not exactly a full citation yet but very usable. However, I'm not terribly keen on it being Windows only and it's asking for a hefty graphics card and a lot of disk space for something I can do by hand. I think this is good news and it shows that Nvidia is, if haphazardly, listening to the real concerns with LLMs. Otherwise it's becoming an extremely tired subject.

Yoh, Will definitely spend some time playing with this. Was using chat to help me write a story & lore bible. But Chat can only remember so much before you have to start a new conversation. Not to mention Chats constant need to equivocate over nuanced or political ideas. I spent so much time getting it to see holes in its logic. No doubt this system will still have issues. But at least i wont have to keep starting over. Thanks as always

Anything LLM is probabbly one of the best rags chat with documents software. its open source, the developers are dedicated and it rock a TON of configurations

It's not training doccumintation/dataset it's document retrival. It literally just takes pieces of your documents and inserts them into the prompt.

Interesting usage for AI. Seems like it could be handy. For me, I really want an easier time localizing my game. I still need to figure out the optimal way to do that.

Excellent! Thanks.

LM Studio is great ... I use it quite often ... there is also Ollama but it as far as I know it doesn't have a UI, but it's easy to use.

You mentioned copilot. Does this work as well as copilot if you give it your code folder to train on?

Easy to install and use, easy to train on my own data? Man this thing is gonna be killer for brainstorming and worldbuilding!

The quote from HAL made me have to check this video. Local LLM is a cool idea. Regarding NVidia, they basically bailed on Linux which really sucks, but hopefully this will not stop this from being made available soon on Linux.

Awesome video! thank you for sharing

Nice video. This is good content

Llama 3 with Pinokio works great for this as well.

My experience is that these small models fall apart really quickly, especially when it comes to generalized questions. For programming, they seem do to a bit better, but the difference is still quite noticeable if you ask small and large models the exact same question. The first "oh, hey, this actually feels pretty close to at least ChatGPT3.5" for me was llama 3 70b, clocking in at 42GB in size. I can only fit about half of that on my GPU, and with the rest running on CPU it's pretty slow. Like 2 tokens per second.

Can confirm this does *not* install on a 2000 series RTX card. Tried on my 2080Ti and the installer goes nope.

Thanks for showing this! Seems maybe 1 other downside is you shouldn't have too many editors open or in use while using it. I wonder what would break first when local compute is maxed out.

Ollama is also an interesting option. It supports Linux, Windows, and Mac. AMD support is in preview on Linux and Windows. It sets up a server that can be accessed via an API or a simple cli chat interface.

I use linux version of Ollama through WSL with OpenWebui as frontend, it already has RAG functionality and everything is installed by basically 3 commands. Llama3 8b works great and I can hook it to VS Code through Continue extension and have personal local Copilot.

Does it work with pycharm code like python and Lua code for game and game engine development

That RAG or 'sanity checker' means that it's possible your data is being distributed to 3rd parties for analysis.

How does it handle cross-referencing? What happens if you ask the math question, then ask how to calculate the same thing in Godot? It would need to know and understand the first question and how it applies to the second question instead of just looking up a direct answer in the documentation you give it.

I have this and while it runs nice with my 4090, I still use online tools for PDFS and general research. Understand if you want to keep files private. But with a chrome extension i can use all the main AI platforms across mutiple devices. Also found gemini pro 1.5 better for large 700 page pdfs.

This is very interesting 🤔

Does it read image based pdfs? Or do you have to convert the pdfs into readable format?

Need this with agents

I use GBT4All for local LLM

How can I fetch Unreal Engine Docs to PDF ? 🤔

Dang it... I have so much text in markdown format that is useless for this training data 😭

This actually sounds pretty cool.

Turn off the internet. What does it respond ?

Neat idea, but unnecessarily high system requirements, make it prohibitive for most people. I can run Ollama with llama3 with lower system requirements and make my own GUI.

One thing to add is ChatRTX requires Windows 11. 70% of the market is Windows 10 so it's only for a limited number of users

So Kobold but Nvidia?

No api, no custom model loading, just a simple ui, no updater... (I already downloaded it three times to update it, each time ~30gb, yes 30GB FOR MISTRAL!).

ChatRTX's installer has lots of bugs, never fixed. My PC has Win11 24H2 192GB DDR5+4090 installed.

This might be the first use-case of LLMs I'm interested in. Local is necessary to address huge env cost of GenAI, and the ability to parse your own documentation is interesting.

I have a feeling that this thing need NVidia hardware that has RTX. My 1060 GTX won't run that.

Does it support the Arabic language?

I've got 2060 with 6 GB :(

Too slow, given it is 7b running on a GPU.

AI : hallo user, what are you doing? please upgrade your nvidia graphics card.. or you can't continue using our AI service

Why do they hate linux so much

Well they've been Stupidly Simple fo a couple years now with the WebUIs like Oobabooga. So this isn't exactly anything impressive.

They're late to the party: not llama3, only windos, only a basic chat interface. Open source RAG tools are already here.

Any HELPFUL comments from those that are already experts on this topic about the better LLMs to use with this from the stand point of game dev? Since Mike admitted this is not his area of expertise.

Nice! but you're too fast man!

That is a bad app. For example, it cannot take into account the previous chat when answering a follow up question.

train unreal engine 5

OMG This is way more cooler than I thought.

Win 11 only ? Lol no..

more ai stuff pls this is awesome

i like some things on AI but this is getting pretty boring now