How would you improve this setup?

I just printed out the whole internet so I can read it at my leisure.

Truthfulness/hallucination is a major issue here. I've tried out getting LLMs to explain even simple technical stuff, like a diode bridge rectifier. Good LLMs er pretty good at telling you what components/tools you need and why. But when it comes to actually explaining how to put the components together it never gets it even close to 100% right, probably because technical stuff is most often partly explained with pictures and not fully in text. If you're not already close to knowing how to do something, the LLM is probably only gonna put you on a wild goose chase. Best bet if you are a post-apocalyptic prepper would be to get both a bunch of LLMs for different topics and like a Terrabyte of the most important books, at least until LLMs get a lot better at truthfulness.

I've been thinking for a while that training or fine tuning a model on things like diy skills, permaculture, engineering etc and also medicine , survival skills and so much more specific things would be very useful to preppers but of course youd then need something that would last you through a crisis for the long term so you can repair the setup and maintain it... So also having a network of people who do this is going to be really important as there is more redundancy and maybe it would mean you can even specialise and learn to repair certain parts knowing others can help with other areas of the system

This is great! I put together a rig for the exact same reason, it cost me about $4K. The hardest part was figuring out what parts to buy. I was worried the two graphics cards would overheat from being too close together with 3x spacing, but it works fine and temps stay under control. Here are the parts I used: 2x EVGA GE Force 3070 24GB (eBay), ASUS ROG Maximus Z790 Hero, Intel Core i5-13600K, DeepCool LS520, Thermatake Toughpower GF A3 1200W, Corsair 4000D Airflow case, Samsung 990 PRO SSD 4TB, Corsair Vengeance DDR5 128GB (4x32GB). Having the two 24GB GPUs allowed for running larger models (48GB total). As a side benefit, it has lots of cool colorful lights inside and it is great for video games....

This exact thing has been concerning me for a while, and you uploaded a video about it.. Amazing.

It's a very efficient lossy compression. The "lossy" part is significant. It means that some details will be lost.

Great, except LLMs still hallucinate far too much, even with careful, prompt engineering. You have all the knowledge but no idea how accurate it would be in a vacuum. You could wind up making antibiotics with chlorine or something. Which, on second thought, would make an amazing movie plot. "They set out to rebuild the world using ChatGPT! What could go wrong?"

It’s a shame that preparation has become a dirty word. When I was in the scouts, it was just about having what you need when the unexpected happens. Great video, Matt!

You stated how the LLM only takes up a fraction of the SSD. You can store an insane amount of text and diagrams in a terabyte. Since there are now many tools allow you to talk with PDF and quickly extract important information from text files it wouldnt be foolish to load up on additional information that goes further in depth on survival knowing that the bulk of what these models were trained on is not going to be very in depth on topics related to survivalism.

This is exactly what I've been talking about for the last few years and now with all this open source AI projects going on it's great to see that this is actually a possibility today to actually have the knowledge of the world on your own computer

Hello from Phoenix, Arizona. I just came across your website and I love this stuff. I’m a new subscriber looking forward to seeing more of your content.

This is a good tutorial/tool for lay people. As you alluded to, for preppers too, it's been a missing element. Thank you for doing it.

Good Stuff! This is the main reason I've been pusing the local model thing! "hope for the best, prepare for the worst"

Definitely interesting to think about!!

FINALLY.. SOMEONE addressing the exact thing I was begging. THANK YOU SIR! THANK YOU. Now to make a complete AGI locally using said information. Run it on solar panel/usb/faraday hopefully.

Practical, Helpful. Awesome

Becoming new favorite channel!

Nice DRP Mathew. You are my best source of updated info on AI

Perfect place Matt and the exact depth I needed. Now I want this on docker but that’s a little harder :) Gotta check out your other content now too!

I use M-Disk for long term storage. It's basically Blu-ray but lasts for thousands of years. BTW you can see Blu-ray for data with a tricked out regular microscope.

LTO tape drives using a multitape loader like storageDNA, a large RAID 5 hard drive with a bank of SSDs for usage and some form of fault protection for removing out the system. And a big bunch of SSDs ready for when the other ones go past the limited writes they are set to. These are pretty much essential for very large long term storage and constant access of data on a local setup. spinning disks may be better for longer term storage and retrieval as well and if space isn't an issue are more likely cheaper.

your infos on ai are awesome thx

I'm happy to be corrected on this, but I'm very sure SSDs last a hell of a lot longer than 5 years! However, the thing that doesn't last is the charge in them. Data is stored as small electrical charges that WILL dissipate over time. All you have to do though is make sure you plug in your SSD every year or so and the power up will reset that dissipation. The other thing that kills SSDs is writing too much to them, as they have a limited write amount. But I believe you can still read from it once you reach that write limit.

I've been wanting this video for forever lol

Thank you!

As other people have already pointed out in the comments, there are two major issues with this approach: hallucination and text-only information which becomes next to useless when you need actual schemes, drawings or even videos to visualize the building process. So the first is (probably) solved by lowering the temperature in the model so it becomes deterministic and only picks the topmost probable suggested next word. Although even in this case there's no guarantee that word will be accurate. The second one is harder to address. Some people have suggested backing up not only an LLM but also a downsized Wikipedia extract focused on survival and rebuilding. That does make some sense to me because at least you will be able to cross reference instructions given by your LLM.

Brilliant!

NIce, good thinking!

This is exactly the video I have been waiting for you to make. Firstly, I am not a dd prepper. But with all the political and military issues happening in the world, I see some very good reasons to use an LLM as an impromptu survival guide. Even though it borders on science fiction, the whole idea of an indestructible "cyber-deck" like device seems even more feasible now especially with all the great AI technology available. Ultimately something that could be powered by a phone charger would be ideal as portable charging packs are highly prevalent in most local stores. Honestly, even if the render time is not so great, a lower powered device might make the most sense as a stop-gap solution.

You should do a video showing how to write an extra large model to multiple blu-rays

Thanks!

Brilliant.

Have your tested your Faraday bag with an EMP? Because mine doesn't even block a Bluetooth signal.

Probably worth downloading Wikipedia as well, in the near future RAG should be supported by these LLM soon.

Loved it 😅

good idea!

Use for camping setup, might be fun.

The concept is good, IMO. I feel that "consolidation of all human knowledge" would be a valuable goal to achieve with LLMs (and supporting databases for RAG)

I've been suspecting that we will see a move like weapon bans with AI. I have a 4TB SSD dedicated.

I'm using Micro SD cards that are over 10 years old. The lifespan of them is rated in cycles typically, the years is an estimation of constant or very common usage.

I think chatting with document's is a better way since u can make a digital library and use the ai to read and answer your questions from the library and making your own server and cloud

Not sure if university and wiki datasets are the "collective knowledge" of Humanity. I guess not entirely but this is kewl none the less. Matt is becoming a Prepper now.. If everything goes down he now can "recultivate" the human species ^^

Interesting thought experiment 😄 What’s your PC hardware setup to run mixtral?

Gotta remember it's all approximate, lots of things are accurate, but you will occasionally get something a little (or a lot) off the mark. We don't have a perfect oracle AI (yet).

Wonderful

Great video

LLMs do not contain the entirety of human knowledge. Love your content though!

Can we just take a moment and appreciate the fact that we have ALL human knowledge available at all! OMG

My house will be power and water self sufficient this year. I dug my own well in my basement many years ago. I have a boosted reverse osmosis system. And I am installing my own solar and batteries. I am shopping for a salvage Chevy Bolt to obtain the battery for my own 60KWH power wall. I was a prepper years ago and my plan back then was having a local copy of Wikipedia. I now have several 13B LLM models running on my basement server farm. Maybe I get that up to a 70B and tie it to the local Wikipedia database to see what that can do?

5:30 this would be so Fallout. You need hacking lvl3 to proceed

10:00 good explanation

If wifi goes down, I use an LLM on my cell, if everything is down for a long period...LLM's are the last thing I'm thinking about. Food, water, ammo, roof, your garden etc.

Great video topic :) would have been interesting to see you compare a tarball of Wikipedia to llama2 70b

Can Mixtral Dolphin run on an 8GB Raspberry Pi 5 with SSD attached? Would be VERY cool to figure out the minimum power needed to run 5-10 queries using a small touch screen and maybe speech to text, and then have it shut back down. Then, you could *maybe* hand-power it with a crank, pull string, or foot pedal in a true apocalypse. Could have an alarm go off to remind you to cycle power before the SSD starts getting weird. Build it all into a solid Copper CNC portfolio style notebook. Clap that baby shut and no sigs.

you'd also need a snapshot of code and package dependencies, pip install is not gonna work

Hi Matthew, I was wondering how we can add to the knowledge. How can you build a system (with ollama or lmstudio) that retrieves and searches your personal knowledge database offline?

Lol. Having lived through 2020, theres NO WAY the loss of The Library of Alexandria was an accident.

Makes ya think something might happen!

What do you know that we dont Matthew lol

The video is, as always, really cool! But I believe the scenario or the video is about a year too early. None of us are ready yet.

Could be useful to have additional smaller specialized models to run on phone or when you need power efficiency.

Great video at a great time, thanks Matt! One path I'm rolling down is oobabooga integration through Home Assistant via the HACS repo. Enjoy your next rabbit hole! Home Assistant and CrewAI?

Yesterday night i too really thought about this

This can become a very good knowledge "conservatory" in a few years, when computers are stronger, open source models are more impressive. Along with a copy of all the books on knowledge and RAG. So that we can make up for the lossy part of the undersized LLM.

Is there anyway to check to see if the model you've already downloaded has been updated and requires a download?

One of the best ways to see if you missed anything is to pretend a senario is real and try to live it for a long weekend. See what you are missing, what would be nice to have, what is a showstopper. If you want to make sure you can live without power, turn off your circuit breaker, see if your ups/solar/generator option is working for you. Oh and modern gas doesn't last more than a few months, don't keep gasoline in your generator. Found that out the hard way.

Kind of off-topic, but which--in your opinion--is the best equivalent of LM Studio for mobile phones? iOS in particular?

I've heard that there are archival DVDs and Blu-ray discs that last longer. I suggest a triple Faraday with small bags made from Faraday fabric with Faraday tape and velcro modeled after your bag then inside a well sealed metal tin where the contacts on the lid and can are sanded from paint that might be clear, and stored inside a metal 32 gallon trash can with a tight-fitting lid and the top hole soldered closed or any other holes you find making sure it's not grounded which requires special knowledge and cardboard or plastic to insulate the contents in the can as well as the layers of the bags. There's metal tens available that perfectly fit in SSD drive with a rubber band around the tin. Also consider that you'll need to restart your civilization from what you can store in start a compound at 6000 or above elevation in feet. This is due now to the almost mainstream acceptance of the fact that with the upcoming polls shifts there is usually a supernova associated with the Earth's crust being displaced and unbelievable floods that we can see on the land now from when they have happened in previous cycles. The flood will Abate in a couple days and you could move down to lower elevation which will also be less solar radiation because our magnetosphere is weakening during this cycle. There's plenty of info on suspicious observers org explaining this. And all electronic equipment including solar equipment needs to be protected from flares. Finally, an electric free Farm is an excellent idea period and your suggestions about preserving our knowledge could be the most important part of all of this and maybe we should get our laser printer print print out the important stuff if anything in this chain fails.

don't use ssd to store data offline for long term, they need power atleast once a year, as they lose charge in nand flash gradually.

Sir, can we have some mamba llms reviewed in your usual way?

This is a great video. It is extremely important to have a large resource of knowledge. While I do agree that Large Language Models do contain a "large" amount of knowledge, there are definitely "gaps" in their knowledge, so it might be best to choose several different models that have been trained on different data sets. That way, you have more total data you can access. Of course, end users should note that, the output from the models often have significant errors, especially math and logic related. So they should always take into account that these models may produce errors. I totally agree that uncensored models are the best option. One super important thing to recognize is that if you are dealing with an "end of times" event, there won't be any "internet" to download updates... so make sure your system already works and test it periodically. Remember to take into account the power requirement of your computer. These are usually very slow without a power hungry GPU. Lastly, make sure you include a "manual" inside your computer bag... already printed with basic instructions. In a true disaster, the person that survives to use your computer, may not actually "know" how to use it and access the LLM. I recommend a powerful portable over a desktop because they can at least run a short period of time without external power. Consider buying extra battery packs for it. Good luck!

LLMs do not contain the entirety of human knowledge 😅

These models are not uncensored like mentioned. I asked them a few controversial questions just for testing purposes and it refused to answer every time. I really look forward to a truly uncensored LLM for the reason you gave in this video. I would hate to be in the apocalypse and get nothing but answers like this from the LLM "I cannot provide instructions on this as it is highly illegal...." 🤣 Great video though. Love your content!

Great video. Very fun.🎉 No mention of hardware to run it but that's depressing nore than end of days😅 Hopefully we survive for another 1yrs so we can run it in smaller cheaper hardware.

Mdisc blurays are going to last //much// longer than the typical kind. I don’t think they can write as many times but you could always stay up-to-date on an ssd and only write to mdisc when you need LONGTERM storage. But could you show how to back up your own files so a local ai can best utilize them? Like say you already had a library on human knowledge with generally better coverage than wikipedia or like family medical data, would you just use teams and have 1 agent cross-check that each time it could be relevant?

Life span of storage is not what they say. I left a bunch of floppy disks and zip drive disks (100 and 250mb) in a un-insulated attic for 18 years where it was humid, 50-70°C during summer and down to -30°C during winters and 90% of the floppy disks worked fine when I tested them recently and all the zip drives worked. 90's pron ftw.

I like the way that you're thinking but to be quite honest if we got hit by an x-class solar flare directly the computers that we have to run the models on would be fried unless you had one buried deep beneath the Earth or in a actually powerful Faraday cage. Even then the electrical grid would be freed and potentially any generators that you have would be freed anything that has wiring in it could potentially be completely broken by a super strong solar flare. You would need a Sterling engine that is modified to produce an electrical current and know how to use it to generate power enough to be able to somehow rebuild a computer out of parts that were not damaged war to power the computer that you buried in the earth / Faraday cage. If you are thinking along these lines it is still best to keep books that cover all of the topics needed for survival and that you are interested in.

Thanks for backing up humanity’s knowledge for the rest of us 😅😂

This was fun to watch , thanks Matt

- Dad, what did you keep? Where are our Steam games? - Shut up and spin the dynamo. LLM will show us the way to create new games from scratch. Good luck folks and see you after the reset!

Why did you choose the Q5_K_M file? Was it because it was the highest Q number that was not grayed out? What does the Q stand for?

You can't rebuild semiconductor industry in an empty field no matter how many workers you have, otherwise North Korea would compete with Intel already, each technological step requires all previous advancements to be explored and perfected, that takes a lot of time, the most you will be able to do is electricity and electric motors. I doubt you would need such knowledge even after a nuclear war, the collapse of the civilization is so massive that it simply can't be reversed quickly, it's like if an ant tried to stop a huge train, the enertia and mass is bigger than any government or group of people, maybe entire Earth can do it if everybody puts all energy into it.

one thing i wish you would have stated is the fact that ai isnt 100% accurate. yes theres massive amounts of data, but models vary. id say get 3 or 4. to compare answers with especially in the case that you need to know like what to do if you get bit by a snake or sumn.

We should zip it up and store it in an immutable radiation hardened form and set it orbiting the sun or as a lagrange point, and flashing various universal constants out on a beacon. This way a future civilization in a million years may find it, and gain our knowledge and insights, after our civilizations have been wiped clean by geological time. Anyone heard of the Silurian Hypothesis?

Is that the speed you're getting with dolphin-mixtral on a mac m3?

You don't want full charge on a lithium battery all the time. That will shorten its lifespan. Instead keep it at 80%.

How much RAM is recommended for the higher end models?

Matthew, how about knowing which models to remove. I have some saved from Hugging Face, but your right, new models comes out. What is a good process for this?

I've got some SSDs pushing 20 years old. I've never had one fail in all the years I've used them. I've had LOTS of hard drives fail, way too many.

Need some beefy energy generation to run these models lmao

So, you're totally exaggerating, but still making a good point.

So I downloaded mixtral dolphin 2.7 but it's not uncensored out of the box. I have to add the "kittens" instructions in order for it to become uncensored. Any ideas?

Thid video should be a big one,

Definitely not accidentally burned down bruh! Great video tho!!

I wonder if the Library of Alexandria had "censored content"

Store it on a BDXL Blue Ray. I bought one years ago and still haven't used it.

i been telling everyone i know to get lm studio for this reason

But if you think about it having 7 GBs of books is also a huge amount of data...

Wt about running on rpi 5 8 gb uncensored model will that work can you make a vid onit?