I have been doing something similar to this for the past two years. $48k is just overkill. Making the most expensive computer today isn't what's necessary to run an AI powered local stack. I have been able to use 64GB of vram (Accelerators at 2x) 12GB vram (GPU 1x), 256GB Syst ram, and 12 cores at 3ghz. I think I got everything for about 2500 and I have been able to really study the technology very well with this workstation configuration. One could get 4rack gpu server for 20k with 8 accelerators at 32 gigs per card. 512 Gigs of sys ram and 4 20TB Hard Drives (Which is what i would like to do for multi-modality research). By the time the H100 chipset generation of technology becomes consumer class hardware I'm almost certain that ASIC hardware will have really started to corner this market. It's just to valuable of a technology to not be optimized with ASIC architectures. I'm sure that in 5 years one will be able to get the second generation of transformer ASICs for perhaps 400 a board in today's evaluation of the dollar then, and have 10000 times more flops per second than the equivalently aged and priced general purpose GPU accelerator. Not to mention that these network architectures are not the end all network designs. As the industry moves toward the obvious advantage of ASIC chips there will be times where the network architecture technology improves and will make the previous generation of ASIC designed hardware exponentially depreciate in value even though the application of the network models and chip hardware are still highly usable and productive for a broad range of tasks. It's going to be a wild decade, let me tell you. LoL. What an amazing time to be alive.

I love how the more we get into the future, the more our softwares look like MSDos

We are living in a mindblowing reality, and you are brilliant at predicting the "AI Future! "

Or how about this: a humanoid robot has such a built-in headless PC for performing tasks such as sending email or doing spreadsheets or editing video. All of this stuff is done on a traditional PC, built into the robot, but the LLM in your humanoid robot communicates silently with that PC on your behalf, even doing things like clicking confirmations buttons when necessary. So you can have a conversation with your humanoid robot and say "yeah that sounds great make that video" and it will use Final Cut Pro X to make the video, sending you a link in the cloud when it is done. This way we leverage decades of engineering without needing to teach an LLM to be a video editing tool.

fore sure smth like this but running completely on local machine. And I get a glimpse that it is possible, because of the openAI api.

Love your videos, would be really interested to see your review of DSPy.

I don't think creating a dedicated super-powerful machine with a new OS is necessary. Apple for example could simply make this part of their OS and run a small, efficient LLM that moves the mouse and enters keystrokes. It could be trained on thousands of hours of videos of using Apple's entire suite of native bundled applications. I already run crazy large LLMs on my Mac Studio no problem. They run extremely fast too. Why recreate an entire platform with things like email apps and PDF readers when it's simple to retrofit Windows or macOS or Linux? Sure, more RAM and faster GPU performance will always be welcome but that is on the way in any event. I suppose at a certain point once it is reliable enough you could run that computer "headless" and simply get the results of what you tell it to do sent to your smartphone.

I'm still hoping these tech companies and A.I developers will discover the difference between intelligence and consciousness... .i won't hold my breath in nervous anticipation.

What software did you use the create that "AI PC Flowchart" ❓ Good work. 👍

I think Windows 12 will be the first ai based operating system

It depends on how much progress is made with SLMs. Currently, yes, that's what is needed in a computer. If the OS is using something along the lines of the MoE architecture, it could have a bunch of different SLMs that complete the tasks. I'm 80% confident, that is what Microsoft is trying to pull off for Windows 12. On the flip side, if Gemini Nano can get to a point where it is as powerful as GPT4 Turbo, you could run all of this on a cell phone.

this can be a thing but as far as its "expensive" for typical user... well :D Good work as always! thanks for the vid!

GREAT! BUT how about showing it to the people while using local LLM like llama or mixtral?

Great video ❤

it could lead to a rise in mainframe computing for general computing again. A business could have one of these $50k computers in the office and the workers just have a thin-client terminal with their apps to automate.

NVIDIA H100 is for training and inferencing, so it is waaaay too much if you only want inferencing. Unfortunately large inferencing chips have yet to arrive, but they should be at least 10x cheaper compared to training chips. The kind of inferencing chip I am working on would be even cheaper, and very fast too, consuming only 10x to 100x less power, but at the cost that most of the weight of the network are baked into the chip. That would be another way, when we have good large foundation models: bake most of the weights into the chip, but let the digital parts to use LoRAs to extend / fix the functionality. So the the networks still remains somewhat trainable. The other way this is extendable is to use flash to store and compute of the weights (or memristors). This is a trade-off obviously.

I think something like Zapier but with AI will be more common. Kinda need a tool that translates human input and containerise individual issues and allow ai to interact with those things. The connection between ai and other tools is still being worked out. No code and AI I think will advance a lot

Unified memory is a must since small Whisper models are twice as fast on M3 Max compared to RTX4090...

I have a beefy CPU and GPU, could this be theoretically possible to run locally? I see there in config URL to API, which theoretically means I can substitute Open AI API endpoint with some local compatible API, for example from LMstudio, right?

Your idea is what we see in movies like Alien or Space Odyssey 2001 but companies will never support local/independent AI usage model for general public. They are interested in selling AI services and charging monthly fees. Not to mention mobile clients won't be able to handle ChatGPT level of AI locally any time soon. However military or government can use whatever they want.

👍Most things I think of, come to reality before I could do it myself ❗

What about using llava for images?

I think it will be a usb c flash drive for the lfs and neural engine can be a plug in GPU dock looking thing with instead of GPU it has just ai features so we get do this stuff on any computer and it can write it's own code by what it hears

@AllAboutAI Bro let me know if you get it working on a low end PC. I mean buying a low end Server with the proper computer power shouldn't be too expensive these days. And buying a few good and well priced RAM bundles shouldn't me all that hard. No I think the GPU' not GPU's and storage will be the most expensive. Because eliminating the bottlenecks of these 2 prices will be costly. But after al that I get this will cost around 3k. And having an A.I. Powered PC that you can verbally dictate too is completely worth it. Now all of this is just off the top of my head so I dont know maybe do a video on it and see. Cause running the tip top of the best of the best parts isn't the DIY way. I hope you enjoy this thought experiment.

Yeah, but there’s one core concept that would be missing here: Reproducibility. With AI, there’s always a certain temperature of the model - the higher it is, the more variety/creativity in the answers. Now, I’d assume we would trade in the time savings created by such an AI computer for the relatively high amount of entropy when it comes to the output. So, unless the time-savings benefit outright the reproducibility drawback, I am struggling to understand how this would become commonplace (though I definitely see this kind of device in consumer settings, as compared to - for instance - a scientific laboratory, where reproducibility matters a lot more).

Can you run this with ollama and local models?

My major prediction is that the new AI chips being developed currently are just the tip of the iceberg. In about 10 years there will be entirely new computing architecture, not anything like that we use today, with portable devices using super efficient batteries that last months on one charge.

There's gotta be an extension or something for that instead of a need for a special computer.

Your mic is so tinny bro. but good video.

Can you please review Sora next please ?

It would be interesting to see this with a local llm (llama or lm) and MoonDream

AMD "strix point halo" npu and Intel Core Ultra npu. You can find references to these running llama. I believe these have access to all of the pc's ram and are faster than cpus. "Strix point" is meant to have 47 tops. Being a pessimist that will be int 8 tops. If it does bfloat it will be half that at ~23 tops. In anycase they could evolve to full llm ai engines. My guess somewhere between the performance if a gpu and that of a cpu but with the benefit of being able to use full system ram. 🤞

If you have an AMD 7040 series CPU (apparently there are already 2 million in the market) you can already run a 7B Q4 LLM locally hardware accelerated... in theory. With modern medium to high end hardware and enough RAM 4GB and 8GB LLMs will run on CPU. Expensive GPUs aren't necessary for single stream workload. The uber-spec hardware in the cloud supports multiple simultaneous prompts. An AI assistant on a personal computer stays loaded all the time and only processes one stream at a time. We will be able to run powerful personal LLMs very soon if not now. Just try GTP for All and see how it runs on your computer. UFO is trying to automate applications that were built for user interaction via keyboard and mouse. The notion that an LLM can infer from screenshots where to place the cursor and paste text, while cool and somewhat amazing, is just ridiculously inefficient. I'm thinking applications will need to be rewritten to enable AI automation, giving a locally hosted AI an API through which it can interact while allowing customary user interaction at the same time. Imagine: Windows 12 AI Enabled with Falcon (upgrade dependent on hardware compatibility)

Why would an AI need to use a human to machine interface?

Wait till tech giants integrate this into their hardware. With these server prices I imagine smth like Siri+ premium subscriptions for extended automation

Get a macbook m1-m3 with 64-96gb of ram and you can run all the biggest LLMs on a laptop.

I don't think running models locally is the future. Maybe smaller highly specialized versions. But if you want to use latest cutting edge Ai, you'll always need a cloud. But the prices will go down, that's for sure, and the response time will decrease.

I think we won't require that much power I think most computers and phones will be hooked up to the cloud to access that level of power because there's no way this many people can afford that so I think a lot of these powerful computers will be running in connection with the cloud

Excuse me , I have a question it seems like the method of giving the gpt a persona does not working anymore , done this with bing A.i , it did earlier but it is no longer useful

Just I want to speak. With my voice all the command will happen. Thats what I want what do you think ?

Guessing MS will embed a helper model into Windows 12 etc to bring the cost down. They will want to make their next OS version to have AI at it's core, even if just for marketing reasons.

97 cents? I'd say in a month or two an open version will cost 20 cents. Still too expensive, but it'll never be zero. :-(

Mutli-ON and Open-Interpreter would do a much better job without any big costs.

AI PC? Absolutely, but don't be an early adopter. There will be a race by all the manufacturers and in the end, it will be just like any other PC market. The best will rise to the top and the components will be standardized but a lot of messy evolution will need to churn and burn first.

5-10 years max and you will not need a ai pc it will be your phone the successful models will be otimized to run localy on dedicated "NPU" or what ever is coming next ai chips designed by ai for ai ai will run on phones(BLE mesh network ) and on the "A.I.of things " i think the consept of a ai pc is out dated already future of AI is small and local

yea more or less useless toward openAI considering their prices, do a tutorial using it with llama2? or is the function calling thru llama2 worthless?

i think we need more 25 years or more to have this locally in reasonable price 😅

This machine definitely lacks a floppy disk drive!

if 1 simple task is going to be $1, how the heck rabbit is doing unlimited tasks for free???

Too Expensive

look like Open-Interpreter,expensive

maybe that’s why Sam wants to have 7 trillion to build that

First comment

Sorry, this sounds ingorant. First, it will take a LONG time until you can run high end AI at home - a LONG time because while you may think that today's model at GPT are high end, by the time you can run those at home, you want more. Second, latency by cloud - yes, but not a problem. You need a proper cloud (i.e. like Azure locally, not like OpenAi does centrally at one location) and good streaming protocols. Here is an idea - people do use that TODAY to stream GAMES. Videos are streamed and no one complains about the latency due to buffering. latency is actually ONLY a problem when you need interactivity - not i.e. talking, but fast first person shooters. Now, for your nice powerful PC - it is NOT perfect, it is NOT what you think it is. They say it has enough memory to run the largest models - that is TRUE - but the performance is abysmal. it is a H100 with 92gb of high end RAM and then a lot more of A LOT SLOWER ram. In fact, it is not even hardware that is made to RUN models - for that you use H100's. The H200 is made to TRAIN models, as you have a lot of software there that needs fast access to the calculated weights and a lot more memory to optimize, hence the H200 where the "slow" memory still has full bandwidth to the fast mamoty. Funny enough that is something the creator is not showing - but it is in all the data sheets. If anything, I can see lower end AI running in villas / apartment complexes, offloading to the cloud when things get too complicated. But not at home - wasted money.

ok i'll come back in 2030👋

Day 20 of asking for a invite

I don't understand why you focus on "A dystopian view of the outcome". Are you creating content about using AI, or about how damnable of a thing AI is and how we should be ashamed to use it?

Never ever MICROSOFT. Corona Bill.🖕

Why use OpenAI api after all this expensive investment in hardware? Use OSS, this video sucks.