no, yes, no, yes, no, yes, no, yes, no, yes, no, yes, no, yes, no, yes, no, yes, no, yes, no, yes, no, yes, maybe?

Finally I can start writing my 1000 word essay at 11:59PM and submit it on time.

Yeah that's just the hype meter

I believe he was being sarcastic ...

True. Especially since Twitter doesn't even exist anymore!

True. Especially since Twitter doesn't even exist anymore!

Maybe I missed a part in the video, though I think you've misunderstood the point of that segment, which I'm assuming is at 5:55. He said they got ratio-ed, which may signify skepticism amongst those who know about the tech; nowhere did he say that this was a good predictor of the company's success. I feel like this criticism is out of place.

so kids, if you fold some paper in half and then stab it with a pencil you get a wormhole!

There is no performance test till now like grok and rest. But once photonic chips are perfected for real world applications, electronic chips would feel like century old technology.

I'd say nothing's iffy there. It's a blatant scam. Simply runnung a RNG and hoping that a trained AI model would come out of it is like trying to implement the infinite monkey theorem without the "infinite" part.

@@mu11668B It's a lot more than just RNG. Take a look at RandNLA speedups for linear algebra. Implementing a hardware accelerator for randomized linear algebra is exactly what they are doing. The hardware problem is needing billions of cores due to the massive amount of matrix multiplications that neural networks do. Instead of a general cpu/gpu/etc it's a chip designed specifically for "noisy" inputs which is what an AI model is using as inputs.

@@caseymurray7722 The thing is, exotropic chip developers claim that speeding up the RNG part makes an impact. That would be true if the computational bottleneck came from the RNG algorithms. It doesn't take a chip dedicated to generating random numbers to make RandNLA more efficient. Moreover, RandNLA achieves its efficiency gain by reducing total operations required to reach the target accuracy. You still need accurate floating point calculations for it to work. That does not seem to align with what exotropic chips are claimed to provide.

sadly.

I think they're a bit far out for now.

Mach Zender.

The developments on them are cool but we're still very far off actually using them for any sort of AI training or inference

I only seen Taichi chip that did something useful

A bubble ? where do you see a bubble ?

@@Alexisz998 everyone hype-jumping in on AI when we have very few actually reliable use cases for it yet? Images are cool, but still full of flaws and other issues Text is cool, but hallucinations and stuff Music is cool but far from perfect

@@MaakaSakuranbo I share your point of view, but remember that ai only took of 1 and a half year ago... and everything is getting better and better every months. Ai won't limit itself to video, song and text generation, i think it'll go far beyond with humanoid robotics. And big tech companies like microsoft, google, amazon and Meta are already making huge profit using Nvidia GPU by lowering their operating cost

@@Alexisz998 Humanoid robots require fixing the inherent issues in the current generation first

@@MaakaSakuranbo What do you mean by the current generation ?

Empirical observation gang!

Model size in GB != Size of data needed to perform operations on that model. As an extreme example, there was a paper where they managed to run a 70B model (if memory serves) on a 4GB GPU by only throwing the matrices actively being operated on into VRAM. The whole model was something like 140GB of information, but realistically, if you divide that across ~70 or so layers it was more like 2GB per layer, plus some change. Adding onto that, the attention matrices and feed forward matrices could be further split up as needed. Granted, in that case it was pretty slow (1 hour for a forward pass, lmao), but if you were planning a compute chip around that there’s a few ways around it like significantly faster RAID arrays to load data quickly, for example, would not be an unreasonable solution for people like Cerebra’s. With that said, there are other approaches, too. Things like Mixture of Experts are much more effective if you’re evaluating inference accuracy per memory access, for instance, or efficiency of parallel compute per intercommunication. If you look at something like Snowflake Arctic it’s pretty efficient in the sense that you only load 2 out of 256 (if memory serves) experts, meaning that while the whole model could be 900+ GB, you could get away with running it on a 32GB GPU, given a sufficiently fast method to load the experts onto the GPU per forward pass. In reality, there’s a lot of solutions for dealing with large models and in particular hardware companies have access to a lot of talent that can sort out these optimizations for them, so even if it sounds crazy usually there’s a way to do what they’re saying they can do.

@@novantha1 Thanks for the response! I am aware of these variants where you only partially load certain parts of the model, there's also the option that Apple is pushing forward of storing most of the model in flash and loading only what's necessary higher up in the hierarchy. But I don't think many of these options (if any) actually work in practice right? Not that they aren't technically possible, but barely practical, the latency caused by the continuous up/down streams of data between memory layers is too much to handle, am I right?

@@user-tg2or8wy2n It depends heavily on the (no pun intended) context. In the case of the paper I mentioned where they ran a 70B model on a 4GB GPU, I'm pretty sure they had extremely slow storage relatively. A typical GPU for ML has between 500 to 2000GB/s of memory bandwidth, but the storage they used had 0.3GB/s (hard drive or SATA SSD, IIRC). On the other hand, if you were planning a piece of enterprise hardware (Cerebras) my assumption is that for a chip that costs over $1 million, that shelling out for a RAID array of 32 PCIe gen 5 SSDs (320GB/s, 64Tb of capacity easily) is by no means insane, and I wouldn't be surprised if they had even faster information delivery systems. In the case where you're doing something in enterprise and plan it out, there's no reason that streaming "cold" parameters (ie: the unchanging values that are multiplied against) couldn't be do-able on modest GPU hardware, or waferscale engine or what have you. My assumption is that the waferscale engine is fast enough to handle the active parameters in memory, and has a well thought out delivery system for streaming new data in as it's needed, though I could be wrong, but that seems like a large oversight given the scale of their operation.

@@novantha1 1 hour for a forward pass... that's a very scalable solution 😄

@@OverbiteGames oh

no operating systems , just hardware and an external scheduler service

Watch the choices that key figures made and do the same, preferably young researchers.

like anything else, learn it from first principles. learn how neural nets, transformers work, learn how to code, how to use OpenAI API etc

@@JazevoAudiosurf Learning how to code on a low level may be a bad idea, on a high level software engineering and algorithms are necessary, but I doubt coding in python or C will be for long

@@Raphy_Afk Personally I chose typescript, any language works for API. I did some neural networks but figured that understanding how it works is more valuable that implementing it (ML is hard). Personally just use OpenAI API now for my projects

I took a shortcut, I slept my way into the industry.

Because it's slow af. Works for LLM inference where the problem is in fitting the models in memory at all, but it would be insanely slow in most DGPU applications

​@@randomobserver9488 no, I was talking about a computer that uses unified memory. on PCs, the problem is the bandwidth of the PCIe bus, which is never the same bandwidth as the GDDR. But GDDR itself isn't that slower than normal DDR5. With smart caching done by the "motherboard", its totally possible to pull that thing. nVidia really knows what they're doing with this myth that GDDR is expensive, it is not, they're the ones selling it as expensive, way above what it really costs because you can't upgrade the memory. That's why GPU memory paging to system RAM would be slow in a PC, there's not enough bandwidth in the memory controller inside GPU and even less on the PCIe bus, and I think that's on purpose. There's another detail that GPU memory is accessed at 1024bits parallel, while the Intel CPU only access 512bits in parallel. That's easily solvable using more memory channels. With some cache, and some driver smartness, it would totally be possible for an ARM CPU to share its RAM with the GPU. Unified addressing, as long as the GPU doesn't touch the memory directly is doable. Not only its doable, it was done before, the Playstation5 runs like that, it has an I/O controller that sits between the memory and both the GPU/CPU to arbiter the bus.

alas, on mobile, all GPU/CPU share memory that way, its not a matter of the speed of the main memory being slow, its a matter of architecture of the hardware, that's what I'm complaining.

I want "VESA LOCAL BUS" back !

@@monad_tcp CUDA has supported unified memory (single address space, on-demand paging) from about 2017 and is limited by PCIe4 x16 to 32GB/s which is ~half of DDR5 bandwidth on consumer platforms. The GDDR bandwidth even on a mid to high end gaming GPU is more than 10x the DDR bandwidth. Might as well use CPU or iGPU when the DDR bandwidth would limit the dGPU to same speed. The number of use cases that benefit from large amount of very slow memory and need dGPU-levels of compute is tiny.