Those curious where the 1.58 comes from - It's log(3)/log(2) = 1.5849625. Basically if you have a long sequence of random three state values you can represent it with no less than 1.58 bits per three state value.

Bro. This is actually game changing! Why is this not a popular technique yet?

Summary: The video introduces a groundbreaking advancement in the field of Large Language Models (LLMs) by presenting the concept of 1.58-bit LLMs, a significant departure from traditional 32-bit or 16-bit floating-point representations used in these models. This new approach utilizes ternary values (-1, 0, +1) for model parameters, drastically simplifying the computational operations needed to run these models. Traditionally, LLMs rely on complex matrix operations, which are computationally intensive and require high-performance hardware like GPUs, optimized for such tasks through technologies like NVIDIA's CUDA. However, the 1.58-bit LLMs leverage simple arithmetic operations, reducing the need for specialized hardware and potentially lowering energy consumption and operational costs. This method significantly cuts down computational complexity, allowing advanced models to run on less powerful hardware, even devices without GPUs. It suggests a shift towards more sustainable AI technology use, with reduced environmental impact due to lower energy requirements. Moreover, it opens up avenues for hardware innovation, with the potential development of new devices optimized for ternary computation rather than complex floating-point matrix calculations. This advancement is not just a technical feat; it represents a shift towards making high-performance AI more accessible and cost-effective, paving the way for future innovations in AI hardware and software. The 1.58-bit LLMs maintain performance levels comparable to traditional models while offering improvements in speed, memory usage, throughput, and energy efficiency. This development could redefine how LLMs are scaled and trained, offering a new paradigm for AI model deployment that is both high-performing and environmentally conscious.

Why not address the elephant in the room. The 1.5 bit model cannot store the same amount of information as 16bit. So why is it performing at an equal level. Very fishy. metrics gamed? most probably.

Amazzing!! You are always very much up to date with research work in this AI field.

This is huge! Thanks for bringing it to our attention ❤

This is truly a big change, ty for the info

So basically the main idea is to get rid of multiplication as multiplication by 1 is the same, by -1 the same but with - sign. And multiplication by zero is zero. So 1 is obvious. 0 is interesting as it allows for additional complexity representation to be encoded in the neural network. I expect they could rediscover some bitwise hack techniques from the era of the beginning of 3D gaming. To use bitwise operations instead of multiplication. In that way you get the same efficiency not doing multiplication but the complexity goes up significantly. I like though the quantization as it introduces some noise in the neural network and by doing so you get probably some interesting results as the NN can outperform the original full precision model on data on which the original was not trained for.

This is phenomenal

Great video. Very cool stuff

Its so hard to believe that 3 possible parameter values can capture the patterns and signal from the data as good as with 16-bit float. Mind blowing stuff for sure.

waw!!! thanks for this amazing information! :)

Bitwise Mamba + bitwise inference for the win.

Thank you for sharing! This process will revolutionize local open source AI!

This isn't even a click bait title. This is an improvement of the original bitnet paper. This is actually a huge deal. Very. Original method, would reduce performance by almost half. We been working with it for our own LM architecture. Trying to figure out clever ways to mitigate these issurs to a degree, which required whole nee activation function for sparse representations, increasing nonlinearity...blah blah lol. This improvement seems it retains majority of this performance. Wow. Just going off vid screenshot. I have to dig into this paper now lol. Listen we won't have gpt4+ models locally without super efficient low depth quantization. This is morr holistic since you have to pretrain. Wow. Also generationing weights directly is more feasible at this depth, we believe that conditional weight generation is going to forever change deep learning. I know i sound crazy, but even compute won't be an advantage soon. We are going to change that. I know i sound crazy..but remember i said this. Within a year you wont have to pay hundreds of thousands/millions to pretrain a model, simply bring your dataset, model code, and hyperparameters. That's your prompt by the way. This year is going to be insane for all.

We have to wait until a Company will implement this technique. Very promising. Thanks for the update on new llm technique.

Very nice, thank you very much.

thanks for sharing!

Hello, how about acc to predict tokens? , even in traditional 16bit fp time there is little acc down the slope on selecting tokens

From 16bits float to 1bit ternary, bring back memory, from CD (16bit int) to SACD (1bit DSD/SDM)... Does a text-to-sound with a 1.58 quantization could output DSD directly ?

Doesn't this do the same thing to an LLM computation that first using UMAP on high density embedding at a 3 dimensional level for visualization does in Topic modeling using classic ML methods? I always use the HDBscan output for hierarchical and algorithmic insights before 3d vis because of the density of the parameters gives out more precise, but harder to handle high density data. The 3d data is still VERY usable and holds most understanding from HD. Is this comparable? Nice to know after struggling with the frustration of Nvidia CUDA update dragging..

can this method be used for only inference or training as well. if I understand correctly the hardware requirements would remain the same for training.

Great explanations

Thank You

"It (BitNet b1.58) matches the full-precision Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance" I don't buy this at all. I will only believe it when I have a model like that running locally in my computer.

did they provide a real example with tests? this is going to be great for low power computational devices like RPI and so on...

This feels like the signal to sell NVDA while they're at their high, before the shift occurs where their GPUs are no longer essential for AI.

When accuracy is so sensitive even to quantization, it's hard to grasp how this makes things better and faster! Every upgrade has a price tag, so what's the catch here? I'm curious to give their model a go, hopefully they add it to Hugging Face.

i wish they released at least one model to test it out lol

How does this differ from all the other quantization methods that we have used so far? I have worked with 2-bit, 4-bit, 5-bit, and 6-bit quantized versions. Is this a completely new approach?

What are the equivalent CPU to a 3090 gpu for this? or you do you still need a gpu for parrel processing ?

Great videos ❤ there it goes Sama 7Ts unless this can't be used in training like quantization. I have been using 2bit xss quants models. they are accurate enough and given you can build them from the ground up to be compatible as 1bit or 1.5bit I think is revolutionary not sure we won't need GPUs at all even with linear mults. I did miss this one with all the noise too.

So what happens to 1bit Mamba models

I still couldn't understand the main thing. In this work, the original model was trained in 16 bits, and then it was quantized in -1,0,1, or did they learn to train the model immediately in -1,0,1, managed to do full backpropagation in this representation, etc.?

Great explanation! It will be great if you can run a more in depth explanation of the paper and share some resources contrasting conventional matrix multiplication with the 1Bit approach. Thanks a lot!

Seems huge! Can't wait to run 70B+ on my phone with millisecond response times

If it ends up being true, I have my doubts but regardless, can we even make models smaller after this? I don't think it's possible really. I think maybe the next step is trying to trim parameters, I think I saw this being talked about once, just trying to find parameters that don't contribute much but it would be computationally expensive

The extreme wording of the title is warranted here if you didn’t know about the October paper, which I did not. So, thanks for the video!

1.58 = Log 3 in base 2. (corrected as below... Uh-doiii!)

Obrigado. Thanks

Some earlier tensor cores of nVidia GPUs (Turing, Ampere A100) have a 1 bit matrix multiplication mode. It was labeled as experimental in the documentation. Not sure if they kept this in later hardware revisions.

This is fantastic news for people running local open source models, if the performance translates.

the idea of having integers as weights made sense to me for a while. but my man 😂only using -1, 0, and 1 is very cool

Hey, where do you get all these papers? How do you stay informed with all the new important papers? I'd like to start reading some specific to my work (data science)

If the 1.58bit model is out performing the traditional model on current hardware what could it do with optimised hardware? Would be interested to know also if it is quicker (and cheaper) to train. Possible energy/environmental benefits too? Could it make putting LLMs in phones and small, portable devices more doable?

Question: If word2vec is able to represent semantically similar words based on them existing near each other in an n dimensional vector space, how can it achieve the same result if there are only three possible positions in each dimension, or are the vectors produced by word2vec solely the data that is fed to the model, but not the model weights itself? Does this also have anything to do with why activations are 8 bit, or is that also unrelated?

Wooo, this sounds pretty cool, and may make CPU inference on low power devices much more realistic. Groq folks probably aren't too happy about this (tho lets be real, this will make groq's already incredible performance even more incredible)

couldn't help but think of quantum computers when seeing the -1 0 1 bits, is there possibility for this?

Is this in any way related to neuromorphic/analogue computers?

Ternary is apparently a very old, alternative number system, which permits decision making with the use of -1. 0, and 1, unlike just binary's 0 and 1. Apparently a computer called Setun which was based on it, was built by the Soviets in the 1950s. It makes me wonder what other innovations might be possible, if we were to look at such obscure and potentially unexplored ideas.

how does it compare to quantized models?

I'm suprised it took so long to study this. Years ago, I realized you could model combinational logic as a "neural net". Weights would have to be +1, 0 (not connected), and -1 (negated). The gates would be like neurons with various activation functions. I never wrote about it because i figured surely someone in the feild had already written a paper on the concept. It seems so obvious.

How does one get reasonable gradients on 1/1.58 Bits?

how would u get the signed bit (+/-) "without multiplying" with the weights

I file this under "I believe it when I see it". I find it suspicious that they only show scores for tiny models (up to 3B), but they have tokens/s and model-size for bigger 1.58-bit models (upto 70B)

if one used this with current hardware, what would happen? Would it be quicker?

no drawback?

This is groundbreaking 😮

hype!

If this is indeed real, thats game changer. But how can the model resolution be maintained with so much compression? LLMs are naturally lossy, but this takes it to the extreme.

SHOCKS THE WORLD

That it took so long to find out that a 1-bit approach is viable, suggests that it was very counter-intuitive, so it is good that we eventually found out, because in terms of hardware, this is revolutionary. It also possibly sheds some light on how neurons work, where the complexity may just be a consequence of the biology, but unnecessary in that it does not add to the performance, but (and this is key) nor does it get in the way. So maybe we can simplify a neuron model along the 1-bit lines.

How do you train such models? I mean, changing a parameter from 1, to 0, to -1 is a big discrete step. How do you know when it's time to change it?

🎯 Key Takeaways for quick navigation: 00:00 *🔄 The video introduces changes in the LLM (large language model) world, highlighting the transition from traditional deep learning models to ones that no longer require GPUs for high-performance matrix multiplication.* 00:43 *🔍 The discussion introduces the concept of 1bit LLMs, suggesting a move towards more efficient computational models that retain performance parity with current LLMs but at a reduced computational cost.* 01:11 *🧮 Explains the shift from 32-bit or 16-bit floating-point representations to a ternary system (using -1, 0, 1) for model parameters, significantly simplifying the computational process by eliminating the need for multiplication.* 03:31 *🆕 Introduces the "B 1.58 model," which uses ternary values instead of binary, enhancing learning capabilities and performance by incorporating a zero value alongside -1 and 1.* 05:08 *💡 Discusses the potential for new hardware development optimized for the ternary computation model, suggesting a significant shift away from GPU reliance and towards more specialized computing solutions.* 06:02 *🚀 Highlights the paper's assertion that the 1.58 bit LLM architecture offers comparable accuracy to traditional models while improving latency, memory efficiency, throughput, and energy consumption.* 07:12 *📈 Provides evidence of the new model's effectiveness through comparisons with the Llama LLM architecture, showing equal or better performance on various metrics, including perplexity and downstream task performance.* 09:58 *🎛️ Elaborates on the technical implementation of the 1.58 bit LLM, retaining the Transformer architecture but altering the numerical representation and computational approach within the model.* 11:48 *🌍 Suggests a significant impact on the scalability and application of LLMs across different hardware platforms, including mobile and edge computing, due to the reduced computational requirements.* 13:11 *📉 Concludes with the potential for dramatic improvements in hardware efficiency and cost reduction for deploying large-scale LLMs, due to the shift to a 1.58 bit computational model.* Made with HARPA AI

Please provide transcripts to study your videos alongside Gemini. Thank you!

Nice paper

Good

Wow. You filled in a lot of gaps for me with your explanation. The really exciting bit is that it's trained at the 1.58 bit level of quantization rather than trained at FP16 and quantized down. Please correct me if I've got that wrong. If I'm right then that should lead to the democratisation of the pretraining stage so any pleb with commodity hardware could do it and we wont have to rely on corporations for pre-training or fine-tuning. I'd be really interested to see the pre-training times and hardware used.

There is a lot of potential here. Originally, I thought this was a completely Ternary process. BUT, I doubt it. So this model is probably using similar method as "BitNet: Scaling 1-bit Transformers for Large Language Models." which explains "BitNet employs low-precision binary weights and quantized activations, while maintaining high precision for the optimizer states and gradients during training."... In other words, this method is not a completely Ternary process. It is still using high precision for gradients etc (If it is using all Ternary, then please explain how it is avoiding the same issues as over quantization)

I asked gemini for about 2 hours how it works.It was mostly unsure.

This seems to be a game changer

Great Video bro. Actually Im doing a project in AI which converts an UI design into front end code. Can you upload videos regarding this. It will be very useful for my project. Thanks in advance

Isn't there some kind of restriction on sale of high-performance GPU/LPU hardware to China? That might be fueling this. It might also be cause for doubt. I hope it's true though!

Wish I could give it two thumbs up. This is almost too good to be true.

reminds me of pcm vs dsd in audio

In computer everything is just 0 or 1. A FP16 is just 16 1-bit digits

This is huge and actually shocking... But, how can the 3 values (-1, 0, 1) do the same work as the 16-bit ones without losing any precision? Also, wouldn't this mean that something on the level of GPT-4 will work with a single powerful GPU (something like a 3090) if it uses this 1.58-bit form, as it will scale down massively both in size and required computational power? edit: A deeper dive on this if possible will be helpful.

I checked the paper for a sec and found these models actually eat more memory than 4-bit quantized ones and don't offer much of a speedup compared to them either. Don't know where the memory inefficiency comes from and whether it could be fixed. If this is the best it can do, it's quite Zzz

huge news

i think using -1,0,1,2 with 2 total bits would be a little smarter but i see why using only one bit is easier because multiplication on a single bit is just AND

If this is true, ternary computing just found its application, finally. But I doubt this representation is useful on current hardware, or faster than 8 or 16 bit integer. Well, unless the matrix library does some mad bit banging and masking.

if instead of 3 you use 4 values (value not present at all being the fourth ) you end up with 16 values instead of 9, when you actually use it in the calculation you just shift by 1 or 2. please explain your 1.58 bits better, how do you get this value?

Are we going to hit a wall with LLM? Garbage in garbage out, we need a system that verifies the AI is not corrupting itself, plus we need more energy to feed this monster. Is the Singularity near or far?

Wow! This takes LLM Design out of the realm of high-level abstraction and pulls it down to the hardware level. Looks like old C++ Guys like me are about to become relevant again. If the performance of these models can match that of FP16 on the output side, then this is truly game-changing. The cost of building out AI Infrastructure dropped by several orders of magnitude. All we need to do is develop efficient libraries for performing matrix math using 2-Bit Unsigned Integers.

Just wonder if anyone can reproduce their result lol

So, going from "All You Need is Attention" to "All You Need is 1.58 Bits"?

GTC is coming up, Nvidia sessions should be online soon. Ive been checking them out more and more and Remix tech is raging.

I thought with quantisation the model performance would drop. Maybe the memory requirement has gone up as a trade off? The matrix dimension has to increase to have all of the information be stored in some way. i m not expert in anyway just feels its not that simple..

One network with many hidden layers = deep neural network

Thala suthudhu how it performs like 16

One more step towards the merging of AI neural networks and quantum annealing style computing...

If I were CEO of Intel I would be working on this right now.

atleast theres no “shocking” this time

Very intresting. I think I'll need to read the paper though. For the neural net to be able to approximate any functions I would of thought you need at least some form of multiplication for it to be a non linear model (as in non-linear classification thresholds or non-linear reggresions). If they are not multiplying values with wheights then I would at least expect that some form of multiplication is being done in the activation function.

I like this title more, it's a lot less click-baity. ;-) Though I still wonder if this even changes anything at all. If the paper is is already from October last year, it's old by todays standards. So there is probably a good reasons why it has not been adopted yet.

everyone makes this viral so it's will get more attentions such research is our future of AI. gooooooooo

The main issue here is these are tiny models they are playing with. There is no proof that this technique scales to anything above (in terms of perplexity/quality). It reminds me of RetNet, which was also supposed to be a big breakthrough and hasn't been released with any open weights since.

This essentially creates different "engines" ... like cars there are v4 and v12, you drive what you can afford! Currently they're all too expensive lol

3 values cannot be represented using 1 bit

I don’t know why they ever used floats. Seems like complete overkill. I suspect just because GPUs were originally used for gaming.