As someone actively working on this stuff, this channel has the best explanations on the internet, and the 'tuber actually understands what is going on.

wow, you've made some difficult i mean extremely difficult algorithms look easy. thank you.

please open your community tab your content is incredible

One small note on RNN's, reservoir computing is a very high dimensional random RNN with linear regression readout, therefore there is no exploding nor vanishing gradient. Reservoir computing is currently the standard for non-linear dynamic time series prediction

Nice video! I just wanted to point out that the parallel scan algorithm can be also implemented in O(n) time (instead of the O(n log(n)) version peresented in the video. and this is the version that the MAMBA uses.

absolutely love the quality and information of this video!!! please keep up the good work this is amazing

Absolutely amazing vid. Just subbed after getting recommended to this channel. Never stop making videos dude

Thank you! Your channel is an invaluable resource on here. Hope you keep making these videos!

Brutal. I'm going to have to watch this about 30 times. Love it.

Currently testing it on molecular generation, so excited to see where these strengths hold and where they falter :)

Wow, excellent explaination. It covers all the essense of the paper with just enough math/algo. Thank you so much ! If you dont mind, plz make a video for RWKV (v6 has some new modifications), which is another strong linear RNN model. I am curious how does it compares to mamba.

This was very good, and I hope you make more videos like this!

Awesome video. I love the speed and the depth of this, it's perfect

Amazing explanation, thank you!

You have such a pleasant voice 😊 Thanks for helping me understand better. Please keep making videos. ❤

About peer review: As one comment noted, there could be many more candidate papers presented than could be accommodated at the venue. However, this video argues, the rejection justification for this paper is inadequate at best. Some comments ask whether the rejection is important; for academics, the answer is yes, because presentations and publications count for tenure, promotions, and raises plus continued funding of the research. Since several comments plus the video indicate that the algorithm had already received a lot of publicity, for the sake of the project it may not matter if it can continue to be funded, especially if commercial implementations are successful. What is interesting in any case is that the paper exists; in effect it has been published; the authors may not get the desired credit for formal publication, but their work and the reviewer comments are out there now. A couple of decades ago that would not have been the case; most people in the field would be unaware of the algorithm. In terms of peer review, in general (outside of AI), in my field, one of the natural sciences, a paper I submitted for publication encountered an editor plus two reviewers who were well qualified in the field; after asking for two revisions to the manuscript, the third version was rejected. Interestingly, all three scientists had published research which my paper undermined; they may well have lost funding for their research or even their position had that manuscript of mine been published (I speculate here). Peer review cuts both ways. While iterating with the editor and reviewers I continued to expand my research project and made some additional discoveries. Following the rejection I wrote a completely different paper which incorporated my initial work supplemented by the new discoveries; happily it was published a few months ago (in a different journal). I'm formally retired now, but continue to do research. To young researchers -- never give up. Learn from rejection, refine your work, be humble, exercise integrity and honesty, and take pride in your accomplishments, even if only a few know about them. Peer review (by humans) is a necessity and will continue to be. There is no such thing as a perfect filter, but science and technology would be overwhelmed by irrelevancy, dishonesty, and duplication of effort without it. AI may become a useful filtering tool, but science is a human endeavor.

Great video! Thanks!

Thanks for this explanation! Phrasing mamba in terms of a Linear RNN makes it much easier to understand. You've done a lot already with this video, but I just want to ask for a little bit more. Since the original Mamba paper presented the model in terms of SSM, many, many implementations of Mamba also use that language. And I have difficulty wrapping my head around trying to map their code back to the concepts in this video. I wish you can explain how concepts in the Mamba paper (∆ A B C D, discretization, etc) maps back to the parameters of a Linear RNN, which would help a lot.

we need more videos from you, especially one from basics

8:50 Analogous to FFT algorithm, which has the same Olog(N) complexity.

Incredible work. I mean REALLY incredible

Thx a lot for the interesting video! 💛💙

I really wish that when you're talking about things happening in parallel, your animations happened in parallel. Like 8:30. I think it would really improve the comprehensibility of your explanation

This is amazing!

Another beautiful exposition. Further points: (1) HiPPO itself comes from attempting to approximate a spiking net with a SSM (Voelker 2017/8), (2) we do have O(NlogN) transformer hacks now, (3) RWKV is a promising arch that deserves a place in this arena.

great video. That trick around the 26 minute mark of doing 16x compute almost for free (in terms of time) because of memory bottlenecks is really neat. I wonder how many other architectures would benefit from that kind of design optimisation?

underrated channel

I believe that the transformer does have a quadratic cost in memory (specifically self attention (SA)). The attention matrix in SA is n by n, thus n^2 (n being the number of tokens). Probably the reviewers is referring to that bit. Anyway, rejecting mamba was hecking stupid. Great video!

Good job 👏

Amazing video, insta-sub!

I appreciate the soothing piano music. Currently the words are only slightly better than Charlie Brown listening to adults talk, but I hope to dive in.

Good video to explain mamba : I understand something

Loved your videos. Which software or library do you use to make these animations? Is it manim?

Here’s an idea that probably wouldn’t work: What if instead of algebraically guaranteeing that some operation is a monoid so that one can use the parallelizing thing that combines n inputs in O(log(n)) steps in n processors, what if you just had some operation, learned by a NN, which has “how much it deviates from being a monoid operation” as part of the loss? Like, suppose you randomly selected some pair of consecutive applications of the operation, and also computed it in the opposite order, and took the L^2 norm of the difference between the results, and multiplied that by some weighting, and made that a term in the loss? Like, within the family of continuous and piecewise-smooth monoidal operations, perhaps some of them would be better at selective remembering?

the channel is great and the material is awesome! the only catch is: the piano in the background makes it hard to focus..

I like how we now call 1 billion parameters small.

This algo is new and you made a video about it I love you I will subscribe your channel keep going

If you ever get the time I would love to see another video on mamba implementation but dumded down even more. Like to the level of statquest videos. They need to make you feel special while also showing the math step by step like its 9th grade.

Peer review is broken nowadays because people have little time to actually read through a manuscript with attention to details given the amount of pressure to publish their own papers. So when you have more papers out there than the time people can spend on reviewing, you get low quality peer review.

There seems to be a growing zoo of related architectures that attempt to supersede the transformer. Besides Mamba, there's also RetNet, GLA, Based, and HGRN. And the secret upcoming xLSTM. Someone also mentioned RWKV. Are all these converging to something? And when will we see a frontier model based on this new paradigm?

Your videos are so good man keep it up, seriously. Although that is probably beneath you, but could you maby make a video on how neural networks are computed on machines in general or maby on GPUs? As someone who did not learn computer science in uni, this would be an interesting topic for me to learn and maby fundamentally understand nn better.

Woah big claim! I’m excited

This just shows how RNNs are way too natural of an architecture to ignore. Maybe solution to a gradient descent problem is to not use gradient descent at all. There has to be a different way to update parameters than this bizarre hack and slash let ||x_0|| = 1 for RNNs.

Nice video! What I didn't understand is what happens to the stable weights during training. Particularly: - How are they kept stable? - How can the model learn while being so restricted? What I'm guessing is that some form of the Delta is also used in training to keep the weights in those ranges + rely a lot more on the accuracy to carry the information. Is this correct? Does it imply that using double instead of float gives it a better ability to learn?

So how close is the weight estimator to the MMSE (minimal mean square error) estimator? Can the MAMBA arch be improved even more, using a sparse covariance matrix and an application of a 'true' Kalman filter? Or is it already as close as it can get?

RNNs are constrained by having to hold all their information in a single embedding space, so this space needs to be extremely large. It needs to hold every piece of information in the context that might come in useful at some point. Transformers can distribute information between many tokens, so can operate with a much smaller embedding space, at least in theory. The memory complexity of a RNN with a given structure is quadratic on the size of the embedding space, meaning we really pay big time for that increased embedding size. I wonder if that is what the reviewer was getting at. The results were impressive, but they haven't been followed up by success at larger model sizes which I would have expected to have already happened if it was going to. It is a cool mathematical trick to make it work, and demonstrates that language is surprisingly linear, but once you start to hit truly non linear questions I would expect it to stop improving. Overhyped IMO.

Would you mind explaining this associativity 10:37 ? My assumption is that f is the linear recurrence function, but how is it equal to a pair of the matmul between W2 and W1 and the second term? Wouldn’t f output a vector, so how could it be equal to the right hand side pair of vectors?

Great video, would prefer no music but that’s me

Can you make an explanation video like this one on Liquid Time Constant Networks 🙏

Enjoyed this. Given that its performance is comparable to or better than transformers as verified independently in several papers, is Mamba gaining a foothold among practitioners?

Appreciate the breakdown. I think there are a few more things at play here for the reject that is somewhat overlooked in the discussion at the end. Specifically, there are issues with anonymity and using "hype" to push a paper through an academic conference. I speculate that this was the underlying reason for rejecting the paper.

Love the video but I have the question: Shouldn't be the approximation at 17:00 be something like n*w^(n-1)*0.001*x, so isn't there an n missing? Or how was the approximation done?

Extremely noob question but, at 13:52 why aren't the input vectors x multplied by P^-1 instead of P? Don't you need to convert them to the eigenbasis before applying the D transformation (or, equivalently, taking the hadamard product with the diag(D) vector)?

One thing I don't understand is the HIPPO matrix, and what they mean by a structured matrix in the context of differential equations.

❤

Mamba Mentality

GPT mafia 😞 Probably just can't lose their faces and title of "the best LLM tech" (and, perhaps, contracts as well).

I haven't found any significant evidence suggesting that Mamba models outperform Transformers, except that their attention mechanism does not scale quadratically with the context length. Am I missing something?

[6:28]: While that sound somewhat good in practice it doesn't work like that Alternating between linear recurrent and non linear dense doesn't give that much of context in advantage :( The gradients vanishes or explodes after a while and requires some sort sigmoid transformation + some value Say for example an architecture like this: ```plaintext Dense -> Sigmoid -> Recurrent -> Dense -> Sigmoid -> Recurrent -> Dense -> Softmax ``` Until the gradients reach the first Recurrent the gradients loses most of it's value :(

Can skip connections be used here to tackle vanishing/exploding gradient problems?

peer review be like thats a nice method for building houses, its a shame it doesn't also cook burgers what

21:48 33%? Dude, it's 3.4x improvement. Measuring improvement relative to accuracy instead of error rate is dumb, since that'd mean that difference between 100% accuracy and 99% is just 1%, which is not representative of anything.

People keep making things that they say are "better than transformers", but none of them are actually getting used. At this point, hearing people say that has sort of become meaningless from the number of false alarms. Feels like every few months we have something "better than transformers", like RetNets were claimed to be. We'll have to wait and see which actually turn out to be better with time.

great video as always but would vbe even better without the distracting background music.

Eh - re controversy, I don't think peer review broke here. The *conference* however, took a much deserved status hit for mismanaging review.

"Small models up to a few billion params" I think people have forgotten what small means 😂

Who cares about these academic peer reviews? 😅 But anyways, the only downsides I have seen in S6 is it is not always stable during training(I have seen huge outliers) and it performs worse than transformers in copying.

It’s faster but not better, bro!

Science paper peer reviewing could become a fight for fame and carreer amongst greedy scientists, more than a filter for truth, using small insignificant mistakes as excuses for paper rejection.

I love those sleep inducing videos with someone just talking gibberish that no one understands 😇

What are the perimeters of the human brain, 86 billion ???

5:02 for people who don't get it, vanishing/exploding gradients is just basically dementia for AI in a way, except it's where the AI is starting to become incapable of learning any further

I'm dizzy

Best review of Mamba on YT, but still fails to address "better" at _what_ than transformers? language modeling? what is that? Next token prediction? (_what_ prediction...). There's still this big hole in AI theory, it doesn't feel like anyone has any idea what framework they're even working under... nobody is sure what they're doing. There are all these papers that show xyz score on xyz benchmark... a benchmark somebodies dog came up with over breakfast... and nobody has good evidence of what property xyz benchmark is even measuring, why we should care, never-mind what it's supposed to 'mean'.

lmao Mamba got rejected from ICLR

Love everything about out your videos thank you so much, but please reconsider the background music (as in removing it). Those who desire it can easily add in their own. I find it distracting.

faster ;yes, better ;how?

Dude, the music in the background is killing me -.- Silence would be much better!

NIce, but please stop uptalking, you are several levels above this habit.

❤