via Pi "The Kolmogorov-Arnold Network (KAN) is a fascinating type of neural network that takes a unique approach to learning compared to traditional multilayer perceptrons (MLPs). Here are some key points about KANs: * KANs have learnable activation functions on edges (or "weights"), while MLPs have fixed activation functions on nodes ("neurons"). * This design choice is based on the Kolmogorov-Arnold representation theorem, which states that any continuous multivariate function can be represented as a superposition of continuous functions of a single variable. * KANs can achieve better accuracy with fewer parameters compared to MLPs. * They also offer advantages in terms of model accuracy and interpretability. * The concept of KANs was introduced in the paper "KAN: Kolmogorov-Arnold Networks" and has since been implemented in various libraries and frameworks, including the PyKAN Python package. Overall, KANs provide an intriguing alternative to MLPs, with their theoretical foundations in the Kolmogorov-Arnold representation theorem and practical advantages in terms of model performance and interpretability."

Thank you so much for such great explanation!

what surprises me is that nobody tried to used splines before (they probably did).. seems like an obvious way to approximate functions

Thanks for the explanation Grabriel!

Wow! The extractability of solutions really catches my eye.

Good explanation! 13:13 btw that order n Bazier curve recursion is just a linear interpolation of two order (n-1) bazier curves.

Thank you very much. You explained very clear

Thank you for this clear and comprehensive explanation. Also your insights about the downsides of the model was quite accurate and helpful.

thank you for the explanation

nice explanation

Great explanation! However, instead of "activation matrix," you should probably say "activation function matrix." An activation is the output of an activation function, which is not what this matrix contains.

Summary: while current RNs work as a universal approximator (probability machines), Kolmogorov networks allow you to leverage a more deterministic function. End point: the axiom of the universal approximator as a pillar of current networks could be replaced by networks of variable geometry. Think about the difference between searching or decoding information in 1 dimension, and now adding 2 dimensions. A data point vs a line of data. Like go from 480p to 8KHD resolution.

Thanks for the greatest video. I have some questions, I can't understand how that algorithm update the c (learnable terms)? back propagation? or something else

"Forest has been lost behind trees"

MLP can't learn this, but their method KAN =)

Greta video! Any chance you can cover xlstms? It just released an hr ago lol

I am not going to lie. I am unsure about the authors motivation of using a B-Spline and paper as a whole. The authors simply want to replace the linear combination of inputs with a non linear combination. So why not just replace a linear layer(with activations) of a pytorch network with a "recursive linear layer", where every weight of the linear layer is simply another linear layer. Much simpler to implement, just a small 5-6 line for loop, and also fits within neural network literature perfectly. No need to prove, with regards to Kolmogorv-Arnold theorems, because our formulation is simply another neural network. Whats more this relieves the model from performance issues that the authors express. What am I missing ?

Thank you for this very helpful video. One thing that I’m curious about is whether the assumption of a spline relationship is a valid assumption for all use cases or not.

The idea of combining the splines for the activation function is in any way similar to RBF networks ?

Someone who made the acronym knew that it will remind us on KAM - Kolmogorov-Arnold-Moser.

It looks like multi layer version of GAMs to me (Generalized Additive Models), which has problem when extrapolate

Gabriel if only you could start writing in a readable way some random postgrad at mit would solve the agi by now. love the vid, big thanks

How about liquid neural network ?

Hello everyone, can you please tell me if I understood correctly that splines are built based on the input data, then the neuron calculates the activation function using these splines (which then go as weights into the matrix) and the result of the calculation is passed on to the other layers?

Hm, suppose you wanted to learn some function in a way that, using some kind of constraint or prior, would be expected to generalize to regions far from a ball bounding the training data (so, not just, “far from any point in the training data” as a consequence of the input data being high-dimensional, but far from a ball bounding the data. (Not like with language modeling or whatever, where the input values should have bounded norm after embedding) How might one impose constraints that would still apply outside that region? Like, I’m imagining using some kind of algebraic property of the non-linear activation function, in order to express some bound on something or other, and like, put that into a loss? Like, for ReLU, one can say some nice things about it, as, for positive inputs it is just the identity, of course. And MLP network using RELU can in principle be expressed as a piecewise-affine function of the input, with finitely many regions (though the number of regions may be exponential in the number of “neurons” I think). The number of these regions is much too large for this to be practical, but, seeing as there are only finitely many of them, the union over the ones of those regions that are finite, is finite. I wonder how large the number of infinite such regions is. Well, I imagine that this depends more on the dimension of the first layer and the last layer? Each of these regions is convex, and is defined by a system of finitely many inequalities. Uh, any infinite region, there will be a ray which it contains. (These regions are subsets of the space of possible inputs to the first layer, to be clear) If we associate to each infinite region, the set of directions such that that region contains a ray with that direction, well, I think that the different regions sets of directions will only overlap at the boundaries. For any direction in the interior of the set of associated directions, I think that moving far enough in that direction will always land one in that region? If there are few enough of them, maybe the loss could have terms based on the affine behavior in these regions, to encourage them to have particular properties? Like maybe invertibility or something? Idk.

Thanks for the video

I published c# KAN, which shows about same performance and accuracy, it does not use any 3rd party library. Interested?

what is the video about splines?

quality content

why don't you think it won't scale?

I was thinking the same thats its gonna be time complex as it sounds, 😅 but they wrote initially that it requires fewer parameters &faster inference.

Nice . Real AI channel. For engineers and not for blogwriters script kiddos

You completely lost me on the splines, not necessarily your fault :)

This is Whats next, show your skills? 1. CodeCraft Duel: Super Agent Showdown 2. Pixel Pioneers: Super Agent AI Clash 3. Digital Duel: LLM Super Agents Battle 4. Byte Battle Royale: Dueling LLM Agents 5. AI Code Clash: Super Agent Showdown 6. CodeCraft Combat: Super Agent Edition 7. Digital Duel: Super Agent AI Battle 8. Pixel Pioneers: LLM Super Agent Showdown 9. Byte Battle Royale: Super Agent AI Combat 10. AI Code Clash: Dueling Super Agents Edition

Im sick of neral networks, we need to shut them down

Its MLP in disguise