Join Shortform for awesome book guides and get 5 days of unlimited access! Get 20% off at shortform.com/artem

Ladies, gentlemen, and fabulous folks of every flavor, the legend is back!

In fact, many proteins function in a _local_ minimum that is _not_ the global minimum. This is why proteins denature irreversibly when exposed to heat; there's an energy barrier that they can never come back from if they cross it.

This is an insanely educational video, as a ML researcher working on representation learning for multi modal retrieval, this is insanely helpful and relatable. I think you just gave me a new area to look at now, how exciting, i owe you one.

John Hopfield wasn't the first to describe the formalism which has been subsequently popularised as "Hopfield networks". It seems much fairer to the wider field and long history of neuroscientists, computer scientists, physicists, and so on to call them "associative memory networks", i.e. Hopfield was definitely not the first/only to propose the network some call "Hopfield networks". For instance, after the proposal of Marr (1971), many similar models of associative memory were proposed, e.g., those of Nakano (1972), Amari (1972), Little (1974), and Stanley (1976), which all have a very similar (or exactly the same) formalism as Hopfield's 1982 paper. Today, notable researchers in this field correct their students' papers to replace instances of "Hopfield networks" with "associative memory networks (sometimes referred to as Hopfield networks)" or something similar. I would encourage you to do the same in your current/future videos. I deeply regret making a similar mistake regarding this topic in one of my earlier papers. However, I am glad to correct the record now and in the future. Refs: D Marr. Simple memory: a theory for archicortex. Philos Trans R Soc Lond B Biol Sci, 262(841):23-81, July 1971. Kaoru Nakano. Associatron-a model of associative memory. IEEE Transactions on Systems, Man, and Cybernetics, SMC-2(3):380-388, 1972. doi: 10.1109/TSMC.1972.4309133. S.-I. Amari. Learning patterns and pattern sequences by self-organizing nets of threshold elements. IEEE Transactions on Computers, C-21(11):1197-1206, 1972. doi: 10.1109/T-C.1972.223477. W.A. Little. The existence of persistent states in the brain. Mathematical Biosciences, 19(1):101-120, 1974. ISSN 0025-5564. doi: doi.org/10.1016/0025-5564(74)90031-5. J. C. Stanley. Simulation studies of a temporal sequence memory model. Biological Cybernetics, 24(3):121-137, Sep 1976. ISSN 1432-0770. doi: 10.1007/BF00364115.

4:00 LOL "The ball doesn't search through all possible trajectories to select the optimal parabolic one." The visualization of the "trajectory space" is even funnier ;) I suspect there's different encodings for proteins with identical function, but which are more robust wrt folding consistently.

At the beginning, when you mention the O(n) problem, as a programmer it just intuitively makes you want to use a tree or a hash map lol In any case, another banger! Its fascinating to see how these things work!

Minor correction: 'Cells that fire together, wire together' was coined by Carla Shatz (1992). Unlike Donald Hebb's original formulation, Shatz's summary of Hebbian learning eliminates the role of axonal transmission delays. By extension, neural networks which remain true to Hebb's original definition should go beyond rate coded models and instead simulate the time delays.

I was thinking about your channel less than an hour ago.

Thanks for the incredible quality in your videos

Finally, you are comeback 🎉

Really good that you’re covering these foundational concepts of NNs. Cross-associative NNs, auto-associative NNs, and unsupervised learning are big missing pieces in today’s NNs.

Exceptional pedagogical skill! I’m not able to hold these types of explanations in my mind, so any attempt at following such a web of relations would quickly have me lost. But this is a masterclass in clear considerate communication 🙏

Respect for using Coldplay 🔥🔥🔥

> "when you throw a ball the ball doesn't search through all the possible trajectories" QM has entered the chat

That excitatory and inhibitory connections remind me of statistical correlation function

This was one of the good oness. I really loved it and hope part 2 comes out sooner. Keep up the amazing production sir.

Your back!!! awesome vid as always.

Waiting for your video so long. Thank you so much

This looks like a run-of-the-mill gradient descent, how does resolves false bottoms?

Man, really thankful to your contents. I was facinated by your video about TEM, and started trying to fully understand that network(and memory in general) in my leisure time since about a year ago. I learned about latent variables, transformer architecture(fantastic videos by andrej karpathy), autoencoders, etc, but got stuck at (modern)hopfield nets, which I think is super important in the architecture of TEM. Very glad to see that you start to touch this field of Hopfield Nets, this is probably the best video about vanilla hns I've ever watched. Really looking forward to your video about Boltzman Machines and Modern Hopfield Nets, always appreciates your videos!

Another great video! I really liked your energy landscape and gradient descent animations especially.

very cool video, keen to see how the broader arguments progresses in this series

Hopfield networks are amazing! They are studied in physics, biology, machine learning, mathematics and chemistry The rabbit hole goes extremely deep

Much love bro incredible video ❤ thank you

I love that you made everything dark mode. (Noticed when I saw the Wikipedia logo)

- Excellent! Thx. - Very well presented: clear/concise, yet fairly comprehensive - and w/ great visualizations. - Keep up the great content!...

This is really cool! Thanks for your work Artem!

Awesome video, like always. Just a small nitpick: your speaker audio jumps between the left and right audio channel, which is quite distracting - especially with headphones. You can easily solve this by setting the voice audio track to "mono" when editing the video. Cheers

Awesome. I just can’t wait for the next video!

Fabulous, bravo

As always, outstanding video!

I wanted to do research on something like this a year or two ago. This is amazing, I've got some work to do with this.

I was reading and watching videos about metacognition and bayesian probability and now you have thrown me into a new rabbit hole! 😅 Your videos are incredible and it's great to have a new one. Thank you!

This was really clear, accurate, and easy to follow. 10/10, would watch again.

Hello. It's been long since you last uploaded the last video. I hope you are well. Best videos, bro. Keep them coming 🎉

So good. Thank you

Man, I just love your videos

One of the best most Clear videos I’ve ever seen EVER❤ 🙏🏻THANK YOU ❤😊

This is incredibly well put together.

Super interesting to see that the hopfield network basically reinvents binary operations like XOR and XNOR for neurons, with the two differentiated by the weight.

Love you man

The protein example got me thinking. Is there only one unique folded configuration of the lowest energy? Can there be multiple stable comfiguration anyway, and transitions between them?

Holy cow, this makes a complex topic so intuitive.

great video as always

Can wait to see this video!!!!!

Waited for this...!!

One reason I like using Neo4J is that graph networks seem like the work a bit like human memory with links between things making finding related items fast.

please try to make more videos, your content is extremely good

Another awesoem Video :)

Thank you so much for creating this video, genuinely one of the most educational I've ever come across. I've been trying to learn more about how brains work since that's always been something I've been very curious about literally since birth, and along with entropy being my favorite physics concept this video has just led to me googling and researching for the last 4 hours (its 3am lol) trying to find out more. Really impressed with how complicated, yet still high-quality and clear, some of the topics are in this is and I'm really looking forward to watching the rest of your videos to learn more on how all of this stuff works in such an intricate way

Thank you. This is unbelievably simple and potentially more accurate than any other speculation. Next step: determining the shape of the proteins and categorizing them. The tools may not be there, yet... but a good hypothetical certain helps to reach certain conclusions.

Great video

This is one of the best channels

Кайф слушать и офигевать) Greetings from Austria, keep doing what you're doing!

Awesome

12:21 Watching this to maximise my happiness. Max happiness Min unhappiness, this is the way. Thank you!

Great video as always. For 22:50, has anyone tried stacking layers of Hopfield networks yet as a work around? Basically each layer acts in it's own feature level space and resolves for that level's most likely feature, then passes it up to the next higher order Hopfield feature space to be resolved there. It seems like this would allow you to store exponentially more overall patterns has they get resolved separately to avoid the overly busy end energy landscape. Also interestingly you can see how it would carve out the energy landscape from just the raw inputs with this. You have the Hopfield network constantly comparing itself to the some abstraction of the source input layer, meaning the more times a pattern seen the stronger it gets in the Hopfield network. Also for faster convergence, it is likely the greater the xi and hi difference the faster xi updates.

Reminds me of gradient descent

My God, your videos are amazing

2:16 a* great introduction

Genius !

Plz make a video about modern hopfield net or dense assosiative memory. Cuz it a different and generalize perspective of mopdern hopfield nets.

This content is so high level, it's almost impossible to tell if it is true or not. Physically and philosophically, I have bought in, but my want of it to be true doesn't make it so. I cannot imagine this is wrong, but where does this come from? This stuff is just crazy and I don't know if it is crazy good or just crazy lol but I'm along for the ride

Can you do a video on the work of Dmitry Krotov showing that attention mechanisms are a special case of associative memory networks

Si that means that our neurons do some type of gradient descent? That’s very interesting to know

Agree 💯

Long time no see😁

Didnt you already upload a video on boltzman machines? I thought I saw it last year.

I've always thought of energy has unhappiness, too.

Great video. Little sad to see that anti-training wasn't mentioned. It doesn't really solve the problem with training two sequences that are 'close' together, so that's fair. But it does help, and has an interesting analogy with physiology. Essentially, if you try to train two sequences that are too close to each other, their valleys will overlap, which means you might try to aim for one specific sequence and end up falling into the other. And if they're really close, you'll actually create a new local minimum that sits between the two. In those cases, what you can do is identify all your local minima and then run the algorithm backwards, training hills on top of all your local minima. For stand-alone minima, this doesn't matter because they're still local minima. But if a minima is a false sequence that sits between two or more neighboring targets, this builds a hill in between those two neighboring valleys, helping to make those nearby sequences more distinct. As Geoffy Hinton has pointed out, this has an interesting conceptual analog to dreaming, where we seem to replay experiences and concepts from our day (to a vague extent) and sleeping/dreaming also seems to help with learning and memory. Similarly by making the Hopfield focus on its memoreis while playing them backwards, so to speak, helps to solidify its own memory. It may be little more than a metaphorical analog, but I think its still quite interesting.

Just in time for my studies into Z12 cyclic groups, which surely have nothing to do with cognition 🧐

Isn't the 2nd law of thermodynamics more directly linked to entropy? Is there an analog for entropy in the associative memory network?

Because the neurons have two edge weights (A->B and B->A) does that might explain the case where we have the memory just out of reach and get lock trying to get it?

Wow

Just as I suspected, everything is just the principle of stationary action! It's all just making an action functional stationary what if we considered non-local actions?

Your font missed the contextual ligatures

AGI will likely need thinking in this way, and creating at least partial ideas of how our brain works.

Do you have merch?

I wonder if the convergent process can be done parrellelly state by state. That would be awsome.

STUFF LIKE THIS IS SO INTERESTING TO ME TO LEARN😮‍💨GOD DESIGNED SUCH AN AMAZING CREATION🤩🙏🏾🙏🏾🙏🏾

This seems like Friston's explanetion of inference. Can you explain that to in your videos?

It seems to me this "fire together wire together" notion is actually also present in the attention mechanism of transformers, except you don't just have a 1D 1 bit + - but rather an nD dot product. This still has the same basic structure: Neurons try to more closely align to each other. But the added dimensions give each neuron more ways to accomplish that: Among three neurons, one neuron can be pretty close to the other two while those two might be pretty far away from each other.

Wait, this is just the Ising model with extra steps!

😍

Great explanation! This applies to fully connected networks, correct? So memories are Dense Distributed Representations. It's not how the brain does it. The brain uses the opposite: Sparse Distributed Representations. Is there any way you could explain it using SDRs?

Hey, someone pointed me to your page because of the links between neuroscience and ML. I think that the thalamus might be a cross-modal auto encoder. Main reason I think this is because there are 10 times as many connections going into the thalamus as there are going out, sounds like encoding to me. I was wondering, does that ring true to you?

❤❤

Does this mean artbros were right about diffusion models being databases?

how does this relate to fristons free energy principle?

I use an excell spreadsheet for all my memories

Hi there great videos! I am curious about your view of free will and agency. I myself explore this philosophical/psychological/biological subject, and your videos are reinforcing notions I found in books like "chaos and nonlinear psychology, Schulter et al" and "determined, Sapolsky" among others. It seems to me that, from a scientific point of view at least, the notion of dualism, and free will, doesn't make sense. Is that something you think about? :) In any case keep up the work, it's really appreciate here. Antoine Dion, Canada

Resumindo: intuição inconsciente guiada pelo princípio da conservação de energia do ego absoluto.

🙇

Comment for the algorithm

Basic question: you are using word "inference" for outcome during the training stage (about 15 minutes into the video). But I've heard that word being used for specifically only the recall stage, not the training stage. For example in the transformers use. Can you clarify? Are you simply wrong using that term? Or does the usage vary?

sir so proteen folding is affected by gravity too 😵😱

I didn't want to sleep anyway.

❤❤❤❤❤❤❤❤❤❤❤❤

Ironically, within the framework of quantum mechanics, one could actually say that the ball *does* "search" every possible path in order to find the "correct" one. It simply performs the "search" in parallel, not sequentially. And it's less of a search and more of an average of all paths. The principle of stationary action is the driving principle behind Newtonian dynamics and itself follows directly from the interference between many "virtual" trajectories, it turns out that the paths which are close to the "true path" (the classical path) have very little variance in their action, which rough speaking means that they end with nearly identical phase shifts (e^iHt/h, Ht ~ action, h = Planck constant) and can interfere constructively, whereas paths which are far from the "true path" have wildly varying actions, even if two paths are similar to each other. So they pick up big phase shifts and end up interfering destructively, leaving only the contributions from the paths "close to" the classically observed path. As far as I know this is the only way to derive the principle of stationary action, and the same basic idea is essential to finding transition probability amplitudes in QFT. It really does seem like the universe simultaneously tries all conceivable paths, superimposed together.