Here is how Transformers ended the tradition of Inductive Bias in Neural Nets

Рет қаралды 6,194

Күн бұрын

In this episode, I discuss Transformers and the role of Attention in Deep Learning and everything one needs to know about them!
Especially how they changed the usual AI research course of aligning towards more inductive bias in neural nets (CNNs with their locality bias, RNNs with their recency bias) and embraced a more general, but data-hungry architecture.
To support the channel and access the Word documents/slides/animations used in this video, consider JOINING the channel on KZfaq or Patreon. Members get access to Code, project files, scripts, slides, animations, and illustrations for most of the videos on my channel! Learn more about perks below.
Join and support the channel - www.youtube.com/@avb_fj/join
Patreon - / neuralbreakdownwithavb
Check out the previous two videos on Attention and Self Attention that covers a lot of the groundwork behind this video:
• Neural Attention - Thi...
• The many amazing thing...
Follow on Twitter: @neural_avb
#ai #deeplearning #machinelearning #neuralnetworks
Useful papers:
arxiv.org/abs/1706.03762
arxiv.org/abs/1409.0473

Пікірлер: 16

@TP-ct7qm 6 ай бұрын

Awesome video! This (together with the last two videos) is one of the best explanations of Transformers I've seen. Thanks and keep it up!

@AI_Financier 6 ай бұрын

the gist of this video: 4:29, a great job, thanks

@JasFox420 6 ай бұрын

Dude, you are a treasure, keep it up!

@hieunguyentranchi947 5 ай бұрын

This is gold I hope it gets the ATTENTION it deserves

@avb_fj 5 ай бұрын

Thanks!! More attention will surely TRANSFORM this channel! 😂

@GabrielAnguitaVeas 2 ай бұрын

Thank you!

@amoghjain 6 ай бұрын

wowww! what a great explanation! helps knit so many individual concepts together in one cohesive knowledge base!! thanks a lot for making this video and all the animations!

@avb_fj 6 ай бұрын

Thanks!!

@matiasalonso6430 6 ай бұрын

Congrats !! Awesome channel !

@avb_fj 6 ай бұрын

Thanks!

@IdPreferNot1 2 ай бұрын

Came here after interest from one blue three brown. It's clear you've got a great explanation style... plus you were earlier ;). Hope your channel following builds to match your outstanding quality.

@avb_fj 2 ай бұрын

Welcome! Thanks a lot for the shoutout!

@user-wm8xr4bz3b 23 күн бұрын

at 2:33, you mentioned that self-attention is more biased, but at 2:54 you also mentioned that self-attention reduces inductive bias?? Sorry but i'm a bit confused.

@avb_fj 23 күн бұрын

Self-Attention indeed reduces inductive bias and adopts a more general learning framework. At 2:33, I am asking a question: "IS Self-Attention more general or more biased?" And then I continue with "I'll argue that Self-Attention is not only more general than CNNs and RNNs but even more general than MLP layers".