Positional embeddings in transformers EXPLAINED | Demystifying positional encodings.

Рет қаралды 64,689

Күн бұрын

What are positional embeddings / encodings?
📺 Follow-up video: Concatenate or add positional encodings? Learned positional embeddings. • Adding vs. concatenati...
➡️ AI Coffee Break Merch! 🛍️ aicoffeebreak.creator-spring....
► Outline:
00:00 What are positional embeddings?
03:39 Requirements for positional embeddings
04:23 Sines, cosines explained: The original solution from the “Attention is all you need” paper
📺 Transformer explained: • The Transformer neural...
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
NEW (channel update):
🔥 Optionally, pay us a coffee to boost our Coffee Bean production! ☕
Patreon: / aicoffeebreak
Ko-fi: ko-fi.com/aicoffeebreak
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
Paper 📄
Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. "Attention is all you need." In Advances in neural information processing systems, pp. 5998-6008. 2017. proceedings.neurips.cc/paper/...
✍️ Arabic Subtitles by Ali Haidar Ahmad / ali-ahmad-0706a51bb .
Music 🎵 :
Discovery Hit by Kevin MacLeod is licensed under a Creative Commons Attribution 4.0 licence. creativecommons.org/licenses/...
Source: incompetech.com/music/royalty-...
Artist: incompetech.com/
---------------------------
🔗 Links:
AICoffeeBreakQuiz: / aicoffeebreak
Twitter: / aicoffeebreak
Reddit: / aicoffeebreak
KZfaq: / aicoffeebreak
#AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research

Пікірлер: 216

@anonymousanon4822 11 ай бұрын

I found no explanation for this anywhere and when reading the paper missed the detail that each tokens positional encoding consists of multiple values (calculated by different sine functions). Your explanation and visual representation finally made me understand! Fourier transforms are genius and I'm amazed in how many different areas they show up.

@yimingqu2403 3 жыл бұрын

love how the "Attention is all you need" paper appears with an epic-like bgm

@AICoffeeBreak 3 жыл бұрын

It wasn't on purpose, but it is funny -- in hindsight 😅🤣

@adi331 3 жыл бұрын

+1 for more vids on positional encodings.

@sqripter256 8 ай бұрын

This is the most intuitive explanation of the positional encoding I have come across. Everyone out there explain how to do it, even with code, but not the why which is more important. Keep this up. You have earned my subscription.

@AICoffeeBreak 8 ай бұрын

Wow, thanks a lot! :)

@444haluk 3 жыл бұрын

This video is a clear explaination of why you shouldn't add your positional encoding but concat.

@AICoffeeBreak 3 жыл бұрын

Extra dimensions dedicated exclusively to encode position! Sure, but only if you have some extra to share. 😅

@444haluk 3 жыл бұрын

@@AICoffeeBreak this method relocates the embeddings in a specific direction in the embeddings space, so that new position in the relevant embedding cluster have "another" meaning to (say there is another instance of the same word later) other words of "same kind". But that place should be reserved other semantics, else the space is literally filled with "second position" coffee and "tenth position" me, "third position" good etc etc. This can go wrong in soooo many ways. Don't get me wrong, I am a clear cut "Chinese Room Experiment" guy, I don't think you can translate "he is a good doctor" before imagining an iconic low resolution male doctor and recall a memory of satisfaction and admiration of consumatory reward, but again, the "he" in "he did again" and "man, he did it again" should literally have the same representation in the network to start discussing things.

@AICoffeeBreak 3 жыл бұрын

You are entirely right. I was short in my comment because I commented on the same issue in Cristian Garcia's comment. But there is no way you would have seen it, so I will copy paste it here: 😅 "Concatenating has the luxury of extra, exclusive dimensions dedicated to positional encoding with the upside of avoiding mixing up semantic and positional information. The downside is, you can afford those extra dimensions only if you have capacity to spare. So adding the positional embeddings to initial vector representations saves some capacity by using it for both semantic and positional information, but with the danger of mixing these up if there is no careful tuning on this (for tuning, think about the division by 10000 in the sine formula in "attention is all you need")."

@AICoffeeBreak 3 жыл бұрын

And you correctly read between the lines, because this was not explicitly mentioned in the video. In the video I explained what an act of balance it is between semantic and positional information, but you identified the solution: If adding them up causes such trouble, then... let's don't! 😂

@blasttrash Ай бұрын

@@AICoffeeBreak new to AI, but what do you mean by the word "capacity"? Do you mean RAM? Do you mean that if we concat positional encodings to original vector instead of adding, it will take up more RAM/memory and therefore make the training process slow?

@woddenhorse 2 жыл бұрын

Multi Dimensional Spurious Corelation Identifying Beast 🔥🔥 That's what I am calling transformers from now on

@rahulchowdhury3722 Жыл бұрын

You've solid understanding of Mathematics of Signal Processing

@speed-stick Жыл бұрын

Bro Where have you been hiding all this time? This is next level explaining

@deepk889 2 жыл бұрын

I had my morning coffee with this and will make an habit!

@AICoffeeBreak 2 жыл бұрын

What a cool comment! Thanks.

@20Stephanus 2 жыл бұрын

"A multi-dimensional, spurious correlation identifying beast..." ... wow. Douglas Adams would be proud of that.

@AICoffeeBreak 2 жыл бұрын

Hope so. 😁

@full-stackmachinelearning2385 Жыл бұрын

BEST AI channel on KZfaq!!!!!

@hannesstark5024 3 жыл бұрын

+ 1 for video on relative positional representations!

@yyyang_ Жыл бұрын

i've read numerous articles explaining the positional embedding so far.. however, it is surely the greatest & clearest ever

@AICoffeeBreak Жыл бұрын

Thanks a lot! Glad we could help. :)

@sharepix 2 жыл бұрын

Letitia's Explanation Is All You Need!

@AICoffeeBreak 2 жыл бұрын

And comments like these!

@dineshagarwal382 2 жыл бұрын

@kryogenica4759 2 жыл бұрын

Make Ms. Coffee Bean spill the beans on positional embeddings for images

@AICoffeeBreak 2 жыл бұрын

😂

@ausumnviper 3 жыл бұрын

Great explanation !! And Yes Yes Yes.

@garisonhayne668 3 жыл бұрын

Dang it, i learned something and my morning coffee isn't even finished. Its going to be one of *those* days.

@AICoffeeBreak 3 жыл бұрын

Sound like a good day to me! 😅 Whish you a fruitful day!

@jayjiyani6641 Жыл бұрын

Very intuitive. I know there is sine cosine positional encoding but it is actually effective that I got it here..👍👍

@maxvell77 14 күн бұрын

Most insightful explanation I have found on this subject so far. I was looking for it for days... Thank you! Keep going, you rock!

@AICoffeeBreak 14 күн бұрын

Thank you a lot! Also for the super thanks!

@yusufani8 2 жыл бұрын

Probably the clearest explanation for positional encoding:D

@AICoffeeBreak 2 жыл бұрын

Thanks!

@karimedx 3 жыл бұрын

Nice explanation

@AICoffeeBreak 3 жыл бұрын

Thanks for appreciating!

@gemini_537 3 ай бұрын

Gemini: This video is about positional embeddings in transformers. The video starts with an explanation of why positional embeddings are important. Transformers are a type of neural network that has become very popular for machine learning tasks, especially when there is a lot of data to train on. However, transformers do not process information in the order that it is given. This can be a problem for tasks where the order of the data is important, such as language translation. Positional embeddings are a way of adding information about the order of the data to the transformer. The video then goes on to explain how positional embeddings work. Positional embeddings are vectors that are added to the input vectors of the transformer. These vectors encode the position of each element in the sequence. The way that positional embeddings are created is important. The embeddings need to be unique for each position, but they also need to be small enough that they do not overwhelm the signal from the original data. The video concludes by discussing some of the different ways that positional embeddings can be created. The most common way is to use sine and cosine functions. These functions can be used to create embeddings that are both unique and small. The video also mentions that there are other ways to create positional embeddings, and that these methods may be more appropriate for some types of data.░

@huonglarne 2 жыл бұрын

This explanation is incredible

@exoticcoder5365 11 ай бұрын

The best explanation of how exactly position embeddings work !

@ylazerson 2 жыл бұрын

Just watched this again for a refresher; thee best video out there on the subject!

@AICoffeeBreak 2 жыл бұрын

Wow, thanks!

@magnuspierrau2466 3 жыл бұрын

Great explanation of the intuition of positional encodings used in the Transformer!

@DeepakKori-vn8zr Ай бұрын

OMG, such a amazing video to explain Positional Embedding....

@AICoffeeBreak 24 күн бұрын

thank you!

@harshkumaragarwal8326 3 жыл бұрын

great explanation :)

@AICoffeeBreak 3 жыл бұрын

Thanks! 😁

@Phenix66 3 жыл бұрын

Great stuff :) Would love to see more of that, especially for images or geometry!

@AICoffeeBreak 3 жыл бұрын

Noted!

@helenacots1221 Жыл бұрын

amazing explanation!!! I have been looking for a clear explanation on how the positional encodings actually work and this really helped! thank you :)

@SyntharaPrime Жыл бұрын

Great explanation - It might be the best. I think I finally figured it out. I highly appreciate it.

@elinetshaaf75 2 жыл бұрын

great explanation of positional embeddings. Just what I need.

@kevon217 9 ай бұрын

Super intuitive explanation, nice!

@khursani8 3 жыл бұрын

Thanks for the explanation Interested to know about rotary position embedding

@tanmaybhayani Ай бұрын

Amazing! This is the best explanation for positional encodings period. Subscribed!!

@AICoffeeBreak Ай бұрын

Thank you!

@MaximoFernandezNunez Жыл бұрын

I finally understand the positinal encoding! Thanks

@AICoffeeBreak Жыл бұрын

Happy to help. :)

@gauravchattree5273 Жыл бұрын

Amazing content. After seeing this all the articles and research papers makes sense.

@raoufkeskes7965 5 ай бұрын

the most brilliant positional encoding explanation EVER that was GOD Level explanation

@AICoffeeBreak 5 ай бұрын

Haha, thanks.

@nicohambauer 3 жыл бұрын

Sooo good!

@mbrochh82 Жыл бұрын

This is probably the best explanation of this topic on KZfaq! Great work!

@AICoffeeBreak Жыл бұрын

Thanks so much! 😊

@WhatsAI 3 жыл бұрын

Super clear and amazing (as always) explanation of sines and cosines positional embeddings! 🙌

@AICoffeeBreak 3 жыл бұрын

Thanks! Always happy when you visit!

@ConsistentAsh 3 жыл бұрын

I was browsing through some channels after first stopping on Sean Cannells and I noticed your channel. You got a great little channel building up here. I decided to drop by and show some support. Keep up the great content and I hope you keep posting :)

@AICoffeeBreak 3 жыл бұрын

Thanks for passing by and for the comment! I appreciate it!

@tonoid117 3 жыл бұрын

What a great video, I'm studying my Ph.D. at NLU, so this came in very handy. Thank you very much and greetings from Ensenada Baja California Mexico :D!

@AICoffeeBreak 3 жыл бұрын

Thanks, thanks for visiting from so far away! Greetings from Heidelberg Germany! 👋

@timoose3960 2 жыл бұрын

This was so insightful!

@matt96920 Жыл бұрын

Excellent! Great work!

@aasthashukla7423 8 ай бұрын

Thanks Letitia, great explanation

@clementmichaud724 Жыл бұрын

Very well explained! Thank you so much!

@EpicGamer-ux1tu 2 жыл бұрын

Great video, many thanks!

@Galinator9000 2 жыл бұрын

These videos are priceless, thank you!

@jayk253 Жыл бұрын

Amazing explanation! Thank you so much !

@user-fg4pr4ct6g 10 ай бұрын

Thanks, your videos helped the most

@ashish_sinhrajput5173 10 ай бұрын

i watched bunch of videos on the positional embedding , but this video makes me very clear intuition behind the positional embedding , thank you very much for this great video , 😊

@AICoffeeBreak 10 ай бұрын

Thanks, that's great to hear! ☺️

@user-gk3ue1he4d 10 ай бұрын

Great work! Clear and deep explanation!

@jayktharwani9822 Жыл бұрын

great explanation. really loved it. Thank you

@jfliu730 Жыл бұрын

best video about position emb i have ever heard

@AICoffeeBreak Жыл бұрын

Thanks!

@oleschmitter55 8 ай бұрын

So helpful! Thank you a lot!

@andyandurkar7814 Жыл бұрын

Just an amazing explanation ...

@deepshiftlabs 2 жыл бұрын

Brilliant video. This was the best explanation of positional encodings I have seen. It helped a TON!!!

@deepshiftlabs 2 жыл бұрын

I also make AI videos. I am more into the image side(convolutions and pooling) so it was great to see more AI educators.

@aloksharma4611 Жыл бұрын

Excellent explanation. Will certainly like to learn about other encodings in areas like image processing.

@hedgehog1962 Жыл бұрын

Really Thank you! Your video is just amazing!

@ai_station_fa 2 жыл бұрын

Awesome. Thank you for making this great explanation. I highly appreciate it.

@shamimibneshahid706 2 жыл бұрын

I feel lucky to have found your channel. Simply amazing ❤️

@AICoffeeBreak 2 жыл бұрын

Happy to have you here!

@erikgoldman Жыл бұрын

this helped me so much!! thank you!!!

@roberto2912 4 ай бұрын

I loved your simple and explicit explanation. You've earned a sub and like!

@AICoffeeBreak 4 ай бұрын

Thanks, see you again soon!

@zhangkin7896 2 жыл бұрын

Really great!

@ColorfullHD 3 ай бұрын

Lifesaver! Thank you for the explanation.

@xv0047 Жыл бұрын

Good explanation.

@richbowering3350 Жыл бұрын

Best explanation I've seen - good work!

@AICoffeeBreak Жыл бұрын

Thank you so much! 🫨

@javiervargas6323 2 жыл бұрын

Thank you. One thing is to know the formula and applying it and other thing is to understand the intuition behind it. You made it very clear. All the best

@AICoffeeBreak 2 жыл бұрын

Well said! -- Humbled to realize this was put in context with our video, thanks. Thanks for watching!

@aterribleyoutuber9039 6 ай бұрын

This was very intuitive, thank you very much! Needed this, please keep making videos

@AICoffeeBreak 6 ай бұрын

@machinelearning5964 Жыл бұрын

Cool explanation

@user-ru4nb8tk6f 10 ай бұрын

so helpful, appreciate it!

@adeepak7 4 ай бұрын

Very good explanation!! Thanks for this 🙏🙏

@AICoffeeBreak 4 ай бұрын

Thank You for your wonderful message!

@ugurkap 3 жыл бұрын

Explained really well, thank you 😊

@AICoffeeBreak 3 жыл бұрын

Thanks, so nice of you!

@avneetchugh Жыл бұрын

Awesome, thanks!

@ravindrasharma85 Ай бұрын

excellent explanation!

@AICoffeeBreak Ай бұрын

@arishali9248 Жыл бұрын

Beautiful explanation

@AICoffeeBreak Жыл бұрын

Glad you think so! :)

@yonahcitron226 Жыл бұрын

amazing stuff! so clear and intuitive, exactly what I was looking for :)

@AICoffeeBreak Жыл бұрын

Thanks for watching and appreciating! 😊

@preadaptation Жыл бұрын

Thanks

@DerPylz 3 жыл бұрын

Thanks, as always, for the great explanation!

@AICoffeeBreak 3 жыл бұрын

It was Ms. Coffee Bean's pleasure! 😅

@amirhosseinramazani757 2 жыл бұрын

Your explanation was great! I got everything I wanted to know about positional embedding. thank you:)

@AICoffeeBreak 2 жыл бұрын

Awesome, thanks for the visit! ☺️

@nitinkumarmittal4369 5 ай бұрын

Loved your explanation, thank you for this video!

@AICoffeeBreak 5 ай бұрын

Glad you like it!

@camotsuchuoi Жыл бұрын

Thank you

@klammer75 Жыл бұрын

This is an amazing explanation! Tku!!!🤓🥳🤩

@justinwhite2725 2 жыл бұрын

In another video I've seen, apparently it doesn't matter if positional embedding are learned or static. It seems as thiugh the rest of the model makes accurate deductions regardless. This is why I was not surprised that Fourier transforms seem to work nearly as well as self attention.

@meechos 2 жыл бұрын

COuld you please elaborate using an example maybe?

@bartlomiejkubica1781 5 ай бұрын

Great! It took me forever, before I had found your videos, but finally I understand it. Thank you soooo much!

@AICoffeeBreak 5 ай бұрын

You're very welcome!

@markryan2475 3 жыл бұрын

Great explanation - thanks very much for sharing this.

@AICoffeeBreak 3 жыл бұрын

Glad you enjoyed it!

@montgomerygole6703 Жыл бұрын

Wow, thanks so much! This is so well explained!!

@AICoffeeBreak Жыл бұрын

Glad it helped. :)

@bdennyw1 3 жыл бұрын

Nice explanation! I’d love to hear more about multidimensional and learned position encodings

@AICoffeeBreak 3 жыл бұрын

Noted!

@omniscienceisdead8837 2 жыл бұрын

you are a genius!!

@antoniomajdandzic8462 2 жыл бұрын

love your explanations !!!

@AICoffeeBreak 2 жыл бұрын

🤗

@Nuwiz 11 ай бұрын

Nice explanation!

@AICoffeeBreak 11 ай бұрын

Glad it was helpful!

@anirudhthatipelli8765 Жыл бұрын

Thanks, this was so clear! Finally understood position embeddings!

@AICoffeeBreak Жыл бұрын

Fantastic!

@alighandour1057 6 ай бұрын

life saver!

@PenguinMaths 3 жыл бұрын

This is a great video! Just found your channel and glad I did, instantly subscribed :)

@AICoffeeBreak 3 жыл бұрын

Thank you so much, you honour us!

@robert75019 11 күн бұрын

Very clear 👏👏

@AICoffeeBreak 11 күн бұрын

Thank you!

@johannreiter1087 10 ай бұрын

Awesome video, thanks :)

@saurabhramteke8511 Жыл бұрын

Hey, Great Explanation :). Love to see more videos.

@AICoffeeBreak Жыл бұрын

Glad you found us! 😁

@ylazerson 2 жыл бұрын

amazing video - rockin!

@AICoffeeBreak 2 жыл бұрын

🤘

@GaryGoldstein1234 Жыл бұрын

Yes.