Swin Transformer paper animated and explained

Рет қаралды 63,607

Күн бұрын

Swin Transformer paper explained, visualized, and animated by Ms. Coffee Bean. Find out what the Swin Transformer proposes to do better than the ViT vision transformer.
📺 ViT explained: • An image is worth 16x1...
📺 Transformer explained: • The Transformer neural...
📺► Positional embeddings (playlist): • Positional encodings i...
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
Thanks to our Patrons who support us in Tier 2, 3, 4: 🙏
donor, Dres. Trost GbR, Yannik Schneider
➡️ AI Coffee Break Merch! 🛍️ aicoffeebreak.creator-spring....
🔥 Optionally, pay us a coffee to help with our Coffee Bean production! ☕
Patreon: / aicoffeebreak
Ko-fi: ko-fi.com/aicoffeebreak
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
Paper discussed:
📜 Liu, Ze, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. "Swin transformer: Hierarchical vision transformer using shifted windows." arXiv preprint arXiv:2103.14030 (2021). arxiv.org/abs/2103.14030
💻 Swin Transformer code on GitHub: github.com/microsoft/Swin-Tra...
Outline:
00:00 Problems with ViT / Swin Motivation
04:16 Swin Transformer explained
06:00 Shifted Window based Self-attention
08:58 positional embeddings in the Swin Transformer
09:29 Task performance of the Swin Transformer
Music 🎵 : Bay Street Millionaires by Squadda B
---------------------
🔗 Links:
AICoffeeBreakQuiz: / aicoffeebreak
Twitter: / aicoffeebreak
Reddit: / aicoffeebreak
KZfaq: / aicoffeebreak
#AICoffeeBreak #MsCoffeeBean #MachineLearning #AI #research
Video and thumbnail contain emojis designed by OpenMoji - the open-source emoji and icon project. License: CC BY-SA 4.0 16x16 pixels comprehensible artificial intelligence

Пікірлер: 99

@SomexGupta Жыл бұрын

Awesome Video, explained concept in very easy to understand, Small query at time 2:21 when we divided 256*256 pixels in 16*16 pixels then total number of token should be 256 according to me, as (256*256)/(16*16) = 256 tokens but in explanation it's mentioned 16 tokens can you guide on this.

@AICoffeeBreak 11 ай бұрын

Hi, you are right, my mistake. Pinned your comment, thanks!

@CristianGarcia 2 жыл бұрын

Alternative title for the paper: Convolutional Transformer.

@astroferreira 2 жыл бұрын

Great video! I think the passage in the abstract is related to the fact that text has a fixed scale compared to images. The smallest piece of text you can have is a single character while for images, a single pixel can represent wildly different scales and can't really be considered the 'smallest scale possible'. In microscopy a single pixel can have scales of 1e-4 m while for astronomy a single pixel can represent kiloparsecs or ~1e19 m.

@minhquanao7492 2 жыл бұрын

I think the idea of applying Transformer over a small window also appears in "Deformable DETR: Deformable transformers for end-to-end object detection". However, like Deformable Convolution, this paper lets the model learn the location where each patch pays attention to rather than fix the attention window (e.g. the immediate 3x3 neighborhood).

@visintel Жыл бұрын

I love the low-key comparison to simple convolution. Looks like we made a full circle lol.

@SuperShadowmasterZ 2 жыл бұрын

I saw a similar tranformer useage in Fastformer: Additive Attention Can Be All You Need

@AnilKeshwani 2 жыл бұрын

My gosh these video explainers are good. Fantastically clear and intuitively presented

@AICoffeeBreak 2 жыл бұрын

Glad you think so! :)

@tane_ma 2 жыл бұрын

I am a new fan of the channel. Always good and quick explanations and logic/storytelling, animations, segmented sections, length of videos, and link for the paper and repo in the description ❤

@AICoffeeBreak 2 жыл бұрын

Hey, thanks for the kind words! Happy to have you here.

@alouped 2 жыл бұрын

Nice videos, thanks for putting in the work.

@erdemakagunduz2078 2 жыл бұрын

great video. But if we must compare a Fyodor Dostoevsky novel to something in vision, it is not an entire single image, it is a Andrei Tarkovsky movie. So moral of the story, vision still rocks! :)

@anirudhthatipelli8765 Жыл бұрын

Thanks a lot! This was very detailed!

@ThamizhanDaa1 2 жыл бұрын

I think SWIN Transformer perforcmance should be compared with other convNets for semantic segmentation, including DeiT regular size.. you're right its pretty deceptive to ignore those results haha. But then again, this is a good idea for self-attention, regardless of this

@littlevu735 2 жыл бұрын

Great channel, keep going!

@nilsmuller9286 2 жыл бұрын

Great content as always. :)

@AICoffeeBreak 2 жыл бұрын

Glad you think so! Thanks for watching.

@RAZZKIRAN Жыл бұрын

Great channel , thank u

@veggeata1201 Жыл бұрын

I'm not sure of the origin of windowed attention, but it is used in big bird along with other sparse attention methods.

@madhavjariwala4548 2 жыл бұрын

Thank you for this video. You're the best!

@AICoffeeBreak 2 жыл бұрын

Glad it was helpful!

@keroldjoumessi 2 жыл бұрын

Very nice video. I have really enjoyed it as it was quite easy to follow with no prior knowledge. Therefore I don't quite understand why we still need to transform the patch vectors (features dimensionality) into another dimensionality C. In other words, what is the idea behind this transformation (from the initial features dimensionality to another C-feature dimensionality)?

@philip2.042 Жыл бұрын

Because we re merging multiple vectors from self attention layer into one, we enlarge our representarion vector (C) under a hypothesis that it will better capture more info coming from larger patches

@Peebuttnutter 2 жыл бұрын

thanks!!

@soumyasarkar4100 2 жыл бұрын

isn't shifted window based self attention similar to local attention in longformer ?

@AICoffeeBreak 2 жыл бұрын

🤫 you're diminishing the novelty.

@LKRaider 2 жыл бұрын

@@AICoffeeBreak LOL

@Harry-jx2di Жыл бұрын

Thanks!

@shubhamsuryavanshi1461 Жыл бұрын

Great work 😃, could you please make a video on deformable transformers for end to end object detection? ☺️

@paoloceric6464 2 жыл бұрын

Nice video, but I think you made a mistake when calculating the number of patches (both in the 256x256 and 1920x1920 example). A 16x16 patch would produce 256 patches in the first image, and 14400 in the second, not 16 and 120.

@AICoffeeBreak 2 жыл бұрын

It's totally possible I made a mistake, but for the moment, I do not get it. We said that a 256^2 pixel image would need 16 of those 16^2 patches. A 1920^2 pixel image would need 14400 of those 16^2 patches. How do you calculate this?

@spongemeryl 2 жыл бұрын

Same comment/doubt here, maybe I didn't quite get it right, but isn't 256^2/16^2 = 256, and 1920^2/16^2 = 14400?

@paoloceric6464 2 жыл бұрын

@@AICoffeeBreak Okay, then it seems I don't get what a patch/image vector actually is. You said "if the image is 256x256 pixels then extracting 16x16 patches would lead to 16 patches", but why only 16? If we divide a 256x256 image into 16x16 squares, we get 256 squares, that's my only point. If we indeed only use 16 of those 256 squares then my question is - why?

@AICoffeeBreak 2 жыл бұрын

@@paoloceric6464 An image of 256x256 pixels has 256 pixels in width, 256 pixels in height. A 16x16 patch, is a pixel tile of 16 pixels in width and 16 pixels in height. How many of these patches do you need to achieve a complete tiling of the image? 16. Because 256/16=16. So We need 16 patches to tile the image with them.

@patakk8145 2 жыл бұрын

@@AICoffeeBreak are you sure? You can’t just divide 256 by 16, that only gives you the amount of patches in one dimensions (e.g. width). In order to fill the whole area you need 256 patches. Or you can think of it as 256x256=65536 total pixels that you’re filling with 16x16=256 pixel patches. There’s obviously 256 of them in the whole image.

@lucasbeyer2985 2 жыл бұрын

That paper where you've seen this before is either HaloNet or SaSaNet (standalone self-attention)

@amreamer362 2 жыл бұрын

Very awesome

@debanjanchakraborty9946 Жыл бұрын

really love your content and i actually shifted algoritms coz they dont run on my system and i wanted more accurate results

@sachinlodhi8542 Жыл бұрын

at 2:23 how 16x16 pixel patches generated from 256x256 image would sum up to 16? Would there not be total of 256 patches of 16x16 ?

@AICoffeeBreak 11 ай бұрын

Hi, you are right, my mistake. I've pinned a comment explaining this, thanks!

@reasoning9273 Жыл бұрын

I think you forgot to square 120. 1920x1920 resolution will generate 14.4k image tokens of size 16x16, which is 3164 times more computation compared to 256x256 case when calculating dot product attention. I don't think any single GPU can manage this calcuation.

@Jack-gb1nw 2 жыл бұрын

was it potentially the Longformer or the Reformer NLP papers that reminded you of localised attention?

@asn9329 2 жыл бұрын

can this transformer be used for super-resolution task, for unpaired data.

@toyuyn 2 жыл бұрын

Isn't that just local attention? "Yeah but you can achieve global attention at later layers because of the receptive fields" Isn't that what CNN's do? Then why bother with transformers? "..." Something something attention, something something dynamic convolutions.

@elinetshaaf75 2 жыл бұрын

Whatever buzzword makes a publication these days.

@AICoffeeBreak 2 жыл бұрын

It seems that a lot of research nowadays is to introduce some of the inductive biases of the CNN into the transformer. What is better than a complete related work? An incomplete one and a paper that claims to be the first to have invented the wheel. :)

@VVi11 2 жыл бұрын

pretty much

@cipritom 2 жыл бұрын

My thoughts exactly. So the gains must come from somewhere else (over the ConvNets). And indeed, a few months later, we have ConvNeXt showing the gains do indeed come from other parts, not from attention

@kristoferkrus 10 ай бұрын

Great video! And I know you published it close to two years ago, but about the window-limited self attention, I guess that's pretty standard in generative LLMs nowadays, such as Llama or the GPT family by OpenAI?

@kristoferkrus 10 ай бұрын

But maybe I'm diminishing the novelty now 😁

@AICoffeeBreak 8 ай бұрын

Yes, it is the case for Long Context Transformers. But the problem there is the network forgets at the end what was said in the beginning. So the paper on attention sinks is a simple hacky solution to that.

@kristoferkrus 8 ай бұрын

@@AICoffeeBreak Thanks; I will check it out!

@DerPylz 2 жыл бұрын

Shifted WINDOWS transformer by Microsoft research 🤔🤔🤔🤔

@AICoffeeBreak 2 жыл бұрын

It's because they could not shift the Linux, lol.

@DerPylz 2 жыл бұрын

@@AICoffeeBreak I prefer shifted Apple transformers. Even though they are often confused with pizza...

@giantbee9763 2 жыл бұрын

Very nice video :) !

@AICoffeeBreak 2 жыл бұрын

Thanks for the visit! I saw you commented something on the "Generalization - Interpolation - Extrapolation video" but the comment is no longer available. Either: 1. you removed it 2. YT removed it (did you have a link in there?) But I did not remove it. I am actually quite curious to know what you had to say there. :) I am mentioning this because I had previous experience of good comments being removed by YT without any of my doing and people were a little perplexed and confused why I am censoring them. 🤐

@giantbee9763 2 жыл бұрын

@@AICoffeeBreak Hi Letita, Yup I did comment on the video but I ended up removing it, so it wasn't the KZfaq algorithm this time. :D That's all :D I had been living under the rock of "not using twitter", so I'm probably quite late to the party anyway.

@AICoffeeBreak 2 жыл бұрын

Haha, great to hear then that YT is not messing with comments this time. :) Still curious what you had to say. I guess it will stay forever a mystery. 🤫

@Jose-pq4ow 2 жыл бұрын

The tricks needed to efficiently run these models on computer vision tasks seem to be too "complex" in comparison to standard CNNs....

@AICoffeeBreak 2 жыл бұрын

Yeah, it looks quite messy at the moment. On the other side, tricks to get CNNs to work were complex at their time too (pooling, dropout, fully convolutional architectures, batch norm, etc.). It's just that we got used to it (and educated about it). After the proliferation of tricks to make the transformer more data-efficient and get it to work on long sequences, there will be half a dozen of tricks that will stick with them and will be taught to posterity as actually quite simple tricks. It looks like quite a mess because we are not there yet.

@syedadzha362 Жыл бұрын

Amazing video

@AICoffeeBreak Жыл бұрын

Thanks!

@undefined-mj6oi 2 жыл бұрын

2:28 Could you please explain why 16*16 patches lead to 16 image tokens here?

@AICoffeeBreak 2 жыл бұрын

😅 No, I can't because it leads to 256 image tokens. See the whole comment and thread by @Paolo Čerić in here where he was the first to make me realize this mistake.

@undefined-mj6oi 2 жыл бұрын

@@AICoffeeBreak Got it! Thanks!

@fast_harmonic_psychedelic 2 жыл бұрын

is it able to encode text or is the image projection able to be compared via cosine similarity like clip? can this replace clip? Let me know in the comments below

@AICoffeeBreak 2 жыл бұрын

It's s transformer, so sure you can have the two branches in CLIP to be replaced by two Swim Transformers.

@fast_harmonic_psychedelic 2 жыл бұрын

@@AICoffeeBreak i tried it but i cant figure it out. so many outputs to swin that are different shapes that are incompatible. I tried to have it encode separately side by side with clip and then maybe get a mean of both encodings but theres just too many errors and parameters to change i ended up giving up. what i dont understand is -- whats the point of this without some sort of text module? Like.. what does it do.. lol .. it just takes the image and outputs the same image?

@fast_harmonic_psychedelic 2 жыл бұрын

Like - i can understand if this was replacing clips VIT it would be magical to get attention on all these different scales. But alone, with no understanding of token embedding similarity to image patches -- is it just good for benchmarking or what lol

@AICoffeeBreak 2 жыл бұрын

@@fast_harmonic_psychedelic I get your problem. So, no. This image-only transformer (in its current form) basically autoencodes the image, yes. But there are special [CLS] tokens to solve tasks like image recognition.

@fast_harmonic_psychedelic 2 жыл бұрын

@@AICoffeeBreak is there some sort of map of CLS tokens that someone could refer to in order to activate certain features?

@ishaqkhan5418 8 ай бұрын

Its really great video, but maybe you had to explain the architectures in a little more details like 3 4 minutes more would have made it the best. Anyways thank you for the great content!

@AICoffeeBreak 8 ай бұрын

Thanks for your feedback! :) Appreciate it!

@mrigankanath7337 11 ай бұрын

if image size is 256 x 256 and patch size is 16 x 16 shouldnt there be 256 tokens? ((256 x 256)/ (16 x16)) = 256

@AICoffeeBreak 11 ай бұрын

Hi, you are right, my mistake. I've pinned a comment explaining this, thanks!

@chez8990 9 ай бұрын

Longforner restricts attention window to expand token limiit

@AICoffeeBreak 9 ай бұрын

Thanks for this, Longformer is a great reference. Even before Swinformer, there were papers restricting the attention window. This idea has now become even more represented.

@sachinlodhi8542 Жыл бұрын

at 3:30 how 256x256 pixels result in 63504 ?

@AICoffeeBreak 11 ай бұрын

Hi, you are right, my mistake. I've pinned a comment explaining this, thanks!

@kaustavdas6550 3 ай бұрын

Casa? Cascading Self attention seems similar?

@subhanshubansal4704 10 ай бұрын

Local Attention ? (Shifted Windows)

@yimingqu2403 2 жыл бұрын

ICCV 2021 best paper

@AICoffeeBreak 2 жыл бұрын

Really? You're attending?

@yimingqu2403 2 жыл бұрын

@@AICoffeeBreak not me, but my colleagues at MSR

@AICoffeeBreak 2 жыл бұрын

@曲一鸣 Well then, congrats to the authors! 👏

@yusufani8 Жыл бұрын

I am putting here a counter for how many times I forget what does Swin Transformer. Counter = 1

@AICoffeeBreak Жыл бұрын

😅

@lucasbeyer2985 2 жыл бұрын

Haha no need to be triggered. By "scale of visual entities" they mean "size of things in the picture", so that sometimes an orange covers just 10 pixels and sometimes it covers 1000 pixels. This effect indeed does not really exist in language.

@shinkai791 11 ай бұрын

A little bit like "local attention Transformer"?

@AICoffeeBreak 9 ай бұрын

Bingo.

@erengurses123 Жыл бұрын

1920x1920 image have 120 image tokens where patch size is 16x16 ???? At least 120 should be the square of something.

@gauravlochab9614 2 жыл бұрын

using detr for face recognition

@CyrusVatankhah Жыл бұрын

Can you get rid of the coffee bean? Or if it is your "brand", at least don't change/move it throughout the video. It is super distracting!

@AICoffeeBreak Жыл бұрын

Thanks for sharing your feedback. We had this discussion in a video before, so I did a poll on this: kzfaq.infoUgkxU0F0Y69SrC6HhZ6uD97gVxrANlH1CElk I personally am quite attached to her.