@deeplearninghero and you mention, that positional embedding is applied to k and q vectors. Is it new with RoPE? Because I thought, that in transformer architecture positional embedding is added to token embedding (which we get from tokenizer). And this summed vector goes to encoder/decoder, where it splitted to k, q and v. And inside encoder/decoder we are not applying any positional encoding.

Yes it's a major change from sinusoidal embeddings to RoPE. As per RoPE's motivation you need the positional distinction between q and k so adding them there is ideal. :)

Hi, I'm speaking as a bit of a newbie to this concept, but I was also curious about your question. From what I found, in the transformer's architecture the meaning of a word, like "dog", within a given context is influenced by both its semantic embedding and its position in the sequence. The model is forced to learn these two pieces of information in conjunction, meaning it will see "dog" in many different positions and many different contexts. The other big thing is that transformers are not isolated word processors; the model processes the entire sequence when generating text, so even though the vector for "dog" is rotated, it's interpreted in the context of the surrounding words and their respective positional encodings. This is combined with the benefits of high-dimensionality. As you add more and more dimensions, it becomes increasingly less likely that the word "dog" could get rotated at any position to match any other word. Since the model processes sequences in parallel, it almost always will have contextual information such as "walk" or "leash", etc. that teaches the model the original semantic meaning during training regardless of how it is rotated.

very intuitive@@yourmomsboyfriend3337

@@yourmomsboyfriend3337 hey, took me a while to answer as I had to mull it over, still ongoing. Thanks for your answer. I suggest to change our PoV a little. Instead of seeing the embedding as a whole, that we look at the individual dimensions. Each dimension is rotated differently. So it would only be a few dimensions at a time, and even fewer important ones, that would be totally distorted by an 180 degrees rotation. So most of the dimensions would still be recognizable? I am still not really sold though.

Hi, thanks for the question. It's a good question, not a naive question :) It's a bit difficult to know exactly what OpenAI is doing due to their secretive way of operating. But I can make an educated guess. There's three popular models; GPT3/3.5/4. GPT3 (arxiv.org/abs/2005.14165) - This came before the rotary embeddings paper, so I'm assuming this uses standard sinusoidal embeddings. GPT3.5 - Don't think there's a paper for this, so again I'm not sure if they use RoPE. But there's a good chance they are, as the timing is right. GPT4 (arxiv.org/pdf/2303.08774.pdf) - They do have a reference for RoPE in their paper, so quite possibly GPT4 is using RoPE But keep in mind these are speculative guesses and unfortunately there's no way to say which type of embedding by looking at the embeddings themselves.

@@deeplearninghero that is absolutely fascinating! Thank you for such an insightful response. The two types of embeddings cannot be distinguished from another if given a set of similar text inputs? I'm not saying I'm savvy enough to figure out how to accomplish something like that, but I suppose that surprises me. If the visual analogy of the clock holds, would similar inputs not read as somewhat similar "times"? I know that's underselling of the complexity of working with high-dimensional embeddings. I guess the rotational nature of capturing information in this way has sparked my imagination as someone with a background in genetics.

> If the visual analogy of the clock holds, would similar inputs not read as somewhat similar "times"? That's an interesting analogy. Thanks for sharing. But IMO, numerically it wouldn't hold. I guess your suggesting that we can perhaps "count" how many time a vector passes through a certain point? The problem would be two-fold. 1) You wouldn't get exact overlaps because these vectors are high dimensional and they may not be rotating in properly divisible pieces of the 360 degree circle. 2) Sinusoidal embeddings does something similar. So if you do this counting on approximate basis, you'd still see sinusoidals passing through this point. So would be difficult. And finally, being high dimensional, it's very hard to reason about (unfortunately).

Kind of dingboard?