Thanks! I’m trying to make shorter videos and learning that it can actually be more challenging than making a longer one

I'd say Quality > Quantity. Time is valuable and that is probably one reason why shorter from videos are becoming so successful. Also, I'd also say making shorter educational videos forces you to cut away everything that is not important, which should leave you with a clearer picture of what the essence of that concept is.

reality , make a base model .. highly tuned and use this as your starting point for new models ... preserve you base at all costs... often online versions are poluted ...

Hey I really really appreciate the kind words and would absolutely love more comments and feedback in the future

Thanks!

Well yes, but actually no... Practically in the common case - you are right and this is how it is done in word2vec and other models as well - we build each word's embedding by its context. But if we take the message of the video and apply it here, you may also decide that this is how you want to define our word embeddings... I'm not sure what the benefits of this would be (it might be somehow reminiscent of LDA, for example) -- but we have the freedom to decide how we build our embeddings.

@@shachafporan8048 I agree that you can have different algorithmic flavors how you derive embeddings, just as you can have different corpus you want to specialize the embeddings for (as pointed by the video). Words can have different meanings/embeddings that are useful for different purposes. It's the old no-free-lunch theory. However, in all the algorithms words end up with similar embeddings not because they co-occur, but because of the same context they appear in. Words "Monday" and "Tuesday" rarely co-occur, but end up having similar embeddings. This is true even for algorithms such as Glove, which is based on co-occurrence, but derives similar meaning/embedding trough the network effect. It's not only word2vec, most tranformers also use the same idea with masked word predictions. I love the video. It's just that co-occurrence is not in the heart of the embeddings. And it's hard later to understand other nice properties of embeddings if you look at it that way.

@@chupacadabra nah man... I mean, we're not disagreeing... But my feeling was that his example of embeddings in a language model is just a specific manifestation of *embeddings*... If you embed an image how would you relate it to a context? Also, if you embed words e.g. in an explicit contrastive learning problem, you can have plenty of success without context. So again... In general for language you are correct, but the world is too diverse to think of embeddings only in this specific manner

Noted! Thanks for the feedback

Glad it helped!

The deep learning models are trained using non-linearities to capture non-linear relationships in the data. Hence, the function (=model architecture) you use to learn the embeddings has non-linearities. When we train a deep learning model to obtain an embedding, we most of the time have an embedding layer as the first layer in our model. We then train the model using a specific objective (goal), which is suitable to obtain word embeddings. After having trained the model, we just take the embedding layer out of the full model and discard the rest. You can imagine the embedding layer as a matrix of size (vocab_size x embedding_dimension). That means each word/token in our vocabulary is represented by a vector with as many numbers as the embedding dimension. The matrix (embedding layer) itself has no non-linearities, it's just a matrix. Therefore, the vectors that represent the tokens can be compared with each other using linear metrics as you said above. Hope it helps :)

@@SierraSombrero Appreciate the response. But I think there's some critical issue lingering. 1. The input is a matrix. It goes through linear -> non-linear -> linear transformations. The back-propagation has to go through the same steps when updating the embedding layer's weight. So it's carrying non-linear information over to the embedding layer, thus breaking the linear properties, right? 2. By "The matrix (embedding layer) itself has no non-linearities", does that mean I can extract any weights before the activation unit in a neuron and use it as embedding?

​@@zeroheisenburg3480 I'll try to answer as best as I can. I'm not sure I'll be able to answer question 1 satisfactorily, though :) I guess I'll start with question because I can explain it better. 2. An embedding layer is not the same as a linear layer. It does not represent neurons and does not output activations (but rather representations). In a linear layer you have an input x that you multiply with the weight w and then you add a bias b. (I don't know of any case where weights have been used as embeddings.) An embedding layer can usually only be the first layer in a network. You don't multiply an input x with a weight w here. Instead you have a number of input classes in the form of integers (that represent e.g. words) that you can feed your model (the number of integers is your vocab size). Each of these input integers is mapped to one row in your embedding layer (vocab_size x embed_dim). You can imagine it like table where you look up which embedding belongs to which word. Once you have looked up the embedding for your current word, you use it as input to the next layer in your model. Now, before having trained your model the embedding is random and the embedding layer is updated during training using backprob just like every other layer (though differently because it is a different mathematical operation than a linear layer). After training the model, the embedding layer has been changed so that every of your inputs words now has a meaningful representation in the embedding space (if your training was successful). Now you can take the lookup table (embedding layer) out of your model, feed it a word and it will give you the meaningful embedding belonging to your word. I suggest you to check out the difference between Linear and Embedding layer in pytorch :) Make sure to understand what kinds of inputs you feed it and what you get as outputs. pytorch.org/docs/stable/generated/torch.nn.Linear.html pytorch.org/docs/stable/generated/torch.nn.Embedding.html Maybe also try to find a good explanation of how the first static embeddings were trained (CBOW, Skipgram). I think this should give you the intuition. 1. It's true that during training backpropagation also non-linear operations take place. However, since you're discarding all non-linear parts of the model and only keep the embedding layer, it is definitely possible in practice to apply linear operations on them. If there are theoretical mathematical issues lingering in the background then I'm certainly the wrong person to answer your question. But since it works so well in practice I would personally not worry too much about it :)

Thanks!

Nothing. At least in the context of a RAG system where you use the embeddings to search through their vector database to retrieve context for the LLM.

We’re getting there 😂

Thanks!

Agreed!