Thank you so much! I’m learning things as I go, so I really appreciate feedback like this!

​@@avb_fj I agree with him. Your pacing is excellent and you're giving a perfect level of detail.

Nice! Glad you enjoyed it!

Awesome! Welcome to the channel and I’m glad you liked the video!

Very true. Vision Transformers are definitely here to stay. The generalization power of transformers/attention is so surprising sometimes… decades of computer vision research suggested that CNNs are best for images because they can encode spatial information about the image… it’s just counterintuitive and mind boggling that ViTs can still learn from images by flattening individual patches and lose spatial structure.

Someone else had the same question, copy-pasting that reply here… So the output token is kind of a common trick that people use in Transformer based models to "aggregate information" about an input sequence. If you are familiar with the Next Sentence Prediction task in BERT models, they also use a similar concept with the [CLS] token. Basically, concept goes as follows: step 1> the output token is a dummy token you append onto the input sequence (say at the very end of the input seq) step 2> pass it through the transformer/attention layers step 3> the attention layers generates a sequence of contextual embeddings, one for each token in the input sequence step 4> you extract the embedding in the index corresponding to the dummy output token (i.e. the last embedding coz that's where you put the output token in the input sequence in step 1) step 5> this embedding now encapsulates or aggregates the entire context of the input sequence and can be used for downstream tasks like classification, etc. Hope that helped. The literature may be a bit thin on output token embeddings in the Segmentation space, but I'll strongly recommend to read about the [CLS] token in the BERT paper for Next Sentence Prediction to get a better understanding.

That's a great question. We should definitely calculate the loss between step 3 and step 4. Every iteration is regarded as an isolated training example, so basically for each training example we input an image and a prompt (with a dense mask) and outputs predicted mask(s)... and then apply the loss over our prediction and the ground truth. As far as "updating the model" is concerned, it is largely a design choice I think. It's not incorrect to update the weights between each iteration, but it's probably better to do gradient accumulation (basically aggregating the losses over multiple iterations) before updating the weights to get a more stable training curve. Hope that helps!

@@avb_fjThanks a lot! Really great explanation!

🙌🙌@@SofieSimp

Good old iPhone. Good luck on your thesis man!

Thanks for the comment! That’s great feedback, I’ll try to share more in the upcoming videos!

Could you add a timestamp?

2:54

@@miyutube1 I see. “p” here is the simply the probability outputted by the model for the classification task. You can find more info in Page 3 of this paper arxiv.org/pdf/1708.02002.pdf

Check out the part about interactive training at around 5:00 Basically the dense mask prompt is used during the training phase to iteratively improve the networks segmentations. Kinda like asking the model, “Hey you gave me this mask last time, but here’s another internal/external point prompt, give me an updated mask”. During inference, we can pass an image full of zeros as the dense mask into the model (meaning we have no idea where the segmentation should be) and ask the model to update it. Once it gives an initial estimation, we might recursively pass the network’s output logits (not binary, but the prob distribution) back into it as a dense prompt to iteratively improve the predicted mask. In other words, don’t assume that the mask we pass in as prompt need to be the “correct one”… they will be incorrect / a gross estimation of the correct mask, and the networks job is to iteratively improve it till it converges somewhere.

Yeah it’s kinda tricky to understand it. During training, they have the GT, so they can calculate the IOU with the 3 predicted masks and train the model to predict IOU scores. During inference, the three IOU predictions are simply considered as “confidence scores” for each of the three predicted masks. This allows them to rank each of the mask outputs according to how good the iou prediction is. For example, if one of the masks has a predicted IOU of 0.99 it suggests that the network thinks it’s strongly overlapping the queried object. That said, this is still a network’s own confidence on its own prediction, and there is no way of knowing the correct iou score coz we won’t have the GT during inference. It’s all an additional helpful output that tells us the network’s confidence on each of the masks. Hope that helps!

@@avb_fj Thanks for your quick reply! If I understand well, the so-called "confidence scores" during inference, which are in fact calculated from a MLP head with input (3, embedding_dim) and some hidden layers of 256 neurons, in the end it outputs a tensor (3, 1) which represent the probability (0, 1) of each mask after pass a sigmoid activation ?

Sorry for the late response, I must've missed the notification. Fwiw, what you said makes perfect sense to me.@@barbaraz5363

- Predicting the IoU scores helps during inference to determine which of the three predicted masks is most likely to be the "correct" mask to show the user. The three IOU predictions are simply considered “confidence scores” for each of the three predicted masks. This allows them to rank each of the mask outputs according to how high the IOU prediction is. They are basically asking the network to output how confident it is for each of its three predictions. For example, if one of the masks has a predicted IOU of 0.99 it suggests that the network thinks it’s strongly overlapping the queried object. Again, note that this is all an inference-time thing. During training, we have the Ground Truth mask and use MSE loss to train the network to output the correct IOU scores. During inference, we just have the three predicted masks and the network's own confidence score as IoU for each of the three masks. Hope that helps. - Reg the text-prompt usage, they did not release it in their web-app. They just documented it in their paper. Don't know if there are plans to release in the future, or if there are other ways to access it.

@@avb_fj thanks. Have you tried if the CLIP text embeddings would do the trick? Essentially, they mentioned they did train the model with the input.

That’s a great question… you are right the network does not explicitly outputs the part/subpart/whole stuff! Just the annotators are asked to annotate in such a way. I believe what ends up happening is that the network automatically learns to map each output head to one of the three labeled GT. Because each output head also has its own unique output token embedding, they technically learn to associate with different output masks as well. They leave it up to backpropagation and gradient descent to handle the rest… Note that neither during inference nor training do they provide labels for whole/part/subpart to the network… and during prediction/inference too, the network doesn’t explicitly output those labels. It just returns 3 distinct masks and their confidence scores…

During training, the ground truth images and their IOU scores are available, so we can train the SAM network to predict it using supervised training. During inference, the network predicts the segmentation masks and also the estimates of the IOU scores.

It is mostly a design/implementation choice. Generally people would save the mask data in an image format (like png) or in any uint8 format (including numpy arrays)… during training though we would need to load/convert them to tensors for easy gradient calculations…

Thanks for the suggestion. I’ll add it to my bucket list for next month!