Very satisfying to know it was helpful @cexploreful.

Very happy to know you liked.

Thanks @caiyu538

You'll typically use a shared base model, for instance, a BERT model. On top of this, you can have task-specific heads. e.g If planning to do both a NER + sentiment classification, you might have a dense layer leading to token-level predictions for NER and a dense layer leading to a single prediction for sentiment. Each batch of data should contain information for both tasks. For Loss Calculation: For each task, you'll have a specific loss (e.g., cross-entropy for both NER and sentiment). The total loss for a batch will then be the sum of these losses, possibly weighted if you wish to give more importance to one task over the other. It would look something like this in PyTorch: ```py loss_ner = criterion_ner(predictions_ner, targets_ner) loss_sentiment = criterion_sentiment(predictions_sentiment, targets_sentiment) total_loss = loss_ner + loss_sentiment ```

Yes absolutely. you just need to extend the existing model and add your layers in the newly defined class.

Even for Text Classification you have to add a custom head layer, which may have a combination of Dropout Layers and Linear Layers (assuming you are just adding plain vanila Neural Networks at the head layer) The body of the BERT model will just output an embedding vector of size 768 in each of the tokens. And then, to use these vectors as an input for our different kinds of NLP applications (e.g. text classification, next sentence prediction, Named-Entity-Recognition (NER), or question-answering) - we need to add custom head layers. For example, for a **text classification task**, we focus our attention on the embedding vector output from the special [CLS] token. This means that we’re going to use the embedding vector of size 768 from [CLS] token as an input for our classifier, which then will output a vector of size the number of classes in our classification task. And that's why we do like below implementation ``` class BertClassifier(nn.Module): def __init__(self, dropout=0.5): super(BertClassifier, self).__init__() self.bert = BertModel.from_pretrained('bert-base-cased') self.dropout = nn.Dropout(dropout) self.linear = nn.Linear(768, 5) self.relu = nn.ReLU() ``` The above code assumes I am doing text-clasification and number of classes are 5. So, the Linear Layer stars with 768 dimensional vector (which is the output from the body of the BERT model), and at the end of the Linear Layer, we have a vector of size 5, each corresponds to a category of our labels or classes.

Thanks for watching.. For the part `sequence_outputs[:, 0, : ]` - Here, `sequence_outputs` is a tensor of shape (batch_size, sequence_length, hidden_size), where batch_size is the number of input examples, sequence_length is the length of the input sequence, and hidden_size is the size of the hidden state for each token in the sequence. The notation [:, 0, :] selects the hidden state corresponding to the first token in each sequence (i.e., the [CLS] token in the case of models that use the BERT architecture). The resulting tensor has shape (batch_size, hidden_size). Going back to the fundamentals, the Last Hidden State output is the sequence of hidden-states at the output of the last layer of the model. The output is usually [batch, seq_len, hidden_state], and it can be narrowed down to [batch, 1, hidden_state] for [CLS] token, as the [CLS] token is 1st token in the sequence. Here , [batch, 1, hidden_state] can be equivalently considered as [batch, hidden_state]. And thats what I am doing with `sequence_outputs[:, 0, : ].view(-1, 768 )` Since Transformers are contextual model, the idea is [CLS] token would have captured the entire context and would be sufficient for simple downstream tasks such as classification. Hence, for tasks such as classification using sentence representations, you can use [batch, hidden_state].

@@RohanPaul-AI It's super clear, thanks ! 🙂

@Hemal, would suggest uninstall the current transformer library and then a fresh reinstallation from source pip install git+github.com/huggingface/transformers.git Check out this github issue - github.com/huggingface/transformers/issues/1878#issuecomment-558238488

@@RohanPaul-AI hey 👋 Thank you for responding but I tried that already and it didn't work. However I'll give a fresh try soon. 😅

Thanks for the suggestion. That may be after a while in future.