So glad you liked it. Your intuition is exactly correct. You would not want to only use the model from the best fold - because that is an overfit model to just that one split. Many times people average across folds, which works well especailly when stacking. However most GMs have perfected the art of re-training the final model on all the data. This sometimes requires additional parameter tuning and is done much more on intuition. For instance training for less epochs because you have more data to train on. Hope that helps!

Thanks for the reply! It'd be great if you could make a future video on retraining the final model on all the data with hyperparameter tuning with perhaps a concrete example. Appreciate your hard work on educating us!

can you please enlight more on this rob, i now understand that cv is a robust method of validating the model performance. but inside the cv loop you fit the model 5times(well according to the fold number) using 5 different train-val dataset. is that mean you're only training 1 model for 5 consecutive time, so each times the model has knowledge of more data. or do you actually make 5 different model using those 5 different dataset?

@@casualgamer91 Here is the sample code for that: ``` from sklearn.model_selection import KFold from sklearn.metrics import accuracy_score from sklearn.pipeline import Pipeline from sklearn.tree import DecisionTreeClassifier import numpy as np models = [] kf = KFold(n_splits=5, shuffle=True, random_state=0) x = df.drop(columns=['target'], axis=1) y = df['target'] tr_acc = [] ts_acc = [] for train_index, test_index in kf.split(x, y): lr = Pipeline(steps=[ ('preprocessing', preprocessing), ('classification model', DecisionTreeClassifier()) ]) x_train, x_test = x.iloc[train_index], x.iloc[test_index] y_train, y_test = y.iloc[train_index], y.iloc[test_index] lr.fit(x_train, y_train) y_train_preds = lr.predict(x_train) y_test_preds = lr.predict(x_test) tr_acc.append(accuracy_score(y_train, y_train_preds)) ts_acc.append(accuracy_score(y_test, y_test_preds)) models.append(lr) def predict_avg(models, new_data): new_data_df = pd.DataFrame([new_data], columns=df.drop(columns=['target'], axis=1).columns) predictions = [] for model in models: prediction = model.predict(new_data_df) predictions.append(prediction) return np.mean(predictions) new_data = [5.9, 3.0, 5.1, 1.8] average_prediction = predict_avg(models, new_data) print(average_prediction) Ground Truth:- 2.0 Predicted Truth:- 2.0 ```

Thanks so much for this thoughtful comment. Even though this is one of my less popular videos I think the topic is really important. Cross validation is super crucial if you want your model to be robust and work on unseen data. A video about classification metrics is a great idea. Thanks again for watching.

Thanks so much David! Glad you found it helpful.

I appreciate that!

Glad you enjoyed it!

Glad you found it helpful! Cross validation is one of the most important things to master.

Thanks Berk! Glad you liked it. Tell your friends. :)

Thanks. I’m nothing special. Just sharing what I’ve learned.

Thanks! Not a professor. Just a guy.

It definitely is important to do cross validation correctly!

Glad you liked it. Optimization for blending and ensembling is a good video idea. I’ll keep it in mind for future videos.

Thanks for the suggestion I’ll add it to the list of ideas for future videos! Thanks for watching.

Thank you! Will do! Cross validation is really important to understand in ML!

Thanks again! I apprecaite the feedback.

Interesting. I’ll have to give it a look.

Glad you liked it. I'll put time series on the list for future videos. If you haven't already check out Konrad's videos on Abhishek's channel: kzfaq.info/sun/PL98nY_tJQXZmT9ZB59T0lsx0ZzzLrYdX4

Thanks for the feedback! `.sample(frac=1)` is just a way of randomly shuffling a dataframe.

Thanks Maqsud. I’m thinking about making a video that goes into detail about it following up on my forecasting video. Thanks for the encouraging comment.

You are exactly right. That is because it prioritizes not having overlapping groups. The very next thing I do in the video is shuffle the target class to give a better example of how StratifiedGroupKFold works in practice.

Great question! If your data size is large, then typrically KFold with shuffle=True is enough. If you are concerned about it you can always create a binary feature and use StratifiedKFold on that feature - or there are some other more advanced stratification packages out there: pypi.org/project/iterative-stratification/ - I could go over those in a future video if you think it might be helpful.

@@robmulla Thanks for the response! I usually KFold with a shuffle. At work, I train a lot of regression models I was just curious if there are general best cross-validation practices for those kinds problems. So a future video on that would definitely be helpful! Maybe another idea for a future video is performance how to identify underperformance of a model on slices/subsections of data. However, no hurry really like the videos and live streams keep it up!

@@maxscheijen I would use pd.cut() to bin my y variable and use stratified k fold with the binned var as a category (temp categorial y var), so that each validation fold has different levels of the y variable represented in it. Obviously discard this binned (cut) version of y-variable before you run fit()

Thanks for watching. Glad you like it.

Great question. There are a lot of resources out there with regards to imblanced datasets. It's a really common problem, but there isn't a one size fits all solution. Generally just having more training data or simpler models is best, otherwise you will find youself overfitting to the few positive samples you have.

Thanks for commenting. I’ve never heard of doing ensembles of cross validation but I don’t see why you couldn’t.

@@robmulla ok thanks. I'd experiment with it

Thanks for watching, feature engineering is #1. Then parameter tuning and pre/post processing. Good luck!

It's funny you say that. I typically do the splits at the beginning and when working in teams we use the same splits so we can share results for the exact same folds. I think this is actually really common on kaggle. With really small data you eventually want to change your random seed and try new folds at some point to make sure you aren't overfitting to a specific seed of folds. Hope that helps!

@@robmulla Thank you very much for the clarification Rob, yes, it helped me a lot.

You're welcome!

Awesome! Glad the video was able to answer your question before I could.

😒

@@robmulla Ungrateful viewers eh