Notebook-1: www.kaggle.com/abhishek/30-days-create-folds Notebook-2: www.kaggle.com/abhishek/competition-day-1-baseline Please subscribe and like the video to help me keep motivated to make awesome videos like this one. :)
@harishchandrasekaran60963 жыл бұрын
I just saw your video thrice, searched for what is kfold once more. And then I was able to understand your code mostly. Since its my first time, I guess I have such a feeling. And the ones after for loop was amazing. I need some more practice to think sequentially that way. Thanks once again for the amazing tutorial.
@malawad3 жыл бұрын
I love those walkthrough especially in a competition like this , any more coming ?
@abhishekkrthakur3 жыл бұрын
obviously! ;)
@praveerparmar81573 жыл бұрын
Hi Abhishek, when I started reading your book AAMLP, I felt a bit confused while creating the folds. But after watching this video, everything has become crystal clear. Thanks a lot for such lucid and crisp explanation.
@williamsokol03 жыл бұрын
Thanks for giving us a place to start, I was absolutely clueless on what do before this video
@TheMISBlog3 жыл бұрын
Great Content Mr.Abhishek, Keep up the good work
@sumeerabhat3 жыл бұрын
I am new to DS ML and kaggle as well and I was confused how to start with kaggle and your tutorials cleared a lot more concepts ..Thank you so much..
@GIT_Somya3 жыл бұрын
Damn it 20 days ,and I'm seeing this now! I feel sad for missing out but I'm finally here
@RaushanKumar-qb3de3 жыл бұрын
Thanks Abhishek sir!😀 I learned a lot, this will improve my level of coding. Will follow your all upcoming videos.Thanks Again for channel!!
@vaishnavibelsare78473 жыл бұрын
Thanks for this video. It was helpful. Please elaborate and explain a little bit more so that if we messed up with concept during these 15 days, those concepts may get cleared.
@naylamp_gaming3 жыл бұрын
A thousand thanks!! Your tutorials are the best :) I will purchase your book
@siddharthganjoo3 жыл бұрын
Thank you for this tutorial, you videos are really helpful
@YuI-lf8dg2 жыл бұрын
thank you, this help me a lots!
@sunnybhojwani31992 жыл бұрын
Great Explanation
@theDrewDag3 жыл бұрын
This is friggin gold.
@dhruvnivatia92223 жыл бұрын
You deserve millions of suscribers....
@somiab58573 жыл бұрын
Your video was very useful for me.thank you very mach for your nice explanations.
@penninahgathu79563 жыл бұрын
Thank you for your amazing work. I'm learning alot from you
@MonikaRabha2 жыл бұрын
26:40 i love this xD
@renelara53393 жыл бұрын
Thank you, nice video
@junsenchen45143 жыл бұрын
Really appreciate your video, learnt a lot from it
@purposeoriented60943 жыл бұрын
It was A little harder and thank you for the great help...
@mithilesh033 жыл бұрын
@Abhishek sir, can you please create a video on pipeline especially performing feature engineering (feature interaction to create new features) through pipeline? This will be really helpful. Thank you
@discsanddata60213 жыл бұрын
You are a legend.
@AlexKite683 жыл бұрын
Wow! Great work! Thank you!
@marinagorden113 жыл бұрын
wow great work, genuine thanks!
@talhaanwer85593 жыл бұрын
@1:56 you are saying… ….you can just simply use k-Fold, All you need to do is you have to see if the distribution of data is same in each fold if it is not use others… I want ask how you will know by looking at histogram that the distribution of data is same in each fold?
@dustin121883 жыл бұрын
Thank you so much for this! Can you explain why we do df_train.loc[valid_indices...] and not train_indices?
@appunram68813 жыл бұрын
I am also having the same question. Did you find it?
@rishabhchandel47863 жыл бұрын
Each time going through loop valid indices form 20% of the data, since kfold is 5 and train indices form 80%. Valid indices would be different for each loop. In 5 loops all data point will become validation data in exactly one loop or in one of the folds. Note this is not the case of training indices. Since a single training data would be utilized across multiple loops(4folds or loops to be exact.). Hence valid indices is utilized to uniquely assigned fold values and not training indices
@appunram68813 жыл бұрын
@@rishabhchandel4786 Thanks. I got it 👍
@M_SAI_VARMA3 жыл бұрын
Thanks a lot 😊
@HARSHRAJ-20232 жыл бұрын
Hi at 4:37 you says we don't need train indicies we need valid indicies. Can you explain reason behind it please?
@adnana23513 жыл бұрын
I was running this following cell final_predictions = [] for fold in range(5): xtrain = df[df.kfold != fold].reset_index(drop=True) xvalid = df[df.kfold == fold].reset_index(drop=True) xtest = df_test.copy() ytrain = xtrain.target yvalid = xvalid.target xtrain = xtrain[useful_features] xvalid = xvalid[useful_features] ordinal_encoder = OrdinalEncoder() xtrain[object_cols] = ordinal_encoder.fit_transform(xtrain[object_cols]) xvalid[object_cols] = ordinal_encoder.transform(xvalid[object_cols]) xtest[object_cols] = ordinal_encoder.transform(xtest[object_cols]) model = XGBRegressor(random_state=fold, n_jobs=4) model.fit(xtrain, ytrain) preds_valid = model.predict(xvalid) test_preds = model.predict(xtest) final_predictions.append(test_preds) print(fold, mean_squared_error(yvalid, preds_valid, squared=False)) But getting the following error AttributeError Traceback (most recent call last) in 5 xtest = df_test.copy() 6 ----> 7 ytrain = xtrain.target 8 yvalid = xvalid.target 9 /opt/conda/lib/python3.7/site-packages/pandas/core/generic.py in __getattr__(self, name) 5463 if self._info_axis._can_hold_identifiers_and_holds_name(name): 5464 return self[name] -> 5465 return object.__getattribute__(self, name) 5466 5467 def __setattr__(self, name: str, value) -> None: AttributeError: 'DataFrame' object has no attribute 'target' Can't figure out what happened
@user-wo1dn4do6x3 ай бұрын
Hello sir, can you reopen the data? This contest is no longer available. Thank you.
@kafaayari3 жыл бұрын
Hi Abhishek, Thx for the great content. I just want to know name of below technique you used. - in each fold, you fit the model with training data of the fold. - with this model, you make prediction test data, and stack this prediction - finally you average the predictions of test data at the end. this is not cross validation, not out of fold prediction whatever.What is it called?
@seemasharma-mn7fk3 жыл бұрын
Thank you!
@Frank-mu6qy Жыл бұрын
Sir,I'm watching your video,how can I download the datasets inyour video?I can't fork your project from kaggle
@romainleclair51193 жыл бұрын
Amazing ! ;)
@shreymittal39072 жыл бұрын
Hey Abhishek just wanted to ask if challenge is open now. Or it is closed
@dhruvnivatia92223 жыл бұрын
i have one question, after applying cross-validation, how can I select the best model ( through which I can make predictions )??
@youssefbakadir26253 жыл бұрын
Hi Abhishek! thanks again and again for your valuable videos! My question is about the autocompletion will codding in kaggle notebook ! plz is there any way to get the autocompletion in kaggle notebook without using the Tab key.
@manisha02093 жыл бұрын
Thankuu so much sir
@harshraj57392 жыл бұрын
Hi Abhishek. Can I get this dataset for practice ? It would be of great help if I get this as csv in my local for learning.
@shailendrayadav31552 жыл бұрын
Hi Abhishek, I have started the playlist, Is there a way you can post the original data as its not available coz the competition is closed. I am not able to follow this video hands on due to the data issue.
@sollysebela22593 жыл бұрын
Keep it up bro
@pankajshaw6743 жыл бұрын
Is there any interesting story behind random_state = 42;
@hemesh56633 жыл бұрын
Could some one pls do explain me why he is assigning kfold value on train data set only on valid indices
@niteshkavathankar13003 жыл бұрын
when i tried oridinal encoding to tranform test data i got error of unknown category but in Abishek sir he didnt got error why this happned BTW i didnt used it inside for loop i tried to do it out side for and was goin to use gridsearch CV
@Name-pss3 жыл бұрын
Thanks for the tutorials, they were really helpful. I am not a participant so I was wondering if there is a way for me to get the train and test data. Still gonna sponge off the content and other's notebooks but was wondering if I could do it on my own too.
@abhishekkrthakur3 жыл бұрын
TPS feb 2021 competition is very similar to the data in this one. you can even follow it with the videos ive posted.
@Name-pss3 жыл бұрын
@@abhishekkrthakur thanks for the tip...will do. Just saw an old comment with the same issue. Thanks for replying.
@akshayshimpi12653 жыл бұрын
I loved the video, but I have one question that. How target column is calculated?
@gaguirre043 жыл бұрын
man, target column is a data column that allows you to perform the learning. It is used when fitting the model.
@penninahgathu79563 жыл бұрын
Hello guys, I am still currently having a hard time understanding this block of code kf = model_selection.KFold(n_splits=5, shuffle=True, random_state=42) for fold, (train_indicies, valid_indicies) in enumerate(kf.split(X=df_train)): df_train.loc[valid_indicies, "kfold"] = fold particularly this line, df_train.loc[valid_indicies, "kfold"] = fold, why excatly are we doing this? I can't seem to wrap my head around it
@abhishekkrthakur3 жыл бұрын
- in each iteration we divide the original data into two parts - one part is 80% and other is 20% - 20% is the validation data - lets say we are in second iteration - training data in 2nd iteration will have overlap with training data in 1st iteration - validation data in 2nd iteration will have no overlap with validation data in 1st iteration - in each iteration, set of validation data (or validation indicies) are disjoint with any other iterations's validation data - after 5 iterations and 20% of validation indicies in each iteration, we cover the whole original training set - thats why we use validation iterations and not training which might have overlap
@penninahgathu79563 жыл бұрын
@@abhishekkrthakur Thank you, I understand it now
@mithilesh033 жыл бұрын
Thank you for this video, sir @Abhishek Thakur. Quick question, at 5:45 duration, you are updating the valid_indices with fold (df.loc[valid_indices,'kfold'] = fold), and in the next line when you type head, all instances including the train_indices get updated with the fold value. Can you please explain this? I didn't understand how the train_indices updated with the fold value. Thank you
@mithilesh033 жыл бұрын
Sorry, I got it, please ignore this comment. Thank you
@HARSHRAJ-20232 жыл бұрын
@@mithilesh03 Hi Can you explain me in understanding because I am also having the same query.
@sylvainthibault97333 жыл бұрын
What is your dark theme setup ? Looks really good.
@abhishekkrthakur3 жыл бұрын
its kaggle notebooks's dark theme. :)
@amrirasyidi3 жыл бұрын
2:05 Do anyone know in which video the kfolds mentioned earlier? Or let met just ask it here, what is the reasoning behind creating the folds? Thanks!
@abhishekkrthakur3 жыл бұрын
its the same reasoning behind any kind of cross validation. since all parts are connected, another reasoning will be clearer in parts 5 & 6
@amrirasyidi3 жыл бұрын
@@abhishekkrthakur I see. Will look up to that Thanks a lot!
@manojkumar-ir8lh3 жыл бұрын
I am trying to read train_folds data like pd.read_csv("../input/30days-folds/train_folds.csv") but its throwing file not found. Can you tell me how I can refer your public uploaded dataset
@abhishekkrthakur3 жыл бұрын
Click on "add data" button on top right corner and then look for "30days-folds". Add it and when its added, you will be able to read train_folds.csv
@md.mahmudulislam48023 жыл бұрын
Sir, Who Missed registration 30daysofml challenge he/she can participant these competition using his kaggle account If can, How to get competition link?
@rajathslr3 жыл бұрын
On which day of "30 days of ML "was the "kfold" stuff explained ?
@abhishekkrthakur3 жыл бұрын
kzfaq.info/get/bejne/Z-Cfq9hks5-YoYk.html
@rajathslr3 жыл бұрын
@@abhishekkrthakur it's such a surprise to get a reply from Abhishek, thanks very much!!
@rfa86683 жыл бұрын
Dear Prof. thank you very much for your effort,. After the day no. 15 I didn't receive any mail, and from that day the rules of competition is not clear for me.
@abhishekkrthakur3 жыл бұрын
on day 15, you received link to competition. so, now you need to join and take part in that competition. ive made 4 parts of tutorials for competition till now.
@rfa86683 жыл бұрын
@@abhishekkrthakur Thank you very much for your help. God save you.
@sagarkhule64393 жыл бұрын
Like wise I m also not able to participate I didn't get an. Y e-mail 🥺
@appunram68813 жыл бұрын
What is the use of putting index=false?
@abhishekkrthakur3 жыл бұрын
put index=true and see the resulting csv to know the reason :)
@trojan123523 жыл бұрын
Sir, why did we create a seperate file with Folds ? Can't we use that for cross_val_score ? Please let me knkow, I'm just trying to understand why create a Folds file and iterate ourselves, while cross_val_score does that for us. Pretty new here so trying to connect the dots. Thanks in Advance.
@abhishekkrthakur3 жыл бұрын
We will know in next couple of videos :)
@sagarkhule64393 жыл бұрын
Hello sirrr , I'm following this playlist since day 1 as I was late for my enrollment I'm not able to make a notebook in competition So what should I do now Like should I jst go through it ? Or make some private notebooks for reference..???
@narendralv63793 жыл бұрын
Please check your email. You should have received a link to competition on day 15. Join the competition using the link and you should be able to create notebooks. I did the same today and it worked.
@sagarkhule64393 жыл бұрын
@@narendralv6379 can you forward the same link here please it will be helpful
@ashrafarzu32853 жыл бұрын
Sir, when i am trying to add df_test= "../30-/test.csv file, it is showing error. Would i have to download test.csv file too and add here? Or i directly use it from competition data? If it is. How will i do it then? @Abhishek Thakur
@abhishekkrthakur3 жыл бұрын
you need to add that dataset to your kernel. please see the beginning of this video: kzfaq.info/get/bejne/o5uJhq6BstDPop8.html
@phucn55893 жыл бұрын
Can anyone share input dataset plz? I'm late to competition, and access restricted to participants only. Thanks
@abhishekkrthakur3 жыл бұрын
try this competition data: www.kaggle.com/c/tabular-playground-series-feb-2021 its very similar to the one going on!
@phucn55893 жыл бұрын
@@abhishekkrthakur Thanks Abhishek!
@sylvainthibault97333 жыл бұрын
When running the loop to set the kfold column I get "Your notebook tried to allocate more memory than is available. It has restarted." Do you have more memory allocated to your account ? If yes, how does one increase allocated account memory.
@abhishekkrthakur3 жыл бұрын
for this dataset, you shouldnt get memory error. did u fix the issue?