Predict The Stock Market With Machine Learning And Python
Рет қаралды 605,779
Күн бұрынIn this tutorial, we'll learn how to predict tomorrow's S&P 500 index price using historical data. We'll also learn how to avoid common issues that make most stock price models overfit in the real world.
We'll start by downloading S&P 500 prices using a package called yfinance. Then, we'll clean up the data with pandas, and get it ready for machine learning.
We'll train a random forest model and make predictions using backtesting. Then, we'll improve the model by adding predictors. We'll end with next steps you can use to improve the model on your own.
You can find an overview of the project and the code here - github.com/dataquestio/projec... .
If you enjoyed this tutorial, check out this link bit.ly/3O8MDef for free courses that will help you master data skills.
Chapters
00:00 - Introduction
01:28 - Downloading S&P 500 price data
03:30 - Cleaning and visualizing our stock market data
04:29 - Setting up our target for machine learning
08:19 - Training an initial machine learning model
17:01 - Building a backtesting system
23:05 - Adding additional predictors to our model
28:45 - Improving our model
33:37 - Summary and next steps with the model
---------------------------------
Join 1M+ Dataquest learners today!
Master data skills and change your life.
Sign up for free: bit.ly/3O8MDef
@vikasparuchuri Жыл бұрын
Hi everyone! You can find the code for this tutorial here - github.com/dataquestio/project-walkthroughs/tree/master/sp_500 .
@feiziihu Жыл бұрын
Thanks Vik!
@AudaiLouri Жыл бұрын
Thanks Vic, However your F1 score is at 0.5. How does that factor in?
@aarondelarosa3146 10 ай бұрын
Thanks, but it's incomplete.
@saip6126 6 ай бұрын
Hey Viki. You should have used the pd.dropna(inplace=True).
@majorkuntz Ай бұрын
Great video. Will you or can you provide additional information on other useful classifiers and also how to merge other data sources like news and sentiment into this code?
@superztnt 5 ай бұрын
Clear and to the point. I hate super long videos full of things that don't provide much value. This one was great. I like that he walked through general data science/machine learning steps. In particular the data cleansing which many skip over, but it is actually an important step. Also, a pet peeve of mine is audio quality. This video you can hear the presenter clearly and he doesn't sound like his is working from a tin can.
@cooltraderf 7 ай бұрын
Excellent. This tutorial corrects an error that pretty much every other video from others that I have seen has made. Don't seek MSE precision in your target as your goal. That's not what practitioners are looking for. Do what this educator has done instead. This model gets it right as used in the real world. Solid base to work with. Well done!
@rstea 5 ай бұрын
I’m new to coding but have always been an avid market watcher and looking for opportunities. Best video I’ve seen since I started scouring the depths of KZfaq for this content last week. Thank you sir!
@tonimeiners8945 Жыл бұрын
Thanks for your great video. Im curious to read more about the whole issue of predicting actual prices versus only the direction. Do you have a good source on this? I can see why the latter is more robust, but once you start accounting for transaction costs, the magnitude of the direction is also important. curious to get your thought on this too.
@Juoa794 6 ай бұрын
I cannot thank you enough! It's very straight to point and I've learned more in this video than in n online courses and articles.
@jwia007 Жыл бұрын
This was an amazing walkthrough. I have learned so much!
@edmundobrown5604 7 ай бұрын
thank yiu so much fir the video. I have taken varius courses in different places, and your video and teaching style are certainly the best !
@logannon 3 ай бұрын
Great video. Thank you for the insights. Going to be tuning into more of your work.
@khushaalb2688 2 ай бұрын
My man is doing noble work. Kudos!
@emadbagheri1083 11 ай бұрын
Searched & watched a LOT of videos. This is the best. Well done man.
@jsonbourne5085 Ай бұрын
have you tried them? do they work on real data?
@circus14 Ай бұрын
Vik, I echo the compliments on the excellent video. I was able to use my own bespoke weekly market timing signals aligned with weekly S&P closes to finally get a grounded statistical "opinion" on the predictability of forward returns - as only my second Python exercise! Thanks!
@charlene6306 11 ай бұрын
Watched up to 2:26 and I already know this is going to be excellent. Clear and concise explanation from the start and you know this is going to be more than your ordinary YT tutorial
@alang.2054 9 ай бұрын
It's not excellent, you can't beat the market as regular person. You basically compete with Harvard graduates with math, computer science, etc. Degrees. Again, one KZfaq video won't make you beat the market
@killerstar718 9 ай бұрын
@@alang.2054 someone had to break this kids dreams of being rich off a youtube vid
@okoo7385 4 ай бұрын
@alang.2054 Where'd you get that she said she would beat the market from her comment? I read an observation just stating that, this video is higher quality than most YT videos that claim to teach you something specific yet just give you fluff..
@gooddude9211 7 ай бұрын
Very thorough and loved it sir. Thanks for the video lesson.
@Ivan-ou5nq 2 ай бұрын
Explaining is on top. Thank you!
@elu1 Жыл бұрын
What a great framework to ML time-series data for prediction. Thanks for sharing!
@Fred-ut7mc Жыл бұрын
Thats a really good video and it seems you really know what you are talking about. Thanks!
@tomx4278 11 ай бұрын
Excellent video, thank you for sharing this. Hopefully I can see more ML related videos going forward.
@colleen.odegaard 6 ай бұрын
The S&P 500 is still up 10% this year. It's not a get-rich-quick scheme, but it's a proven strategy for wealth accumulation over time, Which happens path i'm considering so as to hedge the losses on my $350k portfolio, but are there any drawbacks to buying such quality stocks?
@Curbalnk 6 ай бұрын
Well, one potential downside is that they may not offer the same rapid growth potential as riskier, smaller-cap stocks. So, it depends on your goals and risk tolerance. you may want to work with a financial advisor who can help with right approach.
@TeresaBrickle 6 ай бұрын
this is definitely considerable! think you could suggest any advisors i can get on the phone with? i'm in dire need of proper portfolio allocation
@TeresaBrickle 6 ай бұрын
very much appreciated, your response suggests a person of benevolence.. just inputted her full name on my browser, and came across her site, top-notch qualifications! she seems well-qualified
@upsenloyn 2 ай бұрын
@@TeresaBricklefuck you bots no ones gonna fall for that
@pinecedar180 2 ай бұрын
Spam comment chain, please remove
@ricjrob 8 ай бұрын
Great video. Really clear and at a pace that allowed me to follow it easily and learn some new and simple techniques in how to manipulate data.
@henriquesousa4789 2 ай бұрын
The features used for the random forest cannot be the high, close, low , open values directly without any transformation because what the model is essentially doing is creating a overfit of non linear decisions to certain prices ranges. It is basically memorizing that when the close was above X value and open below Y value predict 1 or 0. You need to normalize the predictors in some way so that the model can use them independently of how high the value the stock is and truly create generalizable rules. Ratios are good since they use percentage instead of using absolute values and allow the model to use information of multiple candles as well.
@kibs_neville Ай бұрын
Quite important comment.
@Templar_of_the_Clean_Code 5 ай бұрын
Very useful man, thanks for show us the way!
@mohibahmad5834 2 жыл бұрын
Sir your explaining skills are top notch
@chiroyce 5 ай бұрын
DUDE THIS IS SO HELPFUL
@idkidkidk3488 11 ай бұрын
This is awesome, instead of showing what you need to learn or try it shows how to actually build a model. This is very usefull. Thank you!
@idkidkidk3488 11 ай бұрын
Could we get a similar video bus featuring a deep learning model instead?
@alang.2054 9 ай бұрын
What are you talking about? Do you really think this guy would show you real ways to make money? On market you compete with professionals in multi billion hedge funds with degrees, you can't beat them with KZfaq video
@anujsaraswat2257 3 ай бұрын
I'm hoping you can do a follow up video to this. Would be great to see how you would incorporate macro data into your model, such as news or interest rates.
@abidson690 Жыл бұрын
Thanks so much, you're a blessing
@SolidBuildersInc 11 ай бұрын
This was very well delivered. Thank yo sharing. I will consider the suggestions you made and see how this works. Very exciting with a bit of 😅.....
@anabondal1471 Жыл бұрын
Thank you very much for this! Truly found this useful for my first ML Project. However, a bit confused by the 'combined' graph - how did you get it? :) (I had to do mine using the train_test_split import.)
@InvestorLondon Жыл бұрын
Incredible video! This helped me a whole lot I really do appreciate it! I Just Liked and Subcribed!
@sergiysergiy8875 9 ай бұрын
Great tutorial!
@sergiysergiy8875 9 ай бұрын
How would you use the volume column? Not sure how to use the volume, can we build some relative volume indicator? Can you give a hint, or maybe a link to a video, where you use volume somehow to improve your model? Volume should influence the model significantly.
@TheWiewiorski Жыл бұрын
Vik thank you for this video! Greetings from Poland. Please explain to me how to connect the model so that operating on a virtual server bought and sold instruments? How do you combine it?
@raushankumar5533 Жыл бұрын
Thanks for your great video. Im curious to read more about the whole issue of predicting actual prices versus only the direction. Do you have a good source on this?
@AVOWIRENEWS 3 ай бұрын
Wow, the concept of predicting the stock market using machine learning and Python is such a fascinating topic! The blend of finance and technology is always an area ripe for innovative approaches. It's impressive how machine learning can analyze vast amounts of data to find patterns that might not be obvious at first glance. Python, with its extensive libraries and community support, is an excellent choice for such complex computations. It's exciting to think about how these tools can provide insights into market trends and possibly even predict future movements. The intersection of machine learning and finance is definitely a space to watch! 📈💡🤖
@whansen101 5 ай бұрын
Super helpful - Thank You !!!
@maurovallefilho4576 Жыл бұрын
Hi Vik. Thank you very much. Is it possible to predict two days in advance instead of just tomorrow?
@Maximus18.6 5 ай бұрын
Congratulations for your explanation and it was very clear. I would like to suggest you to prepare a vide including news about the stock into this model. Thanks
@annarocha9769 Жыл бұрын
thank you thank you !! this is great, suscribed :)
@AddilynTuffin 7 ай бұрын
Buying a stock is easy, but buying the right stock without a time-tested strategy is incredibly hard. I’ve been trying to grow my portfolio of $160K for sometime now, my major challenge is not knowing the best entry and exit strategies
@DanielPanuzi 7 ай бұрын
Investors should be cautious about their exposure and be wary of new buys, especially during inflation. Such high yields in this recession is only possible under the supervision of a professional or trusted advisor.
@RickWatson-xu6gw 7 ай бұрын
I have been speaking with a coach for a long time now mostly because I lack the background knowledge and mental toughness to handle these reoccurring market conditions. I made over $220K during this drop, which proved that there is more to the market than the average person is aware of.
@NormanGhali 7 ай бұрын
I just started a few months back, I'm going for long term, I'm still trying to wrap my head around it, who’s this advisor you work with?
@RickWatson-xu6gw 7 ай бұрын
Credits to *Sharon Louise Count* one of the best portfolio manager;s out there. she;s well known, you should look her up
@louis71350 7 ай бұрын
I Found her online page by searching her full name, I wrote her an email and scheduled a call, hopefully she responds soon. Thanks
@Poppinthepagne Жыл бұрын
Thank you so much, I’m learning to build and plot models, I’m basically copied your code and tried to understand it, What’s your advice to learn how to do it yourself?
@rverm1000 Жыл бұрын
cool went threw the whole process on mini conda.
@tochukwuumunnakwe2300 6 ай бұрын
Hi, great lesson, I have a question. I'm still new to data science. But why didn't you use the data as a predictor? Im asking because say we want to predict what happens in the next day. How do i pass it to the model when i didn't train with it
@Ganndude2004 2 жыл бұрын
Great video , I hope to see more tutorials like this in the future.
@ereztison Жыл бұрын
Great video, thank you!
@johnnydavidsantana1935 Жыл бұрын
Hi, how do I predict the next , for instance in a new data.
@BaoTran-jo8lj 8 ай бұрын
Thank you for your videos. But what if I have multiple stocks to predict, and when I parse one stock id in, I want to get the specific prediction for that id only. will it be feasible?
@MrPotatoHeadFX Жыл бұрын
Hey man, how did you get into this kind of work? Im so keen to find some work doing what you did but am finding limited possibilities
@koopstakh301 Жыл бұрын
These are great for practice Keep em coming
@Dataquestio Жыл бұрын
Glad you like them, Prathamesh! -Vik
@abdulkareemridwan8762 2 жыл бұрын
Great tutorial 🙏
@alexanderrooth1940 Жыл бұрын
Great job! I used the majority of your code but for a specific company. My personal aspect is that this "result" is a bit messy. Do you have any tips on how we could make a clear graph towards the end with "predicted values"? I tried graphing with "Tomorrow" with respect to "Close"m but no difference. Part of that reason could because of the wide X-axis. Thanks again, looking forward to your answer! / Alexander
@kayakablejourneys 7 ай бұрын
Great video. It seems that the yfinance api is no longer functioning. Could you please do an updated video using a different method to collect the date? Thanks.
@SeamlessSolutions 9 ай бұрын
Thank you ❤❤
@arpansaha4688 5 ай бұрын
Which SOFTWEAR used for run this code ?
@nanjundarao9568 Ай бұрын
Hi can we use this for Indian stock markets?❤
@tradercrypto_lad8929 2 жыл бұрын
Cool Video! Thank you!!
@peterbogar3427 3 ай бұрын
Very good explanation, thanks.
@mda99das 11 ай бұрын
How has the model done this year? Does it show a topping formation?
@jacarthymack1111 Жыл бұрын
How do you add additional columns that will display information from yahoo finance such as pe ratio dividens and so on
@ew9373 9 ай бұрын
Thanks, Vic.
@user-eb3hp7kk7s 8 ай бұрын
hi, I wonder how reliable would this be if I predicted the 10, 20, or more candles into the future with an accuracy of 75 to 90 percent. do you think its gonna be useful in the financial markets. since I did create features which predicts the prices with an accuracy of 85 percent.
@MattDouglas-hj9wh 8 ай бұрын
My spouse and I are adding a variety of stocks/ETF to my present holdings for the long term, We've set aside $250k to start following inflation-indexed bonds and stocks of companies with solid cash flows, I believe it is a good time to capitalize on the market for long-term gains, but it wouldn't hurt to know means of actualizing short term profit.
@graceocean8323 8 ай бұрын
The current market might give opportunities to maximize profit within a short term, but in order to execute such strategy , you must be a skilled practitioner.
@hannahdonald9071 8 ай бұрын
Having an lnvestment advser is the best way to go about the market right now, especially for near-retirees, I've been in touch with a coach for a year now mostly because I lack the depth knowledge and mental fortitude to deal with these recurring market conditions, I nettd over $320K in profits so far, Its clear there's more to the market that we avg joes don't know that Investment advisors know.
@hannahdonald9071 8 ай бұрын
Salvatore Fortunato Sofia. You can easily look her up, she has years of financiaI market experience.
@trazzpalmer3199 8 ай бұрын
Thank you for this tip. it was easy to find your coach. Did my due diligence on her before scheduling a phone call with her. She seems proficient considering her résumé.
@jackrozmaryn7905 3 ай бұрын
Amazing video!! Have yiou looked at the performances of other ML techniques, e.g, MLPregressor?
@saadbhai5998 8 ай бұрын
Hello! Why the column "Tomorrow" wasn't used for training? 🤔
@wellsHannahh 3 ай бұрын
Do you think it's a good time to consider selling some stocks, or is it better to hold onto them for the long term? I’m considering rebalancing my $2M portfolios, So I'm curious about the best strategies for potential market downturns
@maddysys 3 ай бұрын
I guess it's important to reassess your investment strategies based on current market conditions. You should also consider a market expert to guide you.
@eastwood224 3 ай бұрын
You're right mate! I’ve been using a fin-market expert for two years now and I own a 7figure diversified portfolio from investing in stocks. Currently, my portfolio is worth over $900k.
@chrissmurrayy 3 ай бұрын
How can I participate in this? I sincerely aspire to establish a secure financial future and am eager to participate. Who is the driving force behind your success?.
@eastwood224 3 ай бұрын
Nicole Desiree Simon is the licensed fiduciary I use. Just research the name. You’d find necessary details to work with a correspondence to set up an appointment.
@chrissmurrayy 3 ай бұрын
She appears to be well-educated and well-read. I ran an online search on her name and came across her website; thank you for sharing.
@jeevanjose6986 2 жыл бұрын
Brilliant video Vik! Towards the end, you mentioned adding news to the model. Could you share how one could integrate that? Thanks!
@Dataquestio Жыл бұрын
Hi Jeevan - the easiest way to do it is to scrape daily headlines from say the new york times, and create a "sentiment" model to indicate confidence in the market. The output of that model could then be a predictor column. Of course, you could get a lot more complicated than this :)
@sikazwe2010 Жыл бұрын
Hello Vik, Thanks for the great tutorial, really informative. Do you know how to add lorentzian classification to the model in your example?
@harryhamtaro7990 8 ай бұрын
Which platform are you coding on? Is it via google collab or jupyter?
@RK-xe3tw 7 ай бұрын
Actually you forgot to measure the expectancy of a trade in the case it has a precision of 42%. Because what makes a strategy profitable is bit the win rate but rather the expectancy of the trades. Although it is a great video and a good tutorial about programming. Thanks and keep up the good work.
@Juoa794 6 ай бұрын
Isn’t there leakeage in the ‘trend’ feature, considering it is a function of future values (‘target’)?
@alrey72 9 ай бұрын
Good and clear explanation :) Although there are other factors to be considered like bid offer spread and commissions. Also, when the market goes against you, do you wait before the end of day to close the losing position? Maybe setting a stop loss and including it in the model and back testing can help. Thanks.
@Mike-fm3km 7 ай бұрын
how would commissions help? lol
@alrey72 6 ай бұрын
@@Mike-fm3km In the back testing of the model, it may seem profitable but after considering the commissions/transaction fees, it might be unprofitable instead.
@christopherreberger5450 11 ай бұрын
I suggest you google the semi strong efficient market hypothesis. Would save a lot of time.
@psimondk 6 ай бұрын
Hint: on a recent macbook you can use all its cores by: import joblib N_CORES = joblib.cpu_count(only_physical_cores=True) ... model = RandomForestClassifier(n_estimators='your value', min_samples_split='your other value', random_state=1, n_jobs=N_CORES) The speedup is amazing
@DonFranciscoUSF 5 ай бұрын
you don't need any information about the system to do this, n_jobs = -1 will use all the available cores with no imports or extra lines :)
@0821vijay 4 ай бұрын
Do we have any latest updates to this model? Adding extended logic for improvements?
@markk364 3 ай бұрын
What did you use for the risk rate as there is no such thing that exists in finance
@aarthibhandary5120 11 ай бұрын
How can we add news/articles into the model?
@skiraf Жыл бұрын
Excellent Video. Thank you for sharing. Question, how can we compare the 'influence' from another stock in the same industry, ie, two retail stocks, or two energy stocks?
@jitendersinghvirk47 11 ай бұрын
correlation maybe.
@thetahayaseen 8 ай бұрын
Hey Real Quick Question, Can We Get Predictions For Each Single Stock?
@govardhanab7223 4 ай бұрын
hello sir , can this be used for day trading , in indian market for options trading of bank nifty and nifty in a 5 minutes candle time frame during market hours and feeding real time data?
@investidorcalejado8344 7 ай бұрын
is it possible to have a view of the daily basis, but also input training on intraday data to improve the daily view?
@alan614 4 ай бұрын
Great stuff!
@ec92009y 9 ай бұрын
Excellen video. I think you have a great teaching ability. I'm surprised you did not start with the usual "THIS IS NOT FINANCIAL ADVICE..." disclaimer 😇
@AtticusDenzil Жыл бұрын
great channel, will try to get some of my time to get to do something meaningful with the help of dataquest
@ajdaria1000 3 ай бұрын
Excellent video!
@harunnmsanee5371 10 ай бұрын
Is there next project where you improved the accuracy of the model to a higher percentage
@MRDEWVOLTAGE 7 ай бұрын
how can i expand... do you have more insructions?
@isaackodera9441 Жыл бұрын
I got an error all_predictions[ ] is not defined. Use all_predictions = [ ] instead and it will work for you. Hope this helps
@bhavyamehra6931 9 ай бұрын
i didnt get the point of shift{1} in the trend column? why shift 1 forward?
@miacustica 6 ай бұрын
Hello, thank you very much for the video, I am new to ML, I would like to know how to use the model? How do I see the prediction for the next day? thanks and greetings
@rosekate965 11 ай бұрын
I need help to install yfinance package
@skiraf Жыл бұрын
In Step:31 (Time 20:00) the 10 year loop looks very similar to LSTM. Why not use LSTM instead?
@DSDoesDS Жыл бұрын
Ill take a notes: the model without hyperparameter tuning. if hyperparamter tuning is done, when backtesting we no longer need to look for the best parameters. In contrast to cross-validation which requires more tuning
@mistletoe91 9 ай бұрын
Great, have you tried to improve the model ?
@tharindumadusanka9727 10 ай бұрын
hey, please can anyone tell me, is this one classification model or regression model?
@panfriedbacon Жыл бұрын
Hi Vik - thank you for the great video This could be a dumb Qs - in "Improving Our Model" section, why didn't you change Predictors to "NEW_Predictors" when you defined the function/ when you've copy paste? Does this matter? Thank you, AL
@dhananjaysharma3255 9 ай бұрын
"NEW_Predictors" was passed while calling backtest function which calls predict function with "New_Predictors". Hence New_Predictors was used for modelling
@abhikmukherjee2926 9 ай бұрын
Hi @Everyone, I am getting this following error when trying to get the predictions for the second time on the new_predictors Code: predictions = backtest(nifty50, model, new_predictors) FYI, I am using Nifty50 dataset. ValueError: Length of values (1) does not match length of index (250) Can anyone guide me through this error, I am not getting it. Any help would be much appreciated.
@bmariani52 9 ай бұрын
What are the Profits after trading every day compared to SPY performance as the benchmark.
@maburwanemokoena7117 11 ай бұрын
when you split the data into the training and testing dataset, you are actually performing what is called Simple Random Sampling, this will cause the training data to have the same elements/characteristics of the testing dataset. If you were to calculate the means of each predictor variable in the testing and training dataset it will roughly be the same due to random sampling. The point I am trying to make is that you cannot claim the model has not "seen" the testing data, yet it managed to capture the majority of its properties due to simple random sampling, how about you train the model using the first 70% rows then leave the remaining 30% at the bottom for predictions? In that way the model does not have any idea what's happening with the remaining 30% (though there is an argument one can put forward about this), I think that approach would be the most realistic. I have used the simple random sampling before and I have gotten results which seemed to be accurate, it was not until I used this method I am suggesting to you that I obtained a little bit higher errors.
@alexanderrooth1940 Жыл бұрын
Great video but where is the clarification that it will go up or down tomorrow?
@rburnettcpa 11 ай бұрын
What a deep voice
Rob Mulla
Рет қаралды 173 М.
Lex Clips
Рет қаралды 114 М.