Great Video! However, there appear to be a number of missing steps. Maybe that was in a prior video but you never said so. For example, where did the transition matrix come from? You mention very briefly about from past data, but never connected what that means or how it’s done. Similar for rest of the process. Maybe a follow up “show me” video would be in order.
@humanparaquat695 ай бұрын
We love your videos dude!
@ritvikmath5 ай бұрын
Love you all 🙏
@ekaterinaburakova86295 ай бұрын
Awesome series, keep up the good work!
@ritvikmath5 ай бұрын
Thanks, will do!
@nathanlewis425 ай бұрын
Is the code for this in your KZfaq repo on github?
@poisonza5 ай бұрын
can you do a notebook demo on how to fit hmm and do inference on out-of-sample data? maybe include these - fit returns on other distribution other than gaussian (single, multi-variate case) - set initial state transition matrix ; like higher value to diagonal entries (since regimes rarely changes) - show us how the input(length of observation) might change the inference value
@ritvikmath5 ай бұрын
Good suggestion!
@gsm74905 ай бұрын
Really great content! If you don’t mind, may be there is another way to use ML and NN in financial market: you can not only predict future returns and prices, you can also predict how different factors affect price fluctuations right at the moment?
@ResilientFighter5 ай бұрын
Thanks Ritvik! Great content as usual
@ritvikmath5 ай бұрын
My pleasure!
@posthocprior5 ай бұрын
Maybe I've missed something: what exactly is the latent variable you're trying to approximate? If, as you've described, the sentiment values of positive, negative, and neutral correspond to distinct normal distributions with respective means, then this is simply a linear model. It would be hidden if the three sentiment values corresponded to the three, respective distributions and the latent variable was the stock price.
@Gingeey235 ай бұрын
Great video - thanks for sharing. I think it would be great to see what the average pct. change of the SP500 was (as a whole, not as an index) to check for general market movement bias. Great to see the comparison of the RNN and random approach. Also I never knew a coffee in the States was 8$!! Outrageous
@ritvikmath5 ай бұрын
Thanks! And appreciate the suggestions.
@VadimChes5 ай бұрын
Maybe you saw some article of guy who says machine learning cannot predict finances. All what we can do is optimize models like Fama and French model to find more important factors affecting stocks prices. Or for example use machine learning to optimize weights of given assets in the portfolio. What do you think? Article with name "Why machine learning can’t predict the prices of stocks or cryptocurrencies: the limits to applying machine learning in empirical asset pricing". Its easy to find I dont want to advertise it. Just wanted to know your thoughts on the subject
@poisonza5 ай бұрын
The Fama and French model is essentially a straightforward application of Ordinary Least Squares (OLS) on factors to analyze a portfolio of asset returns. Since OLS is a machine learning algorithm, I fail to see your point. Perhaps you're skeptical about complex models with too many parameters, suggesting that in essence, the parameters cannot be accurately estimated?
@VadimChes5 ай бұрын
@poisonza you can google the article I mentioned. Its big enough. That's not my opinion. That's the subject of discussion
@poisonza5 ай бұрын
@@VadimChes "All what we can do is optimize models like ..." isnt this your claim? article you suggested says the opposite "Traditional empirical asset pricing models like CAPM and Fama-French have limitations in accurately capturing systematic risks and explaining stock returns. These models may be over-parameterized, mis-specified, or unable to capture non-linear relationships among factors."
@cumibakar105 ай бұрын
It’s true, you cannot “predict stock prices”. Stock prices in an efficient market are semi-martingales: it’s mostly random walk. Why is it random? Because everything that can be predicted has already been predicted and priced into the asset. The price of all assets at any given time already incorporates all available information. That’s why no firm on wall street tries to “predict future prices”. They either sell financial services to companies, provide a market, or other services (or in case of HFT try to arbitrage millisecond differences in prices between markets). This is the danger of data science/machine learning without domain expertise. Or even a basic understanding of the domain.
@gabrielplzdks38915 ай бұрын
@@cumibakar10Right, if I'm not mistaken what quants do at most is manage risk in portfolios by trying to accurately model the components of some commodity rather than predicting any specific future state. If the market is efficient then you can't beat anyone and if it isn't, you aren't the one to find out either.
@Pythoncode-daily4 ай бұрын
Pretty interesting. If you build a transition matrix between the states rising trend - neutral - falling trend, then the probabilities of transitions will be very different from the proposed matrix of transitions between sentiments. The probability of observing a growing trend tomorrow, if we saw a growing trend today, will be about 78%, if it was neutral, then it will remain neutral about 60%, if there was a falling trend, then the falling trend will disappear about 81%. The only problem is with the cost of transition (for example, from a growing market to a neutral market). It can be so high that the mathimatical expectation becomes close to zero, despite such remarkable probabilities.
@decluttercharts3 ай бұрын
Great Video! working on exact model and this helps to validate mine
@notejp-d8wАй бұрын
my sentiment on this video: thumbs up!
@shubh43192 ай бұрын
Great video . Can you do some Multivariate analysis data series with different business case scenarios
@garyc94024 ай бұрын
@ritvikmath I have a trading system that I'm currently adding hidden markov models to. Any chance I can run some ideas by you? It has to do with programmatically detecting which price distribution regime changes using a hidden markov model.
@superman397565 ай бұрын
Can you explain how you traded based off this model specifically? Did you predict sentiment for the next day and sell/buy/hold based on that for a day with a rolling window of 1 ?
@ritvikmath5 ай бұрын
Yes exactly
@saikrishna07165 ай бұрын
Intriguing Content as usual! Btw, how did you come you with the sentiment?
@ritvikmath5 ай бұрын
basically the model outputs hidden states as integers {0,1,2}, each with a corresponding mean for its normal distribution. We map the integers to sentiments via ranking the means.
@rexlaurus58945 ай бұрын
How did you arrive at the transition probabilities for sentiment? Like you said, isnt it hidden? How about the distributions? Would be really helpfull to get the code for these videos. This is cool, but we dont even know how the RNN was implemented.
@ritvikmath5 ай бұрын
So once the HMM is fitted we have 3 hidden states encoded as 0, 1, 2 in the code. We associate the one with the smallest mean as negative sentiment, etc.
@rexlaurus58945 ай бұрын
@@ritvikmath Thank you. That is for the distributions i believe. But how about coming up with the sentiment transition probabilities?
@drdca82635 ай бұрын
@@rexlaurus5894I would imagine you could just do (constrained) gradient descent on the training data?
@petegranneman16233 ай бұрын
Interested concept. How did you estimate transition (and emission) probabilities? Obviously the most likely sequence of hidden states could be inferred with the viterbi algorithm (covered in one of your other videos). I'm assuming you used the baum welch algorithm to estimate state transition probabilities (but this was not explained).
@emmang20105 ай бұрын
great video!
@ritvikmath5 ай бұрын
Thanks!
@VadimChes5 ай бұрын
Do you really stock trade? I think these 1k experiments are just paper experiments, correct? Do you have any specific view on the trading? If you do, what kind of prediction technique would you use considering all possible variants?
@ritvikmath5 ай бұрын
These are actual experiments and I do trade. Of course you can argue why not just run them as paper trading and that’s a totally valid point. It just personally makes me more invested (no pun intended) if I put down real money
@Dani-gs8zf5 ай бұрын
@@ritvikmath And are the fees calculated in these returns?
@ritvikmath5 ай бұрын
Yes any fees are included in the final return calculation
@jamesprivet3 ай бұрын
Very nice! Do you calculate the HMM states and probabilities on the price data or the price return data?
@Sam-tg4iiАй бұрын
Where did the 0.4% for RNN come from suddenly?
@DistortedV125 ай бұрын
How did you train this? How does it map to sentiment if you are just putting Hidden units to 3? As an aside, I think you should start adding jupyter notebooks to your videos like Karpathy
@ritvikmath5 ай бұрын
The model outputs numerical hidden states of 0,1,2 and mu,sigma for each. We map the numerical hidden states to sentiments by ascending mu
@DistortedV125 ай бұрын
"ascending mu?" do you have any sentiment data to validate this on? @@ritvikmath
@user-oj9iz4vb4q2 ай бұрын
0.02% return in a day is 7.5% annual return. Pretty much market index 0.2% daily is a 107% annual return on your investment (you money doubles every year), that's something. 0.4% is something like 330% return per year. There's no way these returns are consistent.
@ritvikmath2 ай бұрын
Definitely not
@lcrocianilcАй бұрын
@@ritvikmath To make things more authentic, I think you should make some backtesting over years. Just to be more clear with your audience 😃
@marcfruchtman94735 ай бұрын
Great explanation. It would be interesting to follow the predictions of the models on a regular basis just to see what happens when the market shifts. Ie, does the model predict that... or more than likely, can not make such predictions. Thanks for the video.
@ritvikmath5 ай бұрын
Great suggestion!
@kab7915 ай бұрын
@@ritvikmathmaybe do a weekly session on the results of previously implemented model. Good idea for useful and interesting content
@anindadatta1642 ай бұрын
Is it possible to predict volatility of an asset using Markov model? As volatility is mean reverting, so should the return distribution curves be narrower?
@andrashorvath24115 ай бұрын
Are you considering commission fees? In my experience that makes the return hard to go to positive at all because the fees make it a negative sum game and so with a negative bias which gives loss on long term with high probability.
@ritvikmath5 ай бұрын
Yes they are included
@andrashorvath24115 ай бұрын
@@ritvikmath Thank you. I recommend you to do a statistical T test on your profit return values (negative and positive values over time) to see if its mean is significantly different from zero to the positive direction. If so, then you know your model has some real power. This helps to see out of the fog of random noise. Also the non normality of the distribution of profits must be corrected to get a better picture. Since the T test converges to be Mean / SE as the number of values grows, you can use this as a simple metric, but I recommend you to use the absolute cubic differences instead of the squared one when calculating SE, this can account for the non normality in a simpler way. I can derive this for you if interested. Keep up the good work. Cheers.
@ekaterinaburakova86295 ай бұрын
@@ritvikmath Amazing, I thought this is a gross return purely from the price diference!
@Ryuuuuuk5 ай бұрын
What about changing the time scale of your model. Currently your "delta t" is 1 day, but what about e.g. the sentiment over the course of a week/ month/ year
@ritvikmath5 ай бұрын
Cool suggestion!
@nuvie695 ай бұрын
How about an EMD (Empirical Mode Decomposition), combined with SVG (Support Vector Regression)? I saw a whitepapaer that said it was able to predit 25 mins ahead. Also, what platform did you use to implement the HMM?
@ritvikmath5 ай бұрын
cool suggestions! I used this framework scikit-learn.sourceforge.net/stable/modules/hmm.html
@xxxs83093 ай бұрын
So, how is it different than technical analysis
@Septumsempra88185 ай бұрын
Triple Barrier Labeling or some of the other work by Prof. de Prado
@vikraal69745 ай бұрын
Does Deep Learning acheive this by default?
@miraclemaxicl5 ай бұрын
can you run a different test on this strategy: buy 5 stock and short 5 stocks ($500 long + $500 short), allocated by historical volatility?
@ritvikmath5 ай бұрын
Interesting idea!
@miraclemaxicl5 ай бұрын
what did S&P500 return on that day?
@Ryuuuuuk5 ай бұрын
I'm living in Italy where a coffee costs less than two dollars so I choked a bit ahaha