I Day Traded $1000 with the Hidden Markov Model

No video

I Day Traded $1000 with the Hidden Markov Model

Рет қаралды 15,346

ritvikmath

Күн бұрын

Пікірлер: 76

@hu5116 5 ай бұрын

Great Video! However, there appear to be a number of missing steps. Maybe that was in a prior video but you never said so. For example, where did the transition matrix come from? You mention very briefly about from past data, but never connected what that means or how it’s done. Similar for rest of the process. Maybe a follow up “show me” video would be in order.

@humanparaquat69 5 ай бұрын

We love your videos dude!

@ritvikmath 5 ай бұрын

Love you all 🙏

@ekaterinaburakova8629 5 ай бұрын

Awesome series, keep up the good work!

@ritvikmath 5 ай бұрын

Thanks, will do!

@nathanlewis42 5 ай бұрын

Is the code for this in your KZfaq repo on github?

@poisonza 5 ай бұрын

can you do a notebook demo on how to fit hmm and do inference on out-of-sample data? maybe include these - fit returns on other distribution other than gaussian (single, multi-variate case) - set initial state transition matrix ; like higher value to diagonal entries (since regimes rarely changes) - show us how the input(length of observation) might change the inference value

@ritvikmath 5 ай бұрын

Good suggestion!

@gsm7490 5 ай бұрын

Really great content! If you don’t mind, may be there is another way to use ML and NN in financial market: you can not only predict future returns and prices, you can also predict how different factors affect price fluctuations right at the moment?

@ResilientFighter 5 ай бұрын

Thanks Ritvik! Great content as usual

@ritvikmath 5 ай бұрын

My pleasure!

@posthocprior 5 ай бұрын

Maybe I've missed something: what exactly is the latent variable you're trying to approximate? If, as you've described, the sentiment values of positive, negative, and neutral correspond to distinct normal distributions with respective means, then this is simply a linear model. It would be hidden if the three sentiment values corresponded to the three, respective distributions and the latent variable was the stock price.

@Gingeey23 5 ай бұрын

Great video - thanks for sharing. I think it would be great to see what the average pct. change of the SP500 was (as a whole, not as an index) to check for general market movement bias. Great to see the comparison of the RNN and random approach. Also I never knew a coffee in the States was 8$!! Outrageous

@ritvikmath 5 ай бұрын

Thanks! And appreciate the suggestions.

@VadimChes 5 ай бұрын

Maybe you saw some article of guy who says machine learning cannot predict finances. All what we can do is optimize models like Fama and French model to find more important factors affecting stocks prices. Or for example use machine learning to optimize weights of given assets in the portfolio. What do you think? Article with name "Why machine learning can’t predict the prices of stocks or cryptocurrencies: the limits to applying machine learning in empirical asset pricing". Its easy to find I dont want to advertise it. Just wanted to know your thoughts on the subject

@poisonza 5 ай бұрын

The Fama and French model is essentially a straightforward application of Ordinary Least Squares (OLS) on factors to analyze a portfolio of asset returns. Since OLS is a machine learning algorithm, I fail to see your point. Perhaps you're skeptical about complex models with too many parameters, suggesting that in essence, the parameters cannot be accurately estimated?

@VadimChes 5 ай бұрын

@poisonza you can google the article I mentioned. Its big enough. That's not my opinion. That's the subject of discussion

@poisonza 5 ай бұрын

@@VadimChes "All what we can do is optimize models like ..." isnt this your claim? article you suggested says the opposite "Traditional empirical asset pricing models like CAPM and Fama-French have limitations in accurately capturing systematic risks and explaining stock returns. These models may be over-parameterized, mis-specified, or unable to capture non-linear relationships among factors."

@cumibakar10 5 ай бұрын

It’s true, you cannot “predict stock prices”. Stock prices in an efficient market are semi-martingales: it’s mostly random walk. Why is it random? Because everything that can be predicted has already been predicted and priced into the asset. The price of all assets at any given time already incorporates all available information. That’s why no firm on wall street tries to “predict future prices”. They either sell financial services to companies, provide a market, or other services (or in case of HFT try to arbitrage millisecond differences in prices between markets). This is the danger of data science/machine learning without domain expertise. Or even a basic understanding of the domain.

@gabrielplzdks3891 5 ай бұрын

@@cumibakar10Right, if I'm not mistaken what quants do at most is manage risk in portfolios by trying to accurately model the components of some commodity rather than predicting any specific future state. If the market is efficient then you can't beat anyone and if it isn't, you aren't the one to find out either.

@Pythoncode-daily 4 ай бұрын

Pretty interesting. If you build a transition matrix between the states rising trend - neutral - falling trend, then the probabilities of transitions will be very different from the proposed matrix of transitions between sentiments. The probability of observing a growing trend tomorrow, if we saw a growing trend today, will be about 78%, if it was neutral, then it will remain neutral about 60%, if there was a falling trend, then the falling trend will disappear about 81%. The only problem is with the cost of transition (for example, from a growing market to a neutral market). It can be so high that the mathimatical expectation becomes close to zero, despite such remarkable probabilities.

@decluttercharts 3 ай бұрын

Great Video! working on exact model and this helps to validate mine

@notejp-d8w Ай бұрын

my sentiment on this video: thumbs up!

@shubh4319 2 ай бұрын

Great video . Can you do some Multivariate analysis data series with different business case scenarios

@garyc9402 4 ай бұрын

@ritvikmath I have a trading system that I'm currently adding hidden markov models to. Any chance I can run some ideas by you? It has to do with programmatically detecting which price distribution regime changes using a hidden markov model.

@superman39756 5 ай бұрын

Can you explain how you traded based off this model specifically? Did you predict sentiment for the next day and sell/buy/hold based on that for a day with a rolling window of 1 ?

@ritvikmath 5 ай бұрын

Yes exactly

@saikrishna0716 5 ай бұрын

Intriguing Content as usual! Btw, how did you come you with the sentiment?

@ritvikmath 5 ай бұрын

basically the model outputs hidden states as integers {0,1,2}, each with a corresponding mean for its normal distribution. We map the integers to sentiments via ranking the means.

@rexlaurus5894 5 ай бұрын

How did you arrive at the transition probabilities for sentiment? Like you said, isnt it hidden? How about the distributions? Would be really helpfull to get the code for these videos. This is cool, but we dont even know how the RNN was implemented.

@ritvikmath 5 ай бұрын

So once the HMM is fitted we have 3 hidden states encoded as 0, 1, 2 in the code. We associate the one with the smallest mean as negative sentiment, etc.

@rexlaurus5894 5 ай бұрын

@@ritvikmath Thank you. That is for the distributions i believe. But how about coming up with the sentiment transition probabilities?

@drdca8263 5 ай бұрын

@@rexlaurus5894I would imagine you could just do (constrained) gradient descent on the training data?

@petegranneman1623 3 ай бұрын

Interested concept. How did you estimate transition (and emission) probabilities? Obviously the most likely sequence of hidden states could be inferred with the viterbi algorithm (covered in one of your other videos). I'm assuming you used the baum welch algorithm to estimate state transition probabilities (but this was not explained).

@emmang2010 5 ай бұрын

great video!

@ritvikmath 5 ай бұрын

Thanks!

@VadimChes 5 ай бұрын

Do you really stock trade? I think these 1k experiments are just paper experiments, correct? Do you have any specific view on the trading? If you do, what kind of prediction technique would you use considering all possible variants?

@ritvikmath 5 ай бұрын

These are actual experiments and I do trade. Of course you can argue why not just run them as paper trading and that’s a totally valid point. It just personally makes me more invested (no pun intended) if I put down real money

@Dani-gs8zf 5 ай бұрын

@@ritvikmath And are the fees calculated in these returns?

@ritvikmath 5 ай бұрын

Yes any fees are included in the final return calculation

@jamesprivet 3 ай бұрын

Very nice! Do you calculate the HMM states and probabilities on the price data or the price return data?

@Sam-tg4ii Ай бұрын

Where did the 0.4% for RNN come from suddenly?

@DistortedV12 5 ай бұрын

How did you train this? How does it map to sentiment if you are just putting Hidden units to 3? As an aside, I think you should start adding jupyter notebooks to your videos like Karpathy

@ritvikmath 5 ай бұрын

The model outputs numerical hidden states of 0,1,2 and mu,sigma for each. We map the numerical hidden states to sentiments by ascending mu

@DistortedV12 5 ай бұрын

"ascending mu?" do you have any sentiment data to validate this on? @@ritvikmath

@user-oj9iz4vb4q 2 ай бұрын

0.02% return in a day is 7.5% annual return. Pretty much market index 0.2% daily is a 107% annual return on your investment (you money doubles every year), that's something. 0.4% is something like 330% return per year. There's no way these returns are consistent.

@ritvikmath 2 ай бұрын

Definitely not

@lcrocianilc Ай бұрын

@@ritvikmath To make things more authentic, I think you should make some backtesting over years. Just to be more clear with your audience 😃

@marcfruchtman9473 5 ай бұрын

Great explanation. It would be interesting to follow the predictions of the models on a regular basis just to see what happens when the market shifts. Ie, does the model predict that... or more than likely, can not make such predictions. Thanks for the video.

@ritvikmath 5 ай бұрын

Great suggestion!

@kab791 5 ай бұрын

@@ritvikmathmaybe do a weekly session on the results of previously implemented model. Good idea for useful and interesting content

@anindadatta164 2 ай бұрын

Is it possible to predict volatility of an asset using Markov model? As volatility is mean reverting, so should the return distribution curves be narrower?

@andrashorvath2411 5 ай бұрын

Are you considering commission fees? In my experience that makes the return hard to go to positive at all because the fees make it a negative sum game and so with a negative bias which gives loss on long term with high probability.

@ritvikmath 5 ай бұрын

Yes they are included

@andrashorvath2411 5 ай бұрын

@@ritvikmath Thank you. I recommend you to do a statistical T test on your profit return values (negative and positive values over time) to see if its mean is significantly different from zero to the positive direction. If so, then you know your model has some real power. This helps to see out of the fog of random noise. Also the non normality of the distribution of profits must be corrected to get a better picture. Since the T test converges to be Mean / SE as the number of values grows, you can use this as a simple metric, but I recommend you to use the absolute cubic differences instead of the squared one when calculating SE, this can account for the non normality in a simpler way. I can derive this for you if interested. Keep up the good work. Cheers.

@ekaterinaburakova8629 5 ай бұрын

@@ritvikmath Amazing, I thought this is a gross return purely from the price diference!

@Ryuuuuuk 5 ай бұрын

What about changing the time scale of your model. Currently your "delta t" is 1 day, but what about e.g. the sentiment over the course of a week/ month/ year

@ritvikmath 5 ай бұрын

Cool suggestion!

@nuvie69 5 ай бұрын

How about an EMD (Empirical Mode Decomposition), combined with SVG (Support Vector Regression)? I saw a whitepapaer that said it was able to predit 25 mins ahead. Also, what platform did you use to implement the HMM?

@ritvikmath 5 ай бұрын

cool suggestions! I used this framework scikit-learn.sourceforge.net/stable/modules/hmm.html