This video is pure gold, a gem buried in the immensity of youtube, this is a lecture I would pay for attend to, and it is here totally free!! I don't really understand how it does not have more views.

0:09: 📚 The speaker, Ernie, discusses the application of machine learning in finance and the challenges it faces. - 0:09: Ernie has a background in physics and transitioned to finance, working at firms like Morgan Stanley and Credit Suisse. - 0:22: Ernie is now a managing member of QTSCapital Management and a founder of Predict. - 0:27: Ernie has authored books on quantitative trading and made contributions to the quantum finance community. - 1:44: He discusses the difficulty of extracting value from applying machine learning to finance. - 5:01: Overfitting and the complexity of models are challenges in applying machine learning to finance. - 6:57: Advances in avoiding overfitting have made machine learning more useful in recent years. 7:26: 📊 Machine learning is becoming more popular in finance due to advancements in techniques, such as dropout and random forest, that help overcome overfitting and make the models more transparent and interpretable. - 7:26: Advancements in machine learning techniques, like dropout and random forest, have helped overcome overfitting in finance. - 10:55: Feature selection is a powerful tool in machine learning that ranks and explains the importance of variables, making the models more interpretable. - 13:07: Machine learning models should be primarily used for risk management and capital allocation, rather than as a primary signal generator. 14:17: 📊 Using machine learning to predict the financial market directly is unlikely to succeed due to competition and market evolution, but it can be successful in predicting private information and optimizing trading strategies. - 14:17: Machine learning programs competing for the same predictions in the financial market would have arbitrated away any detectable patterns. - 14:41: Applying machine learning to fields like radiology is different as the target (e.g., cancer) doesn't evolve to avoid detection. - 15:46: Machine learning programs in the financial market are unlikely to beat those used by large firms like Citadel or Google. - 16:18: Using machine learning to predict private information, such as the spread between stocks, can be successful as it is not publicly available. - 17:40: Machine learning can be useful for risk management and capital allocation in trading strategies. - 18:00: Traditional quantitative strategies focus on modeling prices and fundamentals, while machine learning is more suited for alternative data. 20:46: ✨ Machine learning models in trading can handle non-linear dependencies and provide probabilities for trade success, but assessing statistical significance is challenging. - 20:46: Traditional quantitative models cannot handle non-linear dependencies, while machine learning models can. - 21:57: Machine learning models are difficult to replicate, reducing alpha decay. - 22:04: Machine learning models are opaque and difficult to understand, making it less likely for two people to build the same model. - 23:09: Machine learning models provide probabilities for trade success, allowing for better capital allocation. - 23:46: Traditional quantitative models provide buy or sell signals without indicating the likelihood of success, while machine learning models provide probabilities. - 24:05: Assessing statistical significance is challenging in traditional quantitative models, but machine learning models allow for generating multiple backtests to assess significance. 27:43: 📊 The most difficult and time-consuming step in financial machine learning is financial data science, which takes up 80% of the time in constructing a strategy. - 27:43: Financial data science is the most difficult and time-consuming step in financial machine learning. - 28:06: Financial data often contains numerous problems, such as look-ahead bias and parameter adjustments. - 31:23: Processing raw data into useful features requires domain expertise. - 32:32: Machine learning itself is a solved problem and can be easily implemented. - 34:01: Constructing a trading strategy and backtesting it is a crucial step. - 34:20: Assessing the statistical significance of a strategy is necessary. 34:33: 📊 Using machine learning in finance requires human intelligence and expertise in converting predictions into portfolios. - 34:33: The human intelligence part of financial data science is less difficult than the first part, which is financial data analysis. - 35:02: Converting predictions into portfolios requires domain expertise and following standard routines. - 36:26: Most financial data is not stationary and needs to be transformed before using it as a feature in machine learning models. - 37:28: Linear and logistic regression models are commonly used in financial machine learning, while deep learning models are not suitable due to limited data. - 38:18: Predicting the profitability of your own strategy, rather than the market, using meta-labeling can lead to successful machine learning applications in finance. - 39:40: Machine learning models can provide valuable insights and signals for trading decisions, such as detecting market risks and suggesting strategy adjustments. 41:37: 📚 Machine learning in finance requires domain expertise and is not a replacement for human traders. - 41:37: Feature selection is important in explaining investment losses. - 41:48: Machine learning can accurately predict macro variables. - 43:14: Stationarity is necessary for time series models to make accurate predictions. - 45:07: Machine learning models can handle both numerical and categorical data. - 46:33: Machine learning in finance requires deep domain expertise. - 47:55: Worship of deep learning is a common mistake in financial machine learning. 48:25: 🎯 Deep learning may not be effective in solving problems with limited data and directly predicting returns, but it can be useful in generating simulated price years and providing a more nuanced understanding of the market. - 48:25: The blind worship of deep learning can be detrimental to financial machine learning. - 48:42: Deep learning may not succeed in directly predicting returns or competing with well-funded organizations. - 51:02: Capital allocation can be improved by using machine learning to provide a more sophisticated expected return as an input. - 52:50: Traditional quant finance methods provide static probabilities, while machine learning offers dynamic probabilities that vary with the input. - 54:01: Recurrent convolutional neural networks and other deep learning methods can be useful in reducing the need for engineered features in time series data. - 54:18: Deep learning methods may not eliminate the need for engineered features, especially for cross-sectional diversity. 55:15: 💡 Reinforcement learning can work well at high frequency for trading, but there is no convincing evidence that it works in longer time scales. - 55:15: Different kinds of inputs are necessary for successful predictions. - 55:22: Domain expertise is required for making successful predictions, not just deep learning. - 55:49: Reinforcement learning can react to people placing orders on the order book, making it useful for high-frequency trading. - 56:33: There is no convincing evidence that reinforcement learning works in longer time scales. - 56:39: Reinforcement learning has not been successful in longer time scales. - 57:06: Predict Now AI is a no-code financial machine learning tool provided by Dr. Chan's company. Recap by Tammy AI

One of the most insightful videos about ML for trading I've ever watched. Definitely deserve more views.

The MAN Dr Ernest Chan. Thanks for having him. People are really missing out.

Thanks so much to Dr Ernest Chan. A valuable interview.

This is an unbelievably powerful and valuable video. Really great.

great presentation, the great questions from the host. Thank you so much, top-notch content

Just fantastic!! 57 minutes of pure great content. Thanks so much to Dr Ernest Chan and the guys who interviewed him.

The knowledge shared in this video is unimaginable. Grasping it is big task.

Excellent presentation Dr. Chan at approximately minute 31 you make the asseveration that you don't trust anyone, due to the fact that originators can change data. Being that DLT can assure the data integrity we posit that an oracle network running on a private federated DLT could solve this challenge.

Chan has nailed what will differentiate winning ML from losers: The ability to take the input data and apply creative transformation on that input. "Dumb" linear models on raw, widespread and well-known data just won't work (moving avg's, daily % return, etc), because everyone is doing it, and the Big Boys have long since abandoned it because it has stopped working long ago.

great video. i note several of the challenges of financial data science are quite equities specific... just one incentive for considering other asset classes (e.g. futures).

That video was great. So many insights!

Just one question I forget to pose is regarding the cpcv algorithm , do you apply this technique of back-testing for each potential strategy during the development phase in your fund Ernie ? If so what are the outcomes you see till now

Dr. Ernest 👏 well job done

I found one way to avoid the reflexivity problem as a retail investor is to avoid highly liquid big markets. Focus on small assets or assets in foreign countries where the big boys either can't play or isn't worth their effort to play. Liquidity of such assets will be big enough for retail traders but not for institutions, therefore avoid competing with all the PhDs in the institutional quant funds.

Deep Learning on order book (in conjunction with several other variables such as z-scored IV, HV, etc) provides execution edge.

if ML is better to be used for risk management and capital allocation ,then what piece of technology is suggested as the primary signal generator?

I didn't really like Ernest Chan's books too much, but this video is just great

Brilliant, Sir!

What was the other platform he mentioned after QuantConnect around 34:28?

Thanks Chen for the work you have done so far in educating the community , however one thing am quite surprised that you have mentioned is the part of modelling / strategy development process using ML framework which is regarding the uniqueness weighting and sequential bootstrap procedure so in your Experience Ernie do these concepts have marginal benefits in terms of more accuracy of say RF model or higher returns with and without incorporating them ? As I can see you are in favor of metalabeling and feature selection techniques than any other concepts in Marcos laprados book . Your response is highly appreciated

impressive presentation

Great content. Only disagreement is the outright rejection of deep learning. I agree that simply throwing raw data into DL models would not work, but the same is true if you do that with random forest. Feature engineering is crucial. I also don't think the lack of data is the show stopper for DL in trading. There are lots of high frequency market data available nowadays (yes you gotta pay for most of them). Plus data generation/augmentation techniques are widely available. So quantity of data isn't an issue imo, but instead a properly feature engineered dataset is. In addition, there are many other important pre-requisites like noise reduction, regime change detection, etc.. that would be needed to make any model work, DL or traditional ML. Another advancement in DL that is very useful in my opinion is stochastic deep learning model, which can offer the ability to not only make a prediction but also the confidence in the prediction. There are many open-source stochastic DL libraries around, eg. TensorFlow Probability to do that quite easily nowadays.

Great video

Wow 0 comment on this. The knowledge is huge in this video.

.thanks.

Finally, fractional differentiation. Why does nobody mention this

tough quants have such junky internet connection what a shame ! but anyway i wanted to ask if that metalabeling mr Chan talk about is just like hidden makov chains market regimes ?

This guy is epic

Why only ML, not Computational finance ?

2006 it was a fringe activity?! I doubt it, in 2009 when I learned a lot more about it it already felt like an insurmountable wall of established practices so there was no way of getting a piece of the cake.

Imho "capital allocation, not primary signal generation" thing is a BS. It's just a different name for the same thing. Say, I have 3 strategies that normally get allocated $1 each. Then at some moment my ML model suggest to allocate $100 to model #2. I can, of course, call this "capital allocation", but it's in essence a signal generation as well, 'cause the rest of allocations are close to noise. But maybe I misunderstood Ernest and he meant allocations like 0.3, 0.3, 0.4? But hard to believe such high % can be given to strategies that are not expected to work well. Also predicting "whether certain trading parameters will perform well today' seems like, still, indirect prediction of market movement. It can't be other way round, 'cause every trading policy should be based on expected asset price movements. Having a 'certain parameters' filter works 2-ways: it reduces train dataset size for ML (which is bad), but it theoretically allows ML to focus more on particular search space (which is probably good, as the model transitions from jack of all trades, master of none, to master of some). Which force prevails, is a matter of experimentation and asset/TP subtleties.

forwarded to the Reinforcement learning part and have to disagree, RL is way more powerful in my experience than other methods, and I have tried them all

9:30(((, but the general idea is great if, to be honest in algorithm applying!

“Do you REALLY think that your ML algo can beat Renaissance’s ML algo??” Well not with that attitude, Ernie.

BIG BUCKS! :)

34

S/o to Chan 😂 you know it’s some correct mathematical info when a Chinese guy is saying it 😂

so painful

people that can automate trading must be able to make a trade in 5sec.. then and only then you can automate it with ML dont predict the future price .. you need to look more to when to trade and if you buy or sell All the ML gurus that predict the price are fake , because they dont understand the problem and there by never will find the answer.

Alpha decay as new machine learning experts enter the field is real. So the question is - what does Dr Ernest Chan even gain from any of this? 😆

The only thing that I've got from this video is the Chinese speaker repeats many times how difficult and extremely difficult is predicting stocks price and he promotes his web page. This video is purely advertisement, not even a single line of code.

BTC for $75K by end of this year& Control of The Currency is already Decentralised And now the China disruption would simply Decentralise the Mining setup for the better

ml,ai, will never win the game of trading

You guys should join the subreddit: www.reddit.com/r/mltraders/ or visit the Website: mlalgotrading.com for more sources about Machine Learning and/or Algotrading. Both highly recommended as I'm using it daily.