Why ML Fails at Stock/Crypto Prediction

The use of ML (machine learning) has been on the rise for the past 10 years. The idea of machine learning has been a trend for quite a while now and many people coming from different domains try to grasp the topic in hopes for a better opportunity career-wise and financial-wise.

From my personal view, there has been lots of side-projects made by newcomers in the field, of analysis that involves stock price prediction or crypto price prediction. Don’t get me wrong, I’m not against the idea of using ML for these use-cases, but I just want to point out how traditional ML techniques are misused a lot for this specific domain.

Beyond price and time: Missing market signals

Almost all (if not all) of the side projects I’ve seen that tried to predict stock/crypto price settled with a open/close price through time with some added lagged features to predict future prices of stock/crypto. As some who’s been a crypto trader for more than 5 years now, and a data scientist for more than 3 years, I knew that there are a lot more factors affecting crypto prices. Some of these factors involves market structured factors, technical, adoption, regulatory, and market sentiment factors.

Honestly speaking, there are more factors than what I have listed out and I knew that making a model out of all these factors as features or parameters for your ML will only make it better and more complex. If you could have a model that captures all these features well (if you have a way of representing these factors well enough) then maybe, you really have a shot at actually predicting the future stock/crypto price.

Trading against Quant Traders, Hedge Funds, and Banks

I knew that using a very simple model to predict stock/crypto prices would be disrespectful to the years of domain experience professional traders have when it comes to trading. They knew factors that we didn’t know exist that is a key thing for predicting stock/crypto prices.

I’m not saying that we can’t beat professional traders through traditional ML techniques in building a stock/crypto price prediction model, maybe we can build a better one, but at the end of the day these guys have the ‘data advantage’. Being in data science for quite a while now, I knew that nothing beats a good and clean data that you can use for modelling. My personal long term goal was to also build an algorithm that could aid in my trading journey. I knew that it’s a long and continuous process, but it’s something I’m willing to be a part of.

Conclusion

The reality is that when you are trading, you are going against institutions with years of domain experience that has helped develop a superb market intuition, and an unending pocket of money to buy whatever data advantage they would need in order to create a better model. The gap between side projects to predict stock/crypto prices and professional trading systems is beyond most people's expectations. This doesn’t entirely mean that developing your own model for stock/crypto price prediction as an individual or ‘retail’ won’t ever work, it just means that there is room for improvement and more chances in making whatever model you have to be better. In a game where your opponents have a billion dollar budget and decades of experience, it is good to have a little humility that will push you to try and learn more. That is a lesson worth learning before risking your capital before trying to use your model.