Attempting to enhance outdated investing or trading strategies and ideas that have not yielded expected results is rarely successful due to data snooping.
It is tempting—so tempting that even people who understand the dangers of data snooping cannot escape it.
Data snooping refers to decisions made during the development and analysis stages based on prior knowledge of the price series. For example, some strategies suffered a large drawdown during the bear market of 2022, and some developers decided to add an indicator, or even new assets to allocations, to minimize the equity curve volatility and drawdown.
In other words, the strategy developers decided what to add after looking at the data and the strategy’s performance.
The developer believes that the addition of another indicator or asset results in a new strategy. There are some issues with these notions, other than technical ones about statistical significance based on prior knowledge of the data set.
1) While the addition of a new indicator typically improves performance during recent periods, it often negatively impacts performance during other periods.
2) Due to the non-stationary nature of price series and dynamic correlations, the new indicator or assets may be either insufficient or detrimental to performance in the future.
3) More importantly, adding new indicators, conditions, or new assets to an already existing strategy increases the parameter optimization space and results in a higher dispersion of performance in future paths.
In other words, making improvements to existing strategies by adding conditions or indicators increases selection and survivorship bias and could result in a higher probability of underperformance or even failure in the future under different market conditions. Let us look at an example of a popular strategic allocation known as the All Weather Portfolio, from 2007 to 2021.
Although there may be several different versions of this portfolio, this is the one that invests 40% in TLT, 15% in IEF, and 30% in VTI, 7.5% in IAU, and 7.5% in DBC. There is annual rebalancing in the portfolio.
Up until the end of 2021, this portfolio appeared to be a reliable investment, boasting an 8% annualized return, a maximum drawdown of 14.4%, and a Sharpe ratio marginally above 1. The beta was only 0.14. Leverage this 2x and you have a respectable alpha. But who could have thought of a rare regime where both stocks and bonds became highly correlated and fell? This is what happened after 2021.
The loss in 2022 was more than 19%. “It was about time,” a savvy quant would argue. The annualized return plunged to 6.5%, and Sharpe fell to 0.80.
In retrospect, we found that this straightforward allocation, which was based on historical data, did not work well in the new regime. Now that we know the regime, can we fix the strategy by adding some indicators or new assets?
Fixing this portfolio without transitioning to a tactical allocation could be challenging. Next time, it’s possible that stocks, commodities, and gold will become highly correlated and fall together. We could add filters, but the correlation typically occurs with a larger lag, transforming a simple “allocation dream” into a tactical one. Who wants to be tactical? Everyone wants to play golf or travel while some asset allocation prints money, and there are no tax issues until the final exit. But this “dream state” has probably come to an end.
“I will change the allocation to TLT to 10% and increase the allocation to stocks to 60%,” someone from the golf course said. This is the point where data snooping is in full swing.
After doing that, the 2008 return changed from a +3.2% to a -18.1%! In other words, the issues with this allocation shifted from the recent past to the distant past, but the 2022 negative performance did not improve significantly.
As previously mentioned, we have the ability to incorporate tactical elements into All Weather; however, this shift from a strategic approach to a tactical one may not make sense to many individuals.
Trading strategies
Similar considerations as above apply to trading strategies. When a strategy fails or does not perform as expected, it is better to discard it and focus the effort on finding something new.
Finding something new is not easy, as there is a shortage of genuine edges. However, trying to improve something that has been developed in the past suffers from dangerous biases, and the probability of failure is high.
Another choice is to extract strategies from the data, but this suffers from data mining bias. Although I developed software for this purpose in the past and was a pioneer in the field, quantitative easing and high deficits have highly distorted trading dynamics and the behavior of time series, along with the advent of thematic ETFs.
Conclusion
I do not recommend trying to improve old strategies that have failed or do not perform as expected. On the other hand, the task of identifying new, sound ideas is becoming increasingly challenging. Trading and investing are challenging and are bound to become harder, especially with the advent of artificial intelligence.
Disclaimer: No part of the analysis in this blog constitutes a trade recommendation. The past performance of any trading system or methodology is not necessarily indicative of future results. Read the full disclaimer here.
Charting and backtesting program: Amibroker. Data provider: Norgate Data
If you found this article interesting, you may follow this blog via RSS or Email, or on Twitter
