We have received complaints from blog readers about backtests in some websites with inflated returns that indicate deliberate choice of starting date for analysis. This is a common source of data snooping bias in the trading literature.
Data snooping occurs when analysis is made after looking at the data and making deliberate choice of rules for obtaining better performance. This form of bias is of the worst kind and leads with probability near 1 to strategies that fail in the future.
Data snooping bias is exceptional harmful when knowledge of the past is used to implement certain filters that are known to have avoided bear markets.
The 50/200 moving average golden cross in S&P 500 (SPY) is essentially such a filter that is often used with other strategies to generate good looking backtests. However, this is data-snooping bias.
A frequent deliberate choice of starting date for backtests that works well with data snooping and filters is 01/03/2007, just before the start of the 2008 bear market.
Below is an example of the long-only 50/200 moving average cross in SPY with a backtest that starts on 01/03/2017 and ends 11/04/2019.
It may be seen that the equity curve is much smoother than the buy and hold curve (blue line) and the total return is about the same. But this is only due to data snooping bias and our knowledge of the past to start the backtest in 2007.
This is what happens if we start the backtest in 2010.
In this case there is a wide divergence between the strategy and buy and hold return (blue line.)
The argument is that the filter will help the system to avoid a bear market in the future. We do not know that for sure. It will depend on how fast the bear market evolves. Moving averages have a long lag in adjusting to price action. Here is what happens if we start the backtest in October 2018, just before a 20% correction.
The strategy was losing money until recently and it is now flat only due to the rally to new all-time highs, while buy and hold recovered fast and stayed in the black for almost the whole year.
Summary
Often, starting a backtest on a convenient date cannot be avoided especially when trading ETFs that have short history. But choice of starting date before a bear market may result in questionable analysis especially if deliberate choice of filters is made that are known to have avoided the drawdown.
Next time you see a backtest starting just before a bear market you may want to ask the developer if they use a bear market avoidance filter. Although there is nothing wrong with that in principle, ask them to start the backtest a few years after that and compare the results to the benchmark. If the divergence with previous backtest is high, the results may be due to data snooping bias. More work may be required to establish that.
In another article we will look at how these deliberate choices of bear market filters affect results of strategies for equity portfolios.
Charting and backtesting program: Amibroker
Data provider: Norgate Data
Technical and quantitative analysis of major stock indexes and 34 popular ETFs are included in our Weekly Premium Reports. Market signals for position traders are offered by our premium Market Signals service
If you found this article interesting, you may follow this blog via RSS or Email, or in Twitter
