My paper, “Limitations of Quantitative Claims About Trading Strategy Evaluation”, is now available for download at SSRN. More details below.
The main idea in the paper is that while the quantitative evaluation of trading strategies is useful, especially in raising awareness about the perils of backtest overfititng and selection bias, it does little to address the more important problem of changing market conditions that practitioners constantly face. See the abstract below:
Abstract
One of the key assumptions of quantitative trading strategy evaluation is that Type II errors (missed discoveries) are preferable to Type I errors (false discoveries.) However, practitioners have known for long that the statistical properties of some genuine trading strategies are often indistinguishable from those of random trading strategies. Therefore, any adjustments of statistics to guard against p-hacking increase Type II error unless the power of the test is high. At the same time, the power of the test is limited by insufficient samples and changing market conditions. Furthermore, genuine strategies with statistical properties that are similar to those of random strategies may overfit due to favorable market conditions but fail when market conditions change. These facts severely limit the effectiveness of quantitative claims about trading strategy evaluation. Practitioners have instead resorted to Monte Carlo simulations and stochastic modeling in an effort to increase the chances of identifying robust trading strategies but these methods also have severe limitations due to changing market conditions, selection bias and data snooping. In this paper we present two examples that demonstrate the limitation of quantitative evaluation of trading strategies and we claim that the most effective way of guarding against overfitting and selection bias is by limiting the applications of backtesting to a class of strategies that employ similar but simple predictors of price. We claim that determining when market conditions change is in many cases fundamentally more important than any quantitative claims about trading strategy evaluation.
Paper URL: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2810170
Any comments or suggestions will be appreciated as this is a working paper.
