In a recent podcast, Clifford Asness, who cofounded AQR Capital Management, mentioned three types of backtests, all quants should be aware of.
The podcast was one of the best in quantitative finance and I highly recommend it. I will not say when in the podcast Dr. Asness mentioned the three types of backtests because I believe everyone should carefully listen from the start. It is a long podcast, but in reality, it is a free course in state-of-the-art quantitative finance.
The three types of (profitable) backtest according to Dr. Asness:
- A result of torturing data.
- Compensation for actual, rational risk.
- People make errors due to behavioral biases.
1. A result of torturing the data
We have talked about torturing data numerous times in this blog. Nowadays, this is the easiest thing to do with machine learning. These otherwise profitable backtests are noise with high probability (Type-I errors), even if validation or cross-validation schemes are used. The main reason for that is not only the reuse of data but also selection bias: after testing thousands or even millions of strategies, directly or indirectly, a few of them will look good even out-of-sample. See this article for more details.
More importantly, and even paradoxically, quants usually think that by trying harder with backtesting they have a chance of finding something profitable. In reality, it is exactly the opposite of what they expect that happens: as the number of backtest trials increases, the probability of finding a spurious strategy goes to 1. See this article for more details.
The solution for avoiding Type-I errors (false positives) is to start with a sound and unique hypothesis about the markets and the behavior of their participants. These are 2 and 3 below.
2. Compensation for actual, rational risk
I believe mean-reversion, or what some people refer to as convergence, is an example of compensation for actual, rational risk. Below is an example of our B2S2 long-only mean-reversion strategy for Dow 30 stocks. The backtest takes into account delistings, thanks to Norgate Data.
It’s amazing how this strategy has provided a (hypothetical) large alpha by trading long-only Dow 30 stocks. The strategy is simple, but this is exactly the point: anything complicated is probably the outcome of trying too hard and torturing the data. This strategy shows excellent backtests because it is based on the purest form of mean reversion and there are no filters of any kind to protect against corrections and bear markets. This is an example of real, rational risk. It is foolish to allocate a lot to a strategy that does not use stop-loss orders or any other type of protection in case of a left-tail event. Therefore, the allocation must be kept low. Still, this type of strategy can play a role in a diversified portfolio.
3. People make errors due to behavioral biases
Herd behavior, chasing trends despite valuation levels, and irrational exuberance can lead to the formation of long-term trends in the markets. Trend-followers and momentum traders get compensated for exploiting these biases. Below is an example of our PSI5 algorithm in divergent mode (trend-following) trading 23 major futures contracts.
There is significant (hypothetical) alpha here, but it comes from patiently exploiting the behavioral biases of traders and investors. This is not an easy strategy to trade, and the attrition rate is very high. The strategy is fundamentally simple, as it exploits the dynamics of markets as described by a simple formula from a probability textbook. The hard part is the patience to follow the trends, and the risk and money management.
Summary
Too much backtesting is a dangerous practice. Winning strategies are fundamentally simple but exploit either rational risk compensation or behavioral biases, as Dr. Asness explained. In most cases, complicated strategies that are not based on an idea with economic value are the outcome of torturing the data.
Premium Content
Online Books
Premium Articles
Systematic Market Signals
By subscribing you have immediate access to hundreds of articles. Premium Articles subscribers have immediate access to more than two hundred articles and All in One subscribers have access to all premium articles, books, premium insights, and market signals content.
Free Book
Subscribe for free notifications of new posts and updates from the Price Action Lab Blog and receive a PDF of the book “Profitability and Systematic Trading” (Wiley, 2008) free of charge.
Disclaimer: No part of the analysis in this blog constitutes a trade recommendation. The past performance of any trading system or methodology is not necessarily indicative of future results. Read the full disclaimer here.
Charting and backtesting program: Amibroker. Data provider: Norgate Data
If you found this article interesting, you may follow this blog via RSS, Email, or Twitter.
