Validation of trading strategies is important for minimizing probability of Type-I error, or false discoveries. Below is an example of how to quickly validate strategies developed with DLPAL S on correlated but more importantly, anti-correlated securities.
Many trading strategy developers do not realize that some validation methods are ineffective when they become part of the development process. An example is out-of-sample testing when it is repeatedly used to validate data-mined strategies or unique hypotheses that are adjusted after validation. The reason for the failure is this: if we use out-of-sample backtests to adjust a strategy or to develop a new one, then the probability of Type-I error increases because the out-of-sample is essentially part of the in-sample and cannot offer an unbiased estimate of future performance.
Some trading strategy developers understand the limitations of out-of-sample testing and use alternative methods. A popular alternative is to test the strategy on other securities. However, this validation test can also be abused and rendered ineffective and actually this is a widespread practice, especially by strategy vendors.
About a week ago I saw an article about a mean-reversion strategy for SPY with exceptional returns accompanied by claims that the strategy works well on a number of other securities. Besides that fact that a good part of the performance was achieved during two years and returns were very low in most other years, the validation was performed on highly correlated securities. A better test is to try an anti-correlated security. In the case of SPY, one could use TLT but here is the catch:
The validation on an anti-correlated security must be done on unseen data
An example
Below is an example of a strategy that was developed with DLPAL S (actually an earlier version of the program) in April 2013 using in sample DIA data from 01//20/1998 to 12/31/2009. The results of the search for high profitability strategies based on parameter-less price patterns are shown below:
DLPAL S identified eight long and two short strategies in DIA with 4% profit target and 3% stop loss. Trades are for the open of next day.
Below is the out-of-sample performance from the end of the in-sample to development date and from the end of the out-of-sample to the end of last year with code generated by DLPAL S for Amibroker that combines the 10 strategies. Starting capital is $100K and commission is $0.01/share. Equity is fully invested.
This particular strategy was a success and has outperformed SPY in out and forward samples. This is also shown from a Quantopian backtest with 1-minute data:
Total return of the strategy since the end of the in-sample to the end of 2017 is 190.6% versus 179.8% for SPY buy and hold.
But how was this strategy (DIAT4S3) selected and the probability that it will be a success maximized?
This is a fundamental mistake in strategy development: some try hard to generate strategies in markets and timeframes, or combinations thereof, which have high noise.
A prudent approach is to take what some markets give you rather than trying to force development on markets that have little to offer.
It took me many years to realize this and I share this heuristic with the readers of this blog. I believe this is the first step to success even before risk management.
The DIAT4S3 strategy was validated on TLT, an anti-correlated security, on the out-of-sample, not in the in-sample as some developers do. Below is a chart of the 0-lag, 120-day correlation of DIA with SPY and TLT.
It may be seen that the average correlation between DIA and SPY in the combined out-of-sample/forward sample is 0.962, which is as expected very high. But the same correlation between DIA and TLT is -0.42, which denotes anti-correlation.
How to test strategies on different securities in DLPAL S
This is very easy and efficient in DLPAL S because there is the function “Change File Name” in the results. Using this function we can change the symbol DIA to any other symbol we want and then use “Test Strategies” to backtest in any data sample.
Below are the backtest results after changing DIA to SPY:
It may be seen that the strategy has been profitable in SPY with CAGR of about 5%. Again, this is performance in the combined out-of-sample/forward sample.
However, SPY is highly correlated to DIA. The results do not constitute evidence of a robust strategy because if the original strategy in DIA was an accidental fluke in the out-of-sample, so will probably be the strategy when applied to SPY.
Below are the backtests results after changing the symbol to TLT:
This is a much better test and result since performance is positive with about 5.5% CAGR in the combined out-of-sample/forward sample. This last test offers an indication of a significant strategy but the previous test on SPY does not.
A few remarks
Validating on anti-correlated securities and out-of-sample is a strict test and the probability of a Type-II error (missed discovery) is higher but it offers better protection against flukes.
There are some but not many edges pertinent to a specific security that will fail validation tests on comparable and especially anti-correlated securities. However, these edges are hard to identify and they also tend to disappear fast, especially if they are due to seasonality anomalies.
In general, validation is more of an art than a science. The logic used to justify a specific validation test must be sound to start with.
In the next article we will discuss how to use portfolio backtest capability of DLPAL S to validate strategies.
Click here for a demo version of DLPAL S. For more articles click here.
If you have any questions or comments, happy to connect on Twitter: @priceactionlab
Subscribe via RSS or Email, or follow us on Twitter
Disclaimer: No part of the analysis in this blog constitutes a trade recommendation. The past performance of any trading system or methodology is not necessarily indicative of future results. Read the full disclaimer here.
