We compare the out-of-sample performance of two predictors of price based on the close and open of daily data. We find that for the deterministic data-mining process used in the in-sample and for the markets tested, the out-of-sample performance of the predictor based on open is not robust.
Deterministic data-mining is a process that produces the same results each time it uses the exact same data input. Reproducible processes are very important in finance and especially in trading strategy synthesis because they comply with the requirements of scientific testing and analysis. Note that data-mining algos based on neural networks and genetic algorithms do not in general produce the same results each time they run with the same data due to randomness in initial conditions. One should be skeptical of the results of these algos because market edges are scarce and the presence of a large number of them should be attributed to curve-fitting and selection bias.
Open and close predictors
The open predictor is defined as Y[Oi] and the close predictor is defined as Y[Ci] with i ranging from 1 to 9. This means that the maximum look-back period for the predictors is 9 bars in the data-mining process. These are simple predictors for the purposes of comparing the out-of-sample performance of strategies based on open and close.
Note that the objective of this analysis is not to develop trading strategies but to study the relative performance of open and close predictors. When developing trading strategies out-of-sample validation should be replaced by validation on a portfolio of comparable securities and the whole data history should be used to mine that data. For more details see Chapter 6 of the book Fooled By Technical Analysis.
Data-mining process
We use Price Action Lab for the data-mining of price series. The predictor based on close is already available in the software. For the purpose of this analysis we modified the software to implement also a predictor based on the open. Note that Price Action Lab offers a wide variety of predictors to use in strategy synthesis and the ones based on close and open are of the simpler ones used for the purpose of this analysis.
In-sample and out-of-sample data
Data from inception of SPY and DIA are divided in an in-sample and out-of-sample segment as follows:
| SPY | DIA |
| In-sample: 01/29/1993 – 12/31/2008 Out-of-sample 01/02/2009 – 06/07/2016 |
In-sample: 01/29/1998 – 12/31/2008 Out-of-sample 01/02/2009 – 06/07/2016 |
Price Action Lab in-sample workspace setups for the close predictor
The minimum win rate for the patterns identified by Price Action Lab is set at 85%. The profit target is set at 2% because we would like to avoid as much as possible fitting exits to the data. The stop-loss is set at 4%. The minimum profit factor is set at 1.5 per pattern identified in the in-sample. We also require that each pattern in the in-sample has more than 20 trades and no more than 7 consecutive losers. Below are the two workspaces for the in-sample data-mining (click on images to enlarge):
 |
 |
Note that patterns based on the close predictor enter a position at the close of their formation for the purpose of this study.
In-sample results for the close predictor
The in-sample results for the close predictor in SPY and DIA are shown below (click on images to enlarge):
 |
 |
Each line on the results corresponds to a price pattern that satisfies the performance parameters specified by the user. Trade on is the entry point, in this case the Open of next bar. P is the success rate of the pattern, PF is the profit factor, Trades is the number of historical trades, CL is the maximum number of consecutive losers, Type is LONG for long patterns and SHORT for short patterns, Target is the profit target, Stop is the stop-loss and C indicates % or points for the exits, in this case it is %. Last Date and First Date are the last and first date in the historical data file.
Price Action Lab identified 31 patterns in SPY, 27 long and 4 short, and 18 patterns in DIA, 16 long and 2 short, that fulfilled the performance criteria defined on the corresponding workspaces.
Out-of-sample performance of the close predictor
The out-of-sample performance of the results is obtained after generating strategy code for Amibroker. The long and short patterns are combined with the OR Boolean operator in the strategy. All backtests are based on $100K initial capital and commission of $0.01 per share. The two equity curves for SPY and DIA in the out-of-sample are shown below. The first pane shows the buy and hold equity curve and the second pane the system equity curve (click on images to enlarge):
 Close predictor SPY |
 Close Predictor DIA |
The annualized return for the SPY strategy is 3.23% and for the DIA strategy it is 4.31%. In both cases the predictor based on the close generated positive results in the out-of-sample.
Next, we repeat the above procedure for the open predictor
Price Action Lab in-sample workspace setups for the open predictor
The minimum win rate for the patterns identified by Price Action Lab is set at 85%. The profit target is set at 2% because we would like to avoid as much as possible fitting exits to the data. The stop-loss is set at 4%. The minimum profit factor is set at 1.5 per pattern identified in the in-sample. We also require that each pattern in the in-sample has more than 20 trades and no more than 7 consecutive losers. Below are the two workspaces for the in-sample data-mining (click on images to enlarge):
 |
 |
Note that patterns based on the open predictor enter a position at the open following their formation for the purpose of this study.
In-sample results for the open predictor
The in-sample results for the open predictor in SPY and DIA are shown below (click on images to enlarge):
 |
 |
Each line on the results corresponds to a price pattern that satisfies the performance parameters specified by the user. Trade on is the entry point, in this case the Open of next bar. P is the success rate of the pattern, PF is the profit factor, Trades is the number of historical trades, CL is the maximum number of consecutive losers, Type is LONG for long patterns and SHORT for short patterns, Target is the profit target, Stop is the stop-loss and C indicates % or points for the exits, in this case it is %. Last Date and First Date are the last and first date in the historical data file.
Price Action lab identified 17 patterns in SPY, 12 long and 5 short, and 9 patterns in DIA, 6 long and 3 short, that fulfilled the performance criteria defined on the corresponding workspaces.
Out-of-sample performance of the open predictor
The out-of-sample performance of the results is obtained after generating strategy code for Amibroker. The long and short patterns are combined with the OR Boolean operator in the strategy. All backtests are based on $100K initial capital and commission of $0.01 per share. The two equity curves for SPY and DIA in the out-of-sample are shown below. The first pane shows the buy and hold equity curve and the second pane the system equity curve (click on images to enlarge):
 Open predictor SPY |
 Open predictor DIA |
The annualized return for the SPY strategy is 1.34% and for the DIA strategy it is -0.19%.
It may be seen that out-of-sample results when using the open predictor are inferior when compared to those of the close predictor. Although this study is far from conclusive, it nevertheless provides an initial indication that the open of daily data has much less predictive power than the close. Below are the results for out-of-sample annualized return for the two predictors of price for three other popular ETFs:
| ETF | Close predictor | Open Predictor |
| XLE | +1% | -7.3% |
| IWM | +3.4% | -4.4% |
| XLU | +3% | -1% |
The above results also confirm the initial findings using SPY and DIA ETFs. Therefore, the predictor based on the open can be declared non-robust from these results.
Finally, the value of the above analysis lies in determining the robustness of a predictor used in a data-mining process before developing a strategy. Contrary to common belief, the value of out-of-sample validation is only in determining the robustness of a data-mining process and not of the strategies it generates. Strategies can fail because of many reasons, the most important being dues to a major shift in market conditions. Out-of-sample testing cannot guard against such shifts but can only provide an indication about the robustness of the data-mining process and of the predictors used. A better method of validation in place of out-of-sample testing is described in more detail in Chapter 6 of the book Fooled By Technical Analysis.
Subscribe via RSS or Email, or follow us on Twitter.
Charting and backtesting program: Amibroker
Disclaimer
Technical and quantitative analysis of Dow-30 stocks and 30 popular ETFs is included in our Weekly Premium Report. Market signals for longer-term traders are offered by our premium Market Signals service.
