It is easy for trading system developers to be fooled by randomness especially when the underline process that is used for discovering the trading systems is in itself inherently random. Out-of-sample testing is not enough to guarantee statistical significance of any results obtained because large enough output sets will always contain in-sample results that also validate well in out-of-sample by chance alone. A way of minimizing the probability that machine designed trading systems are not random results is through careful and relevant portfolio backtesting.
Cross-validation of data-mining results is the most important step in the process of discovering trading systems but it is also one that is the least understood. If a trading system does not pass an out-of-sample test then it can be discarded but if it does that is not sufficient for it being non-random. I would like to emphasize this point because system developers can become the victims of such misunderstandings. Specifically, if the performance of a trading system validates in the out-of-sample then this means that the system performed well in that specific out-of-sample. This tells us not much about future performance as conditions may change in the future. The term conditions is the key here. The more conditions that the out-of-sample contains from the set of all possible future conditions, the better the performance of the trading system will be in the future after it validates. However, typical out-of-sample periods reflect only a small subset of the conditions that can emerge in the future. One way of mitigating this highly negative effect on trading system development is by the use of a portfolio backtest.
A portfolio backtest involves testing a trading system across multiple markets and calculating some key parameters that determine if the system was overall profitable. This is not a test over multiple out-of-samples but across multiple histories. Thus, it is a very strict test and it can also turn out to be a conservative one. Here, I will present some results from an already developed system for SPY that has tested well in the out-of-sample. The reasons for this is that I would like to demonstrate a new function that will be available in the near future in Price Action Lab that does exactly this and also to show how an already developed system performs when tested across multiple securities.
The 13 SPY patterns discovered by Price Action Lab in the in-sample from 01/29/1993 to 12/31/2008 and then used to build the system are listed in the results output below:
A new tool was added to the Price Action Lab results called “Portfolio Backtest” and shown on the list of tools at the right lower corner. After clicking on the tool, the user has the option of selecting a directory that contains historical data files of any size and number (limited by memory of course). In this case, we select a directory that contains 11 popular ETFs (DBC, DIA, EEM, GLD, IWM, QQQ, SLV, SPY, TLT, USO, XLF). The historical data files contain daily data since inception of the ETFs:
After clicking OK, the program performs all the necessary calculations for each pattern in the results for the portfolio backtest and the original output is changed as follows:
It may be seen that for each pattern the output is changed to show the portfolio backtest win rate P and the corresponding number of trades and profit factor. In this particular case, all the patterns passed the portfolio backtest and maintain a high profit factor in most cases.
The new portfolio backtest function is supplemented by an individual pattern portfolio backtest that provides more information about performance across the different securities. Here are the results for the first pattern in the list:
The individual portfolio backtest function can be used for manual selection of patterns from the results that fulfil certain user-defined objectives.
This new portfolio backtest feature for the search function of Price Action Lab is an addition to the powerful cross-validation functions already available. Specifically, Price Action Lab offers the following cross-validation methods:
1) One-click out-of-sample testing
2) Search for patterns that are profitable across different markets
3) Portfolio backtest
The new featurewill be available as an upgrade during this coming October. As always, the competition is struggling to keep up, as the random system generators they offer will have hard time surviving portfolio backtest cross-validation.
Disclosure: no relevant position at the time of this post.
