This article introduces the process of Walk-Forward Trading System Synthesis™. I show with an example that in the case of price patterns this process may not offer any advantages over the traditional Trading System Synthesis process that uses the whole available in-sample of data. This is to be expected because with price patterns there is no optimization of entry signal timing and as a result curve-fitting is kept to a minimum.
Trading System Synthesis (TSS) and Forward Synthesis of Trading Systems™ (FSTS) are concepts I have introduced in my books and articles going back at least 12 years (See references at the end of this article). Thus, I am glad to see that some notable trading system developers have invented similar or same terminology lately. I have to admit that I am not expecting to receive any credit for my early introductions of these concepts as this rarely happens in this field.
Price Action Lab (PAL) is a software program based on the above two concepts, TSS and FSTS. PAL synthesizes trading systems from price action based on user-defined criteria. The traditional process of TSS involves the use of an in-sample of data for developing the systems and then an out-of-sample for cross-validation. However, a couple of important issues that arise here regard the relative proportions of the in-sample and the out-of-sample with respect to the original data sample and whether old data or only more recent data are more relevant to the synthesis process. Opinions on these issues vary widely as with any other issue about trading, especially for the latter one. Some well-known traders claim that all available data is relevant while some others claim that only the most recent data should be used because of structural changes in the markets.
Apparently, this kind of issues cannot be resolved in general ways (like a closed form mathematical solution) because trading system performance is in principle path-dependent, amongst other things. Specifically, it may not be reasonable to claim in advance that data in the far past, for example before 1980 in S&P 500 index, are irrelevant to the synthesis of a specific class of trading systems. The old data may be irrelevant to systems using a specific class of indicators but quite useful to systems using other indicators, like price patterns for example. Therefore, it may be best to study the importance of old data within the context of a specific system structure related to a particular synthesis process. In the example that follows, I will compare the performance of a system for SPY developed PAL using the normal TSS process to a system for the same ETF developed using the Walk-Forward Trading System Synthesis™ (WFTSS) process, as defined below.
Case 1: Using the whole in-sample to develop a system for SPY
| Interval | In-sample | Out-of-sample |
| 1 | 01/29/1993-12/31/2009 | 01/04/2010-09/13/2013 |
Case 2: Applying WFTSS to develop a system for SPY with same performance objectives as in Case 2
| Interval | In-sample | Out-of-sample |
| 1 | 01/03/1995-12/31/2009 | – |
| 2 | 01/02/1997-12/31/2009 | – |
| 3 | 01/04/1999-12/31/2009 | – |
| 4 | 0/02/2001-12/31/2009 | – |
| – | – | 01/04/2010-09/13/2013 |
In the first case the sample creation tool of PAL was used to split the data sample since inception in an in-sample and an out-of-sample as shown. In the second case, the data were split in successive out-samples that are then used as the in-samples. For example, for interval 1, first an in-sample was created from 01/29/1993 to 12/31/2009. Then this in-sample was divided in another in-sample from 01/29/1993 to 12/31/1994 and an out-of-sample from 01/03/1995 to 12/31/2009 that was used as the in-sample. Interval 2 was created by creating an out-of-sample from the first out-of-sample from 01/02/1997 to 12/31/2009, and so on.
The rationale in Case 2 is to gradually remove data from the in-sample to determine how that affects pattern generation. This is the reason it is called a walk-forward TSS. It is not a walk-forward process in the traditional sense of moving the start end points of the data forward but in the sense that the starting point of the data is moved forward while the end point remains fixed. The combined results from all 4 intervals of Case 2 will be compared to those of Case 1. Note that PAL automatically removes excess similar pattern generated by the same workspace leaving only the distinct ones.
Results
The search workspace setup for Case 1 is shown below:
PAL found 16 distinct patterns that satisfied the criteria specified on the search workspace, 13 long and 3 short, as shown below:
The performance of the above system of 16 patterns in the out-of-sample is shown below:
With profit factor equal to 1.62, win rate equal to 61.82%, an average trade (expectation) of $0.8016 per trade and with 110 trades this system shows satisfactory performance on the out-of-sample. Note that this performance was based on taking all the signals that were generated by the 16 patterns. This machine generated trading system for SPY is related to a another trading system generated in the past using a different in-sample.
The search workspace setup for Case 2 is shown below
In this case there are 4 search lines, one for each walk-forward in-sample. PAL found 21 distinct patterns that satisfied the criteria specified on the search workspace, 10 long and 11 short, as shown below:
The performance of the above system of 21 patterns in the out-of-sample is shown below:
In this case the profit factor is equal to 1.18, the win rate is equal to 53.33%, the average trade (expectation) is only $0.2810 per trade and 195 trades were generated. This performance is inferior to that obtained by Case 1 when using the whole in-sample from 01/29/1993 to 12/31/2009.
The reason for the degraded performance in Case 2 is primarily due to the increase in the number of short patterns from Case 1, an effect of the gradual elimination of the 1990s uptrend that gave rise to such patterns. This is to show that the WFTSS process is not a generic solution for the development of trading systems as far as performance improvement but results depend on the behavior of the particular price series and on the structure of the trading system. If the results of the WFTSS process, or any walk-forward process for that purpose, violate any inherent bias in the time series then, as a result, the performance of the system will be degraded as compared to that of a system developed using the whole in-sample. If a system is synthesized using only TYPE-III curve-fit indicators with position holding times in the order of a few bars, then the more data that is available the better the results with a traditional TSS process. Trading system development is a complicated process with results improving with knowledge and experience and whoever promises easy final solutions is either irrational or very naive.
References
-“Synthesis of Trading Systems: Beyond Back-testing and Analysis”, Traders’ Magazine, January 2006
-“Price Pattern Autopilot”, Active Trader Magazine, September 2002, p. 70.
–Trading System Synthesis
–Profitability and Systematic Trading: A Quantitative Approach to Profitability, Risk, and Money Management, by Michael Harris, 2008, Wiley & Sons. Out of print.
