Curve-fitting and Optimization

The subject of optimization and curve-fitting has received a lot of attention by trading system developers, especially during the last 10 years. There is a lot of confusion about this subject and different views, some even conflicting.

There are those who claim that optimization and curve-fitting are unavoidable and necessary but others insist that any trading system that is the outcome of such practices and methods will eventually fail. I have edited a post from last year, in which I present a three-level classification of optimized and curve-fitted systems that I think offers a more systematic look into this important issue.

What is curve-fitting?

In mathematics, curve fitting is the process of finding a curve that fits best a collection of data points in the sense that some objective function subject to constraints is maximized (or minimized). For example, least squares is a curve-fitting method that minimizes the sum of squared residuals. A residual is the difference between a fitted and an actual value. The objective function to minimize when using this method in order to get the best fit is the sum of the squared residuals. Thus, one can immediately notice that what is best is defined only relative to the chosen objective and that curve fitting is essentially the result of optimization.

Next, we will see how the notion of curve fitting applies to trading systems design. A trading system is a process that generates a collection of entry and exit signals [1]. Usually, the algorithm, or model, of a trading system includes a set of parameters. The values of the parameters must be selected so that the system performs best during actual trading. It is common practice to set these parameters by back-testing the system on historical data so that some objective function is maximized (or minimized). For example, one can set the parameters so that the net profit is maximized or the maximum drawdown is minimized, just to mention two possibilities.

Curve-fitting and optimization

When one adopts the definition that trading systems are processes that generate collections of entry and exit signals, then one will realize that what it is done, essentially, when any parameters are adjusted via back-testing is that the timing of the signals is varied so that they are fitted on historical data in such a way so that an objective function is optimized. This is not curve fitting in the usual sense because one is not merely trying to find a curve that best fits the historical data but instead find the best collection of entry signals that in conjunction with the exit signals maximize some objective. This process is much more involved and complicated than plain vanilla curve fitting. It involves selection, or timing of entry and exit signals, that maximize an objective function that is related to performance.  It is an optimization problem rather than just a curve fitting problem. As already mentioned, curve fitting may involve optimization but the latter is a process with a much broader scope and includes many more possibilities than curve-fitting. Thus, it is better to refer to optimized systems than to curve-fitted systems although this turns out to be more of a semantics issue for those that understand the process in depth.

For example, let as consider a simple moving average crossover system that generates long entry signals when SMA(t1) > SMA (t2), where t1 and t2 are the periods with t2 > t1, and short entry signals when SMA(t1) < SMA(t2). In its simplest form, this is a stop and reverse system, i.e., when an opposite signal is generated the previous position is closed and reversed. This system cannot be used in practice unless the values of t1 and t2 are selected. One can select those values via optimization of performance using back-testing on historical data. It is a widespread belief that this process results in systems that fail in actual trading because they are “curve fitted”. Is this belief true?

Actually, nobody has ever proven mathematically that the failures of optimized systems, which are well documented, are primarily due to the optimization, or what is commonly referred to as “curve-fitting”.  It may be the case that the failures are merely due to the inability of the indicator these systems are based on to adapt to changing market conditions. As a matter of fact this is more probable, since most indicators lag price. Thus, it is more probable that optimized trading systems will fail for any values of their parameters at some point. It is the nature of the system and not the optimization that causes the failure. The large class of trading systems based on indicators has high probability of failure but that has been wrongly attributed, based on my experience in developing trading systems, to the optimization process for setting parameters.  It does not even matter whether their parameters are set so that small changes in their values result in stable performance. This is not an issue of the integrity of the optimization method used but of the nature of these trading systems. Nevertheless, optimization that causes selection of entry and exit collections is in general a problematic process because it introduces survivorship bias. Selecting collections that performed best in the past overlooks the fact that many such collections failed.

Going back to the simple moving average crossover system, it is easy to understand that given a specific historical data series, changing the values of t1 and t2 will often cause a change in the timing of the entry and exit signals. In this case, selecting any collection of entry and exit signals that results from specific values of the parameters such that some objective function is maximized introduces severe bias because it may be due to chance that the specific collection survived. In the simple moving average crossover example, each collection is different from the others in the sense that both the entry and the exit points are different. What can we do to minimize the survivorship bias so that the integrity of the optimization process is not compromised? This question can be answered if we first understand how different types of systems are affected by optimization of their parameters.

A three-level classification of optimized trading systems

We can distinguish three types of systems in relation to how optimization effects their collection of entry and exit points:

Type-I curve-fit: When the parameters of Type-I systems are adjusted both the entry and the exit signals are affected, like in the simple moving average crossover system considered before. In this case, optimization and curve fitting result in different collections of entry and exit signals and selecting one that performs best introduces bias. These systems have the highest probability of failure.

Type-II curve-fit: When the parameters of Type-II systems are adjusted, only the entry signals are affected. In this case, optimization and curve fitting result in collections of entry and exit signals that differ only in their entry part. Selection in this case introduces less bias than in the case of Type-I systems. These systems have lower probability of failure that Type-I systems. Example: Enter long if SMA(t1) > SMA(t2) and Exit long at P1 or P2 where P1 and P2 are fixed prices (profit price and stop price).

Type-III curve-fit: When the parameters of Type-III systems are adjusted, only the exit signals are affected. In this case, optimization and curve fitting result in collections of entry and exit signals that differ only in their exit part. Selection introduces less less bias than with Type-I or Type-II systems. These systems have the lowest probability of failure because the timing of entry signals is not affected by optimization. Example: Enter long if Close of today > Close of 2 days ago and Exit long at entry price + x points or at entry price – y points, where x and y are the parameter to optimize (profit target and stop-loss).

In general, systems that include indicators involve Type-I curve-fit.  Type-II curve fit systems are rarely used in practice. Type-III curve-fit include the broad class of systems based on pure price patterns and systems based solely on price action.

Most software programs that discover trading systems automatically generate Type-I systems. It is irrelevant how many statistical tests they perform to measure the significance of the results as these systems have high probability of failure during actual trading because of their nature. An example of a software program that automatically discovers only Type-III systems is DLPAL. The search algorithm of this program was designed to search only for price patterns that belong in Type-III class of optimized systems while their selection introduces minimum bias.

As a conclusion we can state that the issue is not whether a system is optimized, because all systems are in one way or another,  but to what degree the optimization impacts the probability that the system will fail in the future due to its nature. Of course, systems fail for all sorts of other reasons but in this short article we dealt with optimization and curve-fitting. Type-III curve-fit systems, as defined above, appear to have the lowest probability of failure if properly designed.

References