QQ 201, 2 min read

Walk-forward and holdouts

A strategy tuned on a history will look good on that history. The question is what it does on data that played no part in building it. That data is called out of sample.

Walk-forward

Split the history into consecutive windows. Tune the parameters on the first window, then run those settings, unchanged, on the next window. Slide forward and repeat.

tune on years 1 to 3   ->  test on year 4
tune on years 2 to 4   ->  test on year 5
tune on years 3 to 5   ->  test on year 6

Join the test pieces together and you have a record made entirely of decisions taken without knowing what came next. That is how the strategy would really have been run.

Anchored walk-forward keeps the start fixed and grows the tuning window instead of sliding it.

A useful summary is how much of the tuned performance carried over to the test windows. If profit per bar out of sample is a small fraction of what it was in sample, the tuning was fitting noise.

The sealed holdout

Set aside the last part of the data before you begin, and do not look at it. Develop everything on the rest. When you are finished, run once on the holdout.

The discipline is the hard part. Every time you look at the holdout and then change something, it becomes part of the search and stops being a fair test. A holdout can be opened honestly once.

The terminal seals the last slice of the data and records how many times it has been opened, so a report cannot quietly claim a clean holdout after ten looks.

What neither can do

Both tests still use the past. A strategy can pass walk-forward and a holdout and still fail when the market changes in a way history did not contain. What they do is rule out the common ways of fooling yourself.

Further reading

• Pardo, R. (2008). The Evaluation and Optimization of Trading Strategies, 2nd edition. Wiley. The standard treatment of walk-forward analysis.
• Lopez de Prado, M. (2018). Advances in Financial Machine Learning. Wiley. The chapters on cross-validation in finance.