QQ 201, 2 min read
Overfitting and multiple testing
After this lesson you can
- Compute the chance of a false positive across many tests
- List the signs of an overfitted strategy
- Explain what a deflated Sharpe ratio charges for
Suppose you test one idea with no real edge. The chance it looks significant at the usual 5% level is 5%.
Now test twenty such ideas. The chance that at least one looks significant is:
1 - 0.95 to the power 20 = about 0.64
That is nearly two in three, and the ideas are no better than before. The only thing that changed is how many times you looked.
This is multiple testing, and a strategy search does it constantly. Every parameter you adjust, every filter you add, every market you try is another look. The luck test on our home page shows it happening with random lines.
Overfitting
Overfitting is tuning rules until they describe the noise in one history. An overfitted strategy has many parameters, each set to an oddly exact value, and stops working when any of them moves a little.
Signs of it:
- performance collapses when a parameter changes slightly
- the result depends on a handful of trades
- it works on one market or one period and nowhere else
- every addition to the rules fixed one specific past loss
Paying for the search
The fix is to count the tries and charge for them.
The deflated Sharpe ratio asks: given how many variants were tried, how likely is it that the best one's Sharpe is above zero for real? The more you tried, the higher the bar.
That only works if the count is honest, so the terminal keeps a trial registry: every variant run on a strategy and dataset is recorded, and every significance test charges for all of them. In the run on this site the agent had tried 13 variants, and the deflated Sharpe check failed.
Key points
- Looking more times finds more false positives
- An overfitted strategy breaks when a parameter moves slightly
- The search has a cost, and it can be counted
Check your understanding
You test 10 ideas that have no edge, each at the 5% level. What is the chance at least one looks significant?
Which is a sign of overfitting?
Further reading
- White, H. (2000). "A Reality Check for Data Snooping." Econometrica 68(5).
- Harvey, C. R., Liu, Y. and Zhu, H. (2016). "... and the Cross-Section of Expected Returns." Review of Financial Studies 29(1).
- Bailey, D. H., Borwein, J., Lopez de Prado, M. and Zhu, Q. J. (2017). "The Probability of Backtest Overfitting." Journal of Computational Finance 20(4).
- Sullivan, R., Timmermann, A. and White, H. (1999). "Data-Snooping, Technical Trading Rule Performance, and the Bootstrap." Journal of Finance 54(5).