SDS Quant Research

Backtest overfitting and why the best result is usually luck

2026-11-21 · Learn

If you try enough variations of a strategy, one of them will look excellent on past data even if none of them has any real edge. That is overfitting, and it is the most common reason published backtests fail in live trading.

The arithmetic of many trials

Each variation you test is a chance to get lucky. Run a handful and the best result is modestly flattered. Run thousands, as automated searches often do, and the best result is almost guaranteed to be an accident of the particular history you tested.

The danger is that the winner looks identical to a real discovery. Only the number of attempts tells you which it is, and that number is rarely reported.

Habits that help

  1. Count every trial. Keep a registry of all configurations run, not only the successful ones.
  2. Separate the periods. Develop on one period, confirm on a second, and keep a final period sealed until the work is finished.
  3. Run the sealed period once. Looking, adjusting and looking again turns the final test into one more training set.
  4. Use statistics that account for the number of trials. Research by Bailey and López de Prado introduced the probability of backtest overfitting and the deflated Sharpe ratio for this purpose.
  5. Compare with random controls. A strategy that does not clearly beat random choices under the same rules has not shown an edge. See our methodology.

What a good report looks like

It states the hypothesis in advance, how many variants were tried, how the final period was handled, and what happened. It includes the results that did not work.

This article is educational and is not investment advice.