| Observation | What it actually tells you |
|---|---|
| A smooth, steep equity curve | Nothing. Overfit strategies produce the smoothest curves — that is what they were selected for. |
| A high Sharpe ratio | Nothing on its own, until you know how many variants it was the best of. |
| It worked across ten years of data | Little, if those ten years also informed the choice of strategy. |
| It works on a different asset or market | Something. Not proof, but harder to arrange by accident. |
| Performance is stable across nearby parameter values | Something. A result that only exists at one setting is usually an artifact of that setting. |
| It holds on data untouched by every decision | The strongest evidence available — and only usable once. |
| A low probability of backtest overfitting | Something, provided the trial count behind it was honest. |
The checklist, in the order worth running it
Count the trials, generously
Write down how many variants you evaluated before keeping this one. Include parameter sweeps, ideas you abandoned after a glance, and rules you added after seeing a chart. If the honest answer is "I have no idea", that is your first finding: the strategy cannot currently be defended, whatever it shows.
Perturb the parameters
Move each parameter to its neighbouring values and re-run. A real effect degrades gently. If a lookback of 20 works and 18 and 22 collapse, you have found a property of this dataset, not of the market.
Check whether any data is still clean
Identify the portion of history that played no part in any decision you made — not the last slice of the same file you have been iterating on, but data you have genuinely never looked at. If none exists, stop and get some. The rest of the checklist depends on it.
Test on that clean data, once
Run the strategy on the untouched period exactly as specified, with no adjustments. The moment you look at the result and go back to change something, that data has been spent and is now in-sample.
Walk it forward
Refit and re-evaluate as the sample rolls forward, the way you would have had to run it in real time. This catches strategies that only worked because a single regime dominated the sample.
Deflate the result and put a number on it
Compute a deflated Sharpe ratio using the trial count from step one, and a probability of backtest overfitting across the configurations you chose from. This converts the checklist into two numbers you can argue with, which is the point of doing it.
Why can I not tell from the equity curve?
Because the curve is the thing that was optimized. If you searched for a strategy with a smooth rising line, you will find one, and it will look exactly like a strategy that has a real edge. The visual evidence and the selection pressure point in the same direction, which is why experienced practitioners get more suspicious as the chart gets better, not less.
The information that would let you distinguish the two cases is not in the chart at all. It is in the history of how you arrived at it.
How many trades does a backtest need to be significant?
There is no fixed number, and the familiar ones are folklore. Thirty, fifty, a hundred, thirty per free parameter — these circulate widely, are mutually inconsistent, and none of them is derived from anything. The rule of thirty in particular is imported from introductory statistics, where it applies to independent draws from a well-behaved distribution.
Strategy returns break every one of those assumptions. They are fat-tailed, serially dependent, clustered in volatility, and — decisively — the strategy was chosen because it looked good, which inflates any naive significance test applied afterwards.
The useful version of the question is different: how long would a track record need to be for this particular Sharpe ratio to be distinguishable from noise, given how many variants were tried? That has an answer, it is called the minimum track record length, and it is usually longer than people expect.
Does out-of-sample testing prove a strategy works?
No, and it is worth being precise about what it does. A clean out-of-sample test is a falsification attempt: the strategy could have failed and did not. That is real evidence, and it is the strongest single piece available, but it is one experiment.
It is also easy to destroy without noticing. The most common way is iteration — testing out-of-sample, seeing a poor result, adjusting the strategy, and testing again. After the second test the data is in-sample, and after the fifth the whole exercise has quietly become a slower parameter sweep. Out-of-sample data is a consumable, and most people spend it and continue to report it as evidence.
What if the strategy is simple? Can a simple rule be overfit?
Yes, and this is the most common false comfort in the category. Overfitting is not about the number of parameters in the final strategy; it is about the number of alternatives you looked at before settling on it. A two-parameter moving-average crossover selected from twelve thousand two-parameter crossovers is heavily overfit despite being trivially simple.
Complexity makes overfitting easier. It is not what causes it. Search does.
What if it is already overfit — is the idea dead?
Usually the idea survives and the specific configuration does not. Finding that your result was driven by selection tells you where you are, not that the underlying thesis is wrong. What it removes is your right to claim the backtest as evidence.
The honest continuation is to state the hypothesis in advance, in a form specific enough to fail, and then test it on data you have not touched. That is slower, produces far fewer strategies, and is the only version of this process that accumulates knowledge rather than confidence.
Backtested and hypothetical results are not indicative of future performance. Nothing on this page is investment advice.
Common questions
My strategy has a Sharpe of 3. Is that overfit?
Probably, but the number alone cannot decide it. A Sharpe of 3 selected from a handful of pre-specified ideas is remarkable; the same Sharpe selected as the best of fifty thousand variants is roughly what you would expect from noise. The trial count is the missing input, and without it the Sharpe ratio is uninterpretable rather than good or bad.
Does adding more data fix overfitting?
Only if the additional data stays out of your decisions. More history that you then iterate against becomes more surface to fit, which makes the problem worse rather than better. Data helps in proportion to how untouched it remains.
Is walk-forward analysis enough on its own?
It is a strong check and it is not sufficient. Walk-forward validates that a strategy adapts sensibly over time, but if the choice of what to walk forward was itself the winner of a large search, the whole procedure inherits that selection. Walk-forward tells you about the strategy; trial counting tells you about how you found it.
What is the fastest single check?
Perturb the parameters. It takes minutes, needs no extra data, and kills a large share of overfit strategies outright. It cannot clear a strategy — surviving it proves little — but a result that evaporates one setting either side of the optimum has answered the question.