| Question | Can a backtest answer it? | Why |
|---|---|---|
| Would this rule have made money historically? | Yes, if the data is clean | This is arithmetic, and it is the only part that is straightforward. |
| Is the rule broken? | Yes, decisively | A rule that fails on history is disproved. Falsification is the one thing a backtest does conclusively. |
| Will this rule make money in future? | No | Nothing about past prices makes future prices follow them. A backtest is evidence about a hypothesis, not a forecast. |
| Is the result real or a fluke? | Not on its own | That depends on how many variants were tried before this one, which the backtest itself does not record. |
| Is the data honest? | No | Look-ahead and survivorship bias corrupt the input. A clean measurement of corrupted data still looks clean. |
| Would it survive real costs? | Only if they were modeled | Spreads, slippage and market impact are assumptions, not observations, and small errors compound over many trades. |
How do you run a backtest properly?
State the idea before you look
Write down the rule, the instruments, the period and what result would make you abandon it. An idea specified after seeing the data is not being tested by that data.
Split the data and mean it
Hold back a period that plays no part in any decision: not in choosing parameters, not in choosing the rule, not in deciding when to stop. Data you have looked at is in-sample whether or not you called it that.
Run the rule on the in-sample period
Use only information that was available at each point in time. Prices restated later, index membership as it stands today, and fundamentals dated by report rather than by publication all leak the future into the past.
Apply costs before you look at the equity curve
Spread, commission, slippage and, for larger size, market impact. Applying costs afterwards invites you to argue them down once you have seen what they remove.
Count the trials
Record how many variants you ran, including the ones you discarded in minutes. This number is an input to every statistic that tells you whether the winner is real, and it is the number people forget.
Check the held-out period once
Once. If the held-out result disappoints and you go back to adjust the rule, that period has become in-sample and you no longer have a test.
What is backtesting in trading?
A backtest replays a decision rule over historical data and records what it would have bought, when, at what price, and what the account would have been worth afterwards. The output is usually an equity curve and a set of summary statistics: total return, maximum drawdown, hit rate, Sharpe ratio.
That description is accurate and it is where nearly every explanation stops, which is the problem. It presents a backtest as a measurement, and a measurement sounds like something you can trust once you have taken it carefully. A backtest is not a measurement of a strategy. It is a measurement of one particular history, under one particular set of assumptions, for a rule that you chose partly because you already had some sense of what that history looked like.
The useful framing is closer to an experiment than to a measurement. You have a hypothesis about how a market behaves. The backtest is the attempt to kill it.
Why does a good backtest usually mean nothing?
Because good backtests are cheap. Take a few hundred instruments, a handful of indicators, a range of parameters for each, and you can generate thousands of distinct rules in an afternoon. Some of them will show excellent performance on any history you pick, for the same reason that some sequence of coin flips in a large enough sample will look like a hot streak.
The strategy you end up presenting is the survivor of that search. Its equity curve looks the way it does partly because of whatever real structure exists in the market and partly because it was selected for looking good. Nothing in the backtest separates those two contributions, and the flattering half is invisible.
This is why the number of variants tried is the most important fact about any backtest and the one almost never reported. A Sharpe ratio of 2 from the first idea somebody tested and a Sharpe ratio of 2 from the best of fifty thousand are the same number describing entirely different evidence.
What is the difference between backtesting and forward testing?
Backtesting runs a rule over data that already exists. Forward testing, also called paper trading, runs it on data as it arrives, with no position taken. The distinction that matters is not past versus future; it is whether the data had any chance to influence the design of the rule.
Forward testing is harder to fool because the data does not exist yet when you commit. Its cost is time: at typical trading frequencies it can take many months to accumulate enough observations to distinguish a real edge from noise, and by then market conditions may have changed. Held-out historical data gives you the same protection immediately, but only if it was genuinely held out.
Neither is a substitute for the other, and neither rescues a rule that was chosen because it fit a chart.
What makes backtest results wrong?
Four failures account for most of the distance between a backtest and reality. The first two corrupt the data, so no amount of statistical care recovers from them. The second two corrupt the conclusion.
- Look-ahead bias: the rule uses information that was not available at the moment it acts. Restated fundamentals, closing prices used for decisions taken during the day, and index membership as it stands today are the usual sources.
- Survivorship bias: the instrument list contains only what still exists. Companies that were delisted, merged or went to zero are missing, so the sample quietly excludes the worst outcomes.
- Overfitting: the rule was selected, or its parameters tuned, on the same data used to judge it. The result describes the search rather than the market.
- Unmodeled costs: spread, slippage, financing and market impact. A strategy trading frequently can be profitable before costs and reliably unprofitable after them.
The first two are checkable by inspecting the data pipeline. The third is not visible in the result at all and has to be measured separately.
Is backtesting a waste of time?
No, but the common reason for doing it is. If the purpose is to produce a chart that justifies a decision already made, the exercise is worse than useless, because it converts a hunch into something that looks like evidence.
The defensible purpose is elimination. Most trading ideas are wrong, and a backtest is the cheapest way to find that out. A research process that kills most of what it tests is behaving correctly. One that keeps producing winners is not finding edges; it is finding the flexibility in its own search.
Read that way, the most valuable backtest is the one that fails, and the appropriate reaction to a strategy that looks excellent is suspicion rather than satisfaction.
Backtested and hypothetical results are not indicative of future performance. Nothing here is investment advice.
Common questions
How much data does a backtest need?
There is no fixed answer, and trade count matters more than calendar length. What you need is enough observations that the result is distinguishable from noise, plus coverage of conditions that differ from each other. Ten years that were all one long bull market test one regime, not ten.
Is backtesting the same as paper trading?
No. Backtesting uses historical data that already exists; paper trading runs the rule forward on live data without taking positions. Paper trading is harder to fool because the data has not happened yet, but it takes months to produce a comparable sample.
Can a backtest prove a strategy works?
No. It can disprove one, which is a different and more useful thing. A strategy that fails a well-constructed backtest is finished; a strategy that passes has only survived one attempt to kill it, and the strength of that evidence depends on how many other candidates were tried alongside it.
Why do backtested strategies fail in live trading?
Usually because the result was selected rather than discovered: the rule was the best of many tried on the same data. Beyond that, costs are commonly underestimated, execution differs from the assumed fills, and some effects genuinely decay once enough capital pursues them.
What is a good return in a backtest?
The question cannot be answered from the return alone. A high figure produced by a large search is weaker evidence than a modest figure produced by a small one. Ask how many variants were tried and how much data played no part in any decision before asking how large the number is.