Handbook
What makes a backtest worth believing.
A backtest is not evidence on its own. What decides whether it means anything is how the strategy was found: how many variants were tried, how much data stayed genuinely untouched, and how the result holds up once you account for the search. These pages explain the statistics that answer that, in plain language, including where the honest answer is unhelpful.
Start here
What the words mean, and what each number is actually measuring. No prior statistics assumed.
- BacktestingBacktesting runs a trading rule against historical data to see how it would have performed. What it can establish, what it cannot, and how it misleads.
- Risk-adjusted returnRisk-adjusted return explained: how the Sharpe, Sortino, Calmar and information ratios differ, which one to use when, and the blind spot all of them share.
- Sharpe ratioThe Sharpe ratio explained: what it measures, the formula and its three inputs, how to calculate and annualize it, and the risks it cannot see.
- Sortino ratioThe Sortino ratio explained: downside deviation, the formula and its target return, what a large gap from the Sharpe ratio means, and what it does not fix.
- DrawdownDrawdown explained: how it is calculated, what maximum drawdown and time under water mean, why recovery is asymmetric, and why a backtest understates it.
Reading a result
You have a backtest and a set of figures. These decide how much of it to believe.
- Does AI trading actually work?Sometimes, and rarely the way it is sold. What AI is good at in markets, why AI backtests look strong, and how to tell a real edge from a well-searched one.
- How do I know if my backtest is overfit?You cannot tell from the equity curve. The checks that separate a real edge from a well-searched one, in the order worth running them, ending in a number.
- What is a good Sharpe ratio?The conventional bands, where they come from, and why a Sharpe ratio from a backtest cannot be read against them until you know how many variants were tried.
- Do trading costs and slippage kill a backtested edge?Often, and the faster a strategy trades the more likely it is. How spread, slippage and impact enter a backtest, and why they are assumptions rather than data.
How the checks work
The validation methods themselves, including the two the rest of the category does not report.
- Look-ahead biasLook-ahead bias is a backtest using information that was not available at the time it acts. Where it hides, how to find it, and why statistics cannot detect it.
- Walk-forward analysisWalk-forward analysis retunes a strategy on a rolling window and tests it on the period that follows. What it fixes, what it cannot fix, and how it is misused.
- Probability of backtest overfitting (PBO)PBO estimates how often the strategy that looked best in-sample lands below median out-of-sample. How it is computed, what 0.8 means, and why it grades a search.
- Deflated Sharpe ratioThe deflated Sharpe ratio is a probability, not a Sharpe ratio. What it measures, what a value of 0.2 means, and why there is no universal good threshold.