Honest Backtest
An event-driven backtesting engine, and a study of what happens to a strategy's Sharpe ratio as the usual shortcuts are removed one at a time. A momentum strategy that can see one bar ahead reports 6.94. Tested honestly, it earns about 0.22 on average, an edge too small for any single eight-year backtest to tell apart from zero.
Why build it
Backtests go wrong in a few well-known ways, and each one pushes the result up. The signal reads tomorrow's price through a missing shift. Normalisation uses statistics from the whole sample, future included. Orders fill at the close for free. The best-fitting parameter is partly fitted to noise. This project builds an engine where the first two cannot happen by accident, then removes each shortcut in turn and measures what it was worth.
The setup
Prices are synthetic: ten years of daily bars (2,520) from a random walk with 20% annual volatility, plus a weak, persistent drift driven by a hidden AR(1) state (persistence 0.94). Because the edge is planted, its true size is known. A trader who knew the hidden state and paid no costs would score a Sharpe of about 1.5 on average across paths (1.30 on the headline path, taking a full long or short position on the state's sign).
The strategy is time-series momentum: the trailing log return over a lookback, divided by a trailing estimate of its own spread, sets the position size. The only parameter fitted is the lookback, chosen from 5, 10, 15, 20, 30, 45 and 60 bars. Every stage is scored over the same bars, 504 to 2,520 (eight years), so no stage gets an easier period than another.
Stage 1 is a deliberately planted bug, not a strawman of anyone's code: its momentum window includes the next bar's close, the very return the position then earns, and its volatility scale is computed over the whole sample. It runs through a separate subclass that refuses to run against the honest data handler.
One price path, five stages
python run_experiment.py prints. The dashed line is the cost-free trader who knows the hidden state, on this path.| Stage | Sharpe | P(edge) | Ann. return | Max DD | Trades | Lookback |
|---|---|---|---|---|---|---|
| 1 Naive backtest | 6.94 | >99.9% | +131.0% | −3.1% | 2132 | 5 |
| 2 + point-in-time data | 0.43 | 77.8% | +4.9% | −26.4% | 1665 | 20 |
| 3 + slippage | 0.27 | 60.9% | +2.7% | −31.1% | 1664 | 20 |
| 4 + commissions | 0.26 | 59.7% | +2.6% | −31.4% | 1665 | 20 |
| 5 + walk-forward OOS | −0.19 | 29.8% | −3.2% | −29.8% | 1789 | 5 to 45 |
- The naive number is impossible, and that is the tell. 6.94 is more than four times what a trader who knows the hidden state can get on this path, with a 3.1% maximum drawdown over eight years. A backtest that beats the data's own ceiling is reading its own answer.
- Stages 2 to 4 still choose the lookback on the bars they report. Walk-forward refits on each two-year training window and scores only the year after it. Across the eight folds the chosen lookback swings between 5 and 45 bars (5, 5, 45, 30, 20, 20, 5, 5), which is what fitting noise looks like.
- P(edge) is the Deflated Sharpe Ratio for the in-sample stages (Bailey and López de Prado, 2014), which allows for the seven lookbacks tried, and the Probabilistic Sharpe Ratio against zero for walk-forward, which chose nothing on the bars it reports. The naive stage's >99.9% shows what significance on a leaky backtest is worth.
Two hundred price paths
One path is one draw. The study was rerun on 200 independent paths with the same settings. The headline path turns out to be unlucky: its −0.19 sits at the 18th percentile.
| Stage | Mean Sharpe | Std dev | Middle 80% | Below zero |
|---|---|---|---|---|
| Naive backtest | 6.94 | 0.21 | 6.68 to 7.20 | 0% |
| + point-in-time data | 0.72 | 0.34 | 0.29 to 1.17 | 1% |
| + slippage | 0.52 | 0.34 | 0.07 to 0.92 | 6% |
| + commissions | 0.50 | 0.34 | 0.06 to 0.91 | 8% |
| + walk-forward OOS | 0.22 | 0.41 | −0.29 to 0.76 | 28% |
- The honest strategy has a real but small edge. The walk-forward mean of 0.22 has a standard error of 0.03 across paths, so it is clearly above zero. That is about 15% of what the hidden-state trader gets.
- No single backtest can see it. One eight-year path's walk-forward Sharpe varies by 0.41 from path to path, nearly twice the edge, and 28% of paths come out negative. Someone holding one history, which is everyone with real data, cannot tell this strategy from a coin flip.
- Choosing the lookback in-sample is worth 0.28 of Sharpe on average, the gap from stage 4 to stage 5. Walk-forward came in below the in-sample figure on 92% of paths.
- The fall is not strictly stage by stage on every path. Sharpe dropped at every stage on 183 of 200 paths. The exceptions are counted, not dropped.
An earlier version of this write-up used 12 paths and found a walk-forward mean of 0.12, not distinguishable from zero, and concluded that the strategy does not work. With 200 paths that conclusion does not hold. The right reading is the one above: a small real edge, invisible in any one history.
On real prices
The same ladder on 21 years of real SPY daily prices (2005 to 2025, from Yahoo Finance, scored over the same 18 years at every stage) tells the same story. The momentum strategy that can see one bar ahead reports a Sharpe of 4.10. Without that one bar it earns 0.05, and walk-forward out of sample −0.16, below simply holding SPY at 0.59. A slow moving-average crossover, by contrast, keeps 0.79 out of sample: honest testing does not sink everything, only what was never there. That beats zero, but one 18-year history cannot show it beats holding the index.
The cost that matters is the one nobody invoices
On the headline path, commissions cost 0.01 of Sharpe and slippage 0.16. Over $462M of traded notional, a half-cent per-share commission came to 0.31bp, against 5.74bp of spread plus square-root market impact. Across the 200 paths, costs together took about 0.22 of Sharpe, nearly all of it slippage.
How it works
The engine is event-driven rather than vectorised. The outer loop advances market time once per bar, and the inner loop drains every event that bar caused before time moves again. The data handler holds the full history but exposes it only through accessors clamped to a cursor; asking for a future bar raises LookAheadError. Look-ahead becomes something the code cannot do, rather than a convention one careless .shift() can break. The price is speed, so a separate fast path replays the same signals without the event loop and is tested to give identical numbers.
Realistic execution
Spread plus square-root impact, optional decaying permanent impact, per-share or percentage commissions, next-bar fills, volume caps with partial fills, passive limit orders, and financing costs.
Honest validation
Rolling and anchored walk-forward, Deflated and Probabilistic Sharpe, and combinatorial purged cross-validation for a spread of out-of-sample paths instead of one.
Audit your own strategy
honest-backtest audit runs the same ladder on any strategy and price file, after a look-ahead check that swaps in a different future and fails the run if any past signal moves.
Live as written
honest-backtest paper trades a growing price file with the same cost models. A replayed feed reproduces the backtest's equity curve and trade blotter exactly, and a test asserts it.
Testing
The 350 tests target what would invalidate the result if it were wrong: that no strategy is ever shown a future bar at any cursor position, that equity reconciles to cash plus positions at every bar, that every slippage model is adverse in both directions, that walk-forward training always ends before testing begins, and that the headline degradation itself holds, so the claim cannot drift away from the code. Results are deterministic to the last digit for a given seed.
Writing the tests caught two real bugs: a constant return series reported a Sharpe ratio around 1016 because an exact zero check missed floating-point residue, and zero-quantity orders were becoming fills and inflating the trade count.
Reproduce it
git clone https://github.com/rohanKraghu/honest-backtest && cd honest-backtest pip install -r requirements.txt python run_experiment.py --seeds 1 # the headline path, a few seconds python run_experiment.py # plus 12 paths, about 30 seconds python run_experiment.py --seeds 200 --fast --workers 4 # the 200 paths, under a minute pytest # 350 tests
Every number on this page was rerun on the current code (commit f2081a2) on 16 October 2026.
Limits
- Synthetic data with one planted effect. Real prices have fat tails, regime changes and many weak effects at once. The study tests the method, not a market. The SPY run above shows the same look-ahead collapse on real data, but there is no oracle there, so the size of the true edge is only known on synthetic prices.
- One instrument, one parameter. A seven-value grid is a mild search. Real research tries far more ideas, and no statistic can count the ideas tried before this one, so real overfitting is usually worse than shown here.
- Assumed costs. The spread, impact and commission settings are plausible defaults for a liquid stock, not measured from a venue. Each is a setting.
- Sharpe scaling. Annualising by the square root of 252 assumes independent returns; the planted drift makes returns slightly autocorrelated, so every Sharpe here is a mild approximation.