← All work
Project write up · Python, NumPy, pytest

Honest Backtest

An event-driven backtesting engine, and a study of what happens to a strategy's Sharpe ratio as the usual shortcuts are removed one at a time. A momentum strategy that can see one bar ahead reports 6.94. Tested honestly, it earns about 0.22 on average, an edge too small for any single eight-year backtest to tell apart from zero.

Rohan Raghu github.com/rohanKraghu/honest-backtest 350 tests MIT license

Why build it

Backtests go wrong in a few well-known ways, and each one pushes the result up. The signal reads tomorrow's price through a missing shift. Normalisation uses statistics from the whole sample, future included. Orders fill at the close for free. The best-fitting parameter is partly fitted to noise. This project builds an engine where the first two cannot happen by accident, then removes each shortcut in turn and measures what it was worth.

The setup

Prices are synthetic: ten years of daily bars (2,520) from a random walk with 20% annual volatility, plus a weak, persistent drift driven by a hidden AR(1) state (persistence 0.94). Because the edge is planted, its true size is known. A trader who knew the hidden state and paid no costs would score a Sharpe of about 1.5 on average across paths (1.30 on the headline path, taking a full long or short position on the state's sign).

The strategy is time-series momentum: the trailing log return over a lookback, divided by a trailing estimate of its own spread, sets the position size. The only parameter fitted is the lookback, chosen from 5, 10, 15, 20, 30, 45 and 60 bars. Every stage is scored over the same bars, 504 to 2,520 (eight years), so no stage gets an easier period than another.

Stage 1 is a deliberately planted bug, not a strawman of anyone's code: its momentum window includes the next bar's close, the very return the position then earns, and its volatility scale is computed over the whole sample. It runs through a separate subclass that refuses to run against the honest data handler.

One price path, five stages

The default seed (20260921), the run python run_experiment.py prints. The dashed line is the cost-free trader who knows the hidden state, on this path.
StageSharpeP(edge)Ann. returnMax DDTradesLookback
1 Naive backtest6.94>99.9%+131.0%−3.1%21325
2 + point-in-time data0.4377.8%+4.9%−26.4%166520
3 + slippage0.2760.9%+2.7%−31.1%166420
4 + commissions0.2659.7%+2.6%−31.4%166520
5 + walk-forward OOS−0.1929.8%−3.2%−29.8%17895 to 45

Two hundred price paths

One path is one draw. The study was rerun on 200 independent paths with the same settings. The headline path turns out to be unlucky: its −0.19 sits at the 18th percentile.

StageMean SharpeStd devMiddle 80%Below zero
Naive backtest6.940.216.68 to 7.200%
+ point-in-time data0.720.340.29 to 1.171%
+ slippage0.520.340.07 to 0.926%
+ commissions0.500.340.06 to 0.918%
+ walk-forward OOS0.220.41−0.29 to 0.7628%
Walk-forward Sharpe on each of the 200 paths. Red bars are below zero.

An earlier version of this write-up used 12 paths and found a walk-forward mean of 0.12, not distinguishable from zero, and concluded that the strategy does not work. With 200 paths that conclusion does not hold. The right reading is the one above: a small real edge, invisible in any one history.

On real prices

The same ladder on 21 years of real SPY daily prices (2005 to 2025, from Yahoo Finance, scored over the same 18 years at every stage) tells the same story. The momentum strategy that can see one bar ahead reports a Sharpe of 4.10. Without that one bar it earns 0.05, and walk-forward out of sample −0.16, below simply holding SPY at 0.59. A slow moving-average crossover, by contrast, keeps 0.79 out of sample: honest testing does not sink everything, only what was never there. That beats zero, but one 18-year history cannot show it beats holding the index.

The cost that matters is the one nobody invoices

On the headline path, commissions cost 0.01 of Sharpe and slippage 0.16. Over $462M of traded notional, a half-cent per-share commission came to 0.31bp, against 5.74bp of spread plus square-root market impact. Across the 200 paths, costs together took about 0.22 of Sharpe, nearly all of it slippage.

Cost per unit of traded notional, stage 4 of the headline run: an 18x gap. The model uses a 2bp half-spread and a square-root impact term (coefficient 0.6) on a $1M account trading a stock that averages a million shares a day, turning the book over about 44 times a year.

How it works

MarketEvent→ SignalEvent→ OrderEvent→ FillEvent

The engine is event-driven rather than vectorised. The outer loop advances market time once per bar, and the inner loop drains every event that bar caused before time moves again. The data handler holds the full history but exposes it only through accessors clamped to a cursor; asking for a future bar raises LookAheadError. Look-ahead becomes something the code cannot do, rather than a convention one careless .shift() can break. The price is speed, so a separate fast path replays the same signals without the event loop and is tested to give identical numbers.

Realistic execution

Spread plus square-root impact, optional decaying permanent impact, per-share or percentage commissions, next-bar fills, volume caps with partial fills, passive limit orders, and financing costs.

Honest validation

Rolling and anchored walk-forward, Deflated and Probabilistic Sharpe, and combinatorial purged cross-validation for a spread of out-of-sample paths instead of one.

Audit your own strategy

honest-backtest audit runs the same ladder on any strategy and price file, after a look-ahead check that swaps in a different future and fails the run if any past signal moves.

Live as written

honest-backtest paper trades a growing price file with the same cost models. A replayed feed reproduces the backtest's equity curve and trade blotter exactly, and a test asserts it.

Testing

The 350 tests target what would invalidate the result if it were wrong: that no strategy is ever shown a future bar at any cursor position, that equity reconciles to cash plus positions at every bar, that every slippage model is adverse in both directions, that walk-forward training always ends before testing begins, and that the headline degradation itself holds, so the claim cannot drift away from the code. Results are deterministic to the last digit for a given seed.

Writing the tests caught two real bugs: a constant return series reported a Sharpe ratio around 1016 because an exact zero check missed floating-point residue, and zero-quantity orders were becoming fills and inflating the trade count.

Reproduce it

git clone https://github.com/rohanKraghu/honest-backtest && cd honest-backtest
pip install -r requirements.txt
python run_experiment.py --seeds 1    # the headline path, a few seconds
python run_experiment.py              # plus 12 paths, about 30 seconds
python run_experiment.py --seeds 200 --fast --workers 4   # the 200 paths, under a minute
pytest                                # 350 tests

Every number on this page was rerun on the current code (commit f2081a2) on 16 October 2026.

Limits