How to automate a trading strategy backtest

Key takeaways

• A backtest is a measurement of a rule under stated assumptions. Automating it means automating the assumptions, so that every run is reproducible and every result can be decomposed.
• Most of the work is not the entry and exit logic. It is data hygiene, cost modelling, and the discipline of holding data out.
• The published research on backtest overfitting is unambiguous: searching many variants and reporting the best one produces impressive results from noise. ^{[1] [2]}

The research question comes first

Write the question down before touching data. "Does buying the first pullback after an open above the prior value area have positive expectancy on NQ, net of costs, over the last five years?" is a question. "Find something that works on NQ" is a search, and searches find things whether or not they exist.

State the hypothesis, the instrument, the period, and what result would make you abandon the idea. That last item is the one people skip.

Data

Use the finest resolution the rule actually needs and no finer. One-minute bars are enough for most intraday rules; tick data adds cost and complexity you should be able to justify.

Continuous futures contracts are stitched from individual expiries, and the stitch is a choice. Volume-stitched, unadjusted series carry price gaps at every roll; back-adjusted series remove the gaps but change historical price levels. Neither is wrong. What matters is that the rule is tested on the series it would trade on, and that the roll window is either handled explicitly or excluded and reported as such.

Clean the data before testing anything: missing sessions, duplicated bars, holiday half-days, and timezone errors are the usual culprits. Each one can manufacture an edge.

Definitions precise enough to reproduce

Every rule must be expressible as a question you could have answered at the time, using only information available at the time.

• Entry: the exact condition and the bar on which it is evaluated.
• Exit: target, stop, time stop, and which takes precedence when several trigger on the same bar.
• Position size: fixed contracts, fixed fractional, or volatility-scaled, with the inputs named.
• Session boundaries: in one timezone, stated.

A rule that references the session high, evaluated on the bar that set the high, has used information it did not have. This is look-ahead bias, and it is the single most common way a backtest flatters itself.

What a bar cannot tell you

A bar-based engine sees open, high, low and close. It does not see the order in which the high and the low were made. When a stop and a target both fall inside one bar, the engine has to decide which was hit first, and it cannot know. Maier-Paape and Platen catalogue exactly these "not uniquely decidable" situations and show that different platforms resolve them differently. ^[3] Their follow-up work proposes test procedures for checking whether an engine resolves them correctly at all. ^[4]

The honest choices are to assume the worst case (the stop first), to drop to a finer resolution for the bars where it matters, or to report both. A backtest that silently assumes the target was hit first is not a backtest of the strategy; it is a backtest of the engine's optimism.

Costs and slippage

A backtest without costs is a description of a market, not a strategy. Model at least:

• Commission and exchange fees per contract, per side.
• Slippage as a function of the order type: a market order pays the spread and then some; a limit order may not fill at all.
• Fill assumptions for limit orders. Assuming a fill whenever price touches the limit is optimistic; requiring price to trade through it is conservative. Report which one you used.

The honest approach is to test under both a base and a pessimistic cost assumption and see whether the conclusion survives.

Implementation

A minimal Python pipeline separates four concerns so each can be inspected on its own:

bars = load_bars("nq", "1m", start="2021-07-01", tz="America/Chicago")
bars = clean(bars)                      # gaps, duplicates, sessions
signals = rule(bars, config)            # pure function of past data only
trades = simulate(signals, bars, costs) # fills, costs, sizing
report(trades)                          # metrics, by period, by regime

The rule is a pure function of the bars up to and including the current one. If it needs anything else, that is a design smell worth investigating before it becomes a bias.

Keep every parameter in one frozen configuration object, and keep every experimental rule behind a flag that is off by default. That is what lets you run an honest A/B: one change at a time, everything else held constant.

Testing the test

• Out-of-sample halves. Split the period and check the second half alone. A result that only exists in the fitted half is not a result.
• Walk-forward. Fit on a window, test on the next, roll forward. Every reported period is then out of sample.
• Parameter sensitivity. A rule that works at a stop of 12 points and fails at 10 and 14 is a coincidence with a decimal point.
• Cross-market replication. A pattern of market behaviour should show up in related instruments without retuning. An edge that exists only on one contract is more often an artefact of that contract's data.
• Monte Carlo. Resample the trade sequence to see the distribution of drawdowns the same trades could have produced in a different order. The one path you observed is one draw.

Why the selection problem dominates

Bailey, Borwein, López de Prado and Zhu showed that the expected maximum Sharpe ratio across many backtested variants grows with the number of trials, so that a strategy selected for its in-sample performance will typically disappoint out of sample even when every individual test was honest. ^[1] Harvey, Liu and Zhu made a related argument for published factors: with hundreds of tests run, the usual statistical thresholds no longer mean what they appear to mean. ^[2]

The practical implication is not to stop searching. It is to count the searches, hold data out that the search never touches, and treat the held-out result as the result.

Where the sources differ

The two lines of research agree on the disease and differ on the remedy. Harvey, Liu and Zhu propose raising the bar for a claimed effect — a t-statistic above 3.0 rather than the conventional 2.0 — to account for the number of tests the literature has run. ^[2] Bailey and colleagues argue that a fixed threshold cannot account for the number of trials behind a particular backtest, and propose instead to estimate the probability that the chosen configuration is overfit from the trials themselves. ^{[1] [5]} Neither is wrong; they answer different questions. For a strategy developer the second is the usable one, because the developer knows how many variants were tried.

What not to conclude

A positive backtest, even a careful one, is evidence that a rule would have worked on the data used. It is not evidence about the next year. Deployment adds execution, latency, and the fact that the market you tested on is not obliged to persist.