Backtests Don't Account for Fear, Uncertainty and Doubt
A backtest lets you observe a strategy's volatility with full knowledge of the outcome. Live trading doesn't. That gap matters more than most people expect.
A common failure mode in systematic trading is shutting down a strategy prematurely in live trading because the short-term volatility feels worse than the backtest suggested. The backtest isn’t wrong — it’s that observing a five-year equity curve is a fundamentally different experience from living through its daily path without knowing the outcome.
The standard backtesting pitfalls — data snooping, overfitting, look-ahead bias — are well-covered elsewhere. What’s less discussed is a subtler form of look-ahead bias: using the emotional comfort of knowing the long-run outcome to convince yourself you can tolerate the short-term path. When you’re looking at a 5-year equity curve, it’s easy to contextualise a 40% drawdown as “just a dip before the recovery.” When you’re living through it in real time, with real money, you don’t have that foresight. You have a screen full of red and the creeping suspicion that your model is wrong.
Call this risk tolerance look-ahead bias: the gap between how much volatility you think you can handle (informed by the backtest’s full trajectory) and how much you can actually stomach when you’re experiencing it day by day without knowing what comes next.
A Case Study: Heating Oil in a Trend-Following Strategy
Consider a trend-focused strategy that’s been backtested thoroughly, with PnL graphs that look like this:
We’re looking at 51 years of data. As you can see, there are ups and downs — slow and fast alike. Somewhere in there is a 1-day loss of over 10% of the starting capital. During another period, there’s a 100% gain followed by a sharp mean-reversion resulting in a 40% peak-to-valley loss. When you look at the graph — which shows the long-run behaviour of the strategy — you might say to yourself “yeah, this will be volatile, but I can stick with it because I know the long run results will be great.”
But once you’re trading live, with real money, are you sure you’d be this confident?
The strategy included the CME Heating Oil contract — for a typical retail or small-PM account, an uncomfortably large position. At current rates it exposes the buyer (or seller) to a daily 1 standard deviation move of around $3.5k USD. That’s the expected daily 1 standard deviation of the entire strategy not including CME Heating Oil. Other contracts in the strategy had the same oversized-position issue.
That’s the kind of reasoning that feels airtight at the whiteboard. It looks very different once the strategy goes live with real positions — long CME:HO alongside five other grown-up-sized contracts.
Day 1: By late morning, the position is down $2k. The internal monologue: “That’s just a few hours in — I can’t draw conclusions from two hours of price action.”
Day 2: The position is being checked twice as often. By mid-morning, HO is down $3k. The internal monologue: “This is starting to not be enjoyable — I guess this is how it’s going to be.” The backtests are pulled up again: the 95th percentile 1-day VaR is , and a 5 standard deviation move based on the observed returns distribution is .
Day 3: HO is flat. But the position is being checked several times per hour. The internal state: dreading a third loss of the same magnitude.
Day 4: HO has resumed selling off. The questions start: “Is the backtest actually representative?” “What if this was just a lucky period for the strategy?” “If I could see into the future and know whether things get better, I’d happily deal with this.”
Day 5: HO begins to sell off at lunch. That’s enough — logically, the backtest says this can happen, but experiencing it in real time is too much to bear. The position is closed at market. The search begins for smaller contracts.
What Can We Do to Avoid It?
Three practices can help close the gap between backtest-inferred risk tolerance and realized risk tolerance:
Simulate the portfolio. Use a paper trading account for a few weeks to experience the feeling of not knowing what’s coming next. It doesn’t solve the issue entirely as we know intuitively that the PnL won’t be real, but it certainly helps.
Start much smaller. A common mistake is going live at full size immediately. When going live with a strategy, start as small as possible. Rather than a big CME futures contract, consider using the Mini or Micro contract. Take 1-lot positions instead of 5-lot clips. Validate the expected behaviour, get acquainted with the daily ebb and flow, and resolve any post-go-live issues during this period.
Phase in with a hedge overlay. When you go live, put on a hedge for your initial set of positions for the first day or two with another asset or contract which is inversely correlated. This will allow you to be trading live at your small initial size without much risk. As you get more confident, you can reduce or entirely unwind the hedge. Once you’re ready and comfortable, ramp up the size over time.
There are, of course, numerous other ways to not freak out — but perhaps these three are a good starting point for your own approach.
Footnotes
-
For a backtest, 5 years is not a long time. 10+ years is ideal, with varied market regimes thrown in. But for the purpose of this article, the length of the backtest isn’t the topic of discussion. ↩