Overfit

Indicator families and pairs: best was good, not distinguishable from luck

· 3 min read

The idea

Instead of searching millions of combinations, group the 24 indicators into families (momentum, trend, oscillators, risk and so on) and test each family and each pair of families: 28 books.

What happened

Training years Sharpe Worst drop Invested
Best book (momentum plus risk) 0.62 30% about 94%
SPY buy and hold 0.41 55% 100%

Deflated Sharpe 0.943 (bar 0.95). Probability of backtest overfitting 0.47 (bar 0.20). Costs at zero and double barely moved the ranking.

The full results

All 28 books, default costs, 2005–2016 training window. Each book holds the ETFs that score above the median on its indicator family (or the average of two families), rebalances every 21 sessions, and keeps idle cash in T-bills; Sharpe, return and worst drop credit that cash. SPY buy and hold on the same window: Sharpe 0.41, 7.47% a year, 55.2% worst drop.

Histogram of the Sharpe ratios of all 28 family and pair books, with SPY buy and hold, the median book and the best book marked.
25 of 28 books scored above SPY buy and hold, and the best (0.62) is only a little above the pack. The median book was invested 92% of the time, so a lot of the story is market exposure with somewhat lower drops. The batch-level gates failed: deflated Sharpe 0.943 (bar 0.95) and probability of overfitting 0.47 (bar 0.20).
BookSharpeYearly returnWorst dropTime investedSharpe, zero costsSharpe, double spread
Pair: momentum + risk Best0.627.56%30.1%94%0.630.60
Family: risk0.617.05%28.8%94%0.630.60
Family: momentum0.588.35%29.7%95%0.590.57
Pair: momentum + liquidity0.567.93%34.4%93%0.580.54
Family: trend0.568.20%34.2%95%0.570.55
Pair: trend + risk0.567.02%29.4%94%0.570.54
Pair: risk + liquidity0.566.92%30.3%92%0.580.53
Pair: momentum + trend0.558.00%31.0%95%0.560.54
All 28 books, best first
BookSharpeYearly returnWorst dropTime investedSharpe, zero costsSharpe, double spread
Pair: momentum + risk Best0.627.56%30.1%94%0.630.60
Family: risk0.617.05%28.8%94%0.630.60
Family: momentum0.588.35%29.7%95%0.590.57
Pair: momentum + liquidity0.567.93%34.4%93%0.580.54
Family: trend0.568.20%34.2%95%0.570.55
Pair: trend + risk0.567.02%29.4%94%0.570.54
Pair: risk + liquidity0.566.92%30.3%92%0.580.53
Pair: momentum + trend0.558.00%31.0%95%0.560.54
Pair: momentum + oscillator0.538.23%35.7%93%0.550.52
Pair: momentum + legacy0.517.49%33.7%93%0.530.49
Pair: reversal + momentum0.487.36%36.6%93%0.500.47
Pair: reversal + trend0.477.44%38.1%92%0.490.46
Pair: reversal + liquidity0.477.47%40.2%91%0.490.45
Pair: trend + legacy0.466.83%37.0%92%0.480.44
Family: reversal0.467.52%43.4%90%0.480.44
Pair: liquidity + legacy0.456.85%44.8%91%0.470.44
Family: liquidity0.456.85%37.3%90%0.470.43
Pair: oscillator + risk0.446.31%36.3%91%0.460.42
Pair: trend + liquidity0.436.42%38.9%93%0.450.42
Pair: reversal + risk0.436.11%34.0%92%0.450.41
Pair: risk + legacy0.435.79%37.1%92%0.450.41
Family: legacy0.426.45%41.7%91%0.440.40
Pair: trend + oscillator0.426.62%41.0%93%0.430.40
Pair: oscillator + liquidity0.426.71%42.0%90%0.440.40
Pair: reversal + oscillator0.416.69%45.2%90%0.420.39
Pair: reversal + legacy0.396.24%45.4%91%0.410.37
Family: oscillator0.386.32%45.2%90%0.390.36
Pair: oscillator + legacy0.355.62%46.3%91%0.360.33

Why it failed

The gates are about the whole batch, not the leader: you can't tell the best of 28 apart from the luckiest of 28. The deflated Sharpe was 0.943 against a 0.95 bar, and the probability of overfitting was 0.47 against a 0.20 bar. No later confirmatory years were opened.

Receipts: the plan was written down and committed first. 28 official trials.