Overfit

144 signal tests, two winners, no edge

· 3 min read

The idea

Before building strategies, ask a simpler question: does any common indicator predict next-period returns on ETFs at all?

What we did

  • 24 daily indicators (returns over several lookbacks, momentum, moving-average ratios, RSI, Bollinger z-score, drawdown from high, volatility, volume ratios).
  • Each tested at 1-, 5- and 20-day horizons, two ways (per-ETF over time, and ranking ETFs against each other each day). That is 144 tests on 23 ETFs over the training years 2005–2016. The screen was exploratory: it also looked at data from 2022-02-01 on, so that later window was not sealed.
  • Guards against a broken pipeline: a "leakage canary" (feeding the answer in as an indicator must be flagged, and it was) and a null where labels are shifted in time (500 shifts).

What happened

Two tests passed a per-test false-discovery check (Benjamini–Hochberg, q = 0.10): the 5-day return and the 2-day RSI, both as short-term reversal signals at the 5-day horizon. Their predictive correlations (IC) were small: −0.061 and −0.046.

But the right comparison is the largest effect across all 144 tests against the largest you'd see by chance. It was not beaten: the biggest observed effect sat inside the noise (p = 0.22 for the per-ETF test, 0.38 for the cross-ETF test).

The full results

The figures below are the 144 tests on the 2005–2016 training years. The screen was exploratory and also looked at data from 2022-02-01 on; that later window was not sealed, and we are not publishing those numbers. IC is the correlation between an indicator today and the return over the next 1, 5 or 20 days; a negative IC means "recently up tends to mean next-period down." The per-fund tests run each ETF over time; the cross-ETF tests rank the ETFs against each other each day.

Two histograms of the training-years IC of the 72 per-fund and 72 cross-ETF tests, with the chance bar marked.
Almost every IC is tiny. The largest |IC| was 0.106 for the per-fund tests and 0.070 for the cross-ETF tests, both inside the chance bars (p = 0.22 and 0.38). None of the 144 tests went past the chance bar.

The 15 tests with the smallest p-values. The two that passed the per-test false-discovery check (q = 0.10) are highlighted:

IndicatorTestHorizonICp-valueFDR q
RSI (2-day) Passed FDRPer fund5 days−0.0460.0010.10
5-day return Passed FDRPer fund5 days−0.0610.0010.10
1-day returnPer fund5 days−0.0370.0030.12
distance below 12-month highPer fund1 days−0.0330.0040.12
Bollinger z-score (20-day)Per fund5 days−0.0530.0050.12
14-day average rangeAcross funds5 days0.0420.0050.12
14-day average rangeAcross funds1 days0.0200.0100.20
14-day average rangeAcross funds20 days0.0700.0120.20
20-day volatilityPer fund20 days0.1060.0140.20
RSI (2-day)Across funds1 days−0.0150.0150.20
3-month return per unit of riskPer fund1 days−0.0260.0150.20
20-day volatilityPer fund5 days0.0590.0170.20
RSI (14-day)Per fund5 days−0.0460.0200.20
14-day average rangePer fund1 days0.0250.0200.20
20-day volatilityAcross funds5 days0.0350.0210.20

Why it failed

The false-discovery screen already adjusted for running 144 tests, and two small effects still passed it (q = 0.10). A stricter, family-wide check then asked whether even the largest of the 144 beat what chance would produce, and it did not: the biggest result sat inside the noise (a p-value of 0.22 for the per-fund test, 0.38 for the cross-ETF test). Those are two different standards, not a case of forgetting to count the tests. We treated this only as “weekly reversal is worth a costed test,” and that test has its own page.

Receipts: 144 trials were recorded in our trial registry.