Overfit
144 signal tests, two winners, no edge
The idea
Before building strategies, ask a simpler question: does any common indicator predict next-period returns on ETFs at all?
What we did
- 24 daily indicators (returns over several lookbacks, momentum, moving-average ratios, RSI, Bollinger z-score, drawdown from high, volatility, volume ratios).
- Each tested at 1-, 5- and 20-day horizons, two ways (per-ETF over time, and ranking ETFs against each other each day). That is 144 tests on 23 ETFs over the training years 2005–2016. The screen was exploratory: it also looked at data from 2022-02-01 on, so that later window was not sealed.
- Guards against a broken pipeline: a "leakage canary" (feeding the answer in as an indicator must be flagged, and it was) and a null where labels are shifted in time (500 shifts).
What happened
Two tests passed a per-test false-discovery check (Benjamini–Hochberg, q = 0.10): the 5-day return and the 2-day RSI, both as short-term reversal signals at the 5-day horizon. Their predictive correlations (IC) were small: −0.061 and −0.046.
But the right comparison is the largest effect across all 144 tests against the largest you'd see by chance. It was not beaten: the biggest observed effect sat inside the noise (p = 0.22 for the per-ETF test, 0.38 for the cross-ETF test).
The full results
The figures below are the 144 tests on the 2005–2016 training years. The screen was exploratory and also looked at data from 2022-02-01 on; that later window was not sealed, and we are not publishing those numbers. IC is the correlation between an indicator today and the return over the next 1, 5 or 20 days; a negative IC means "recently up tends to mean next-period down." The per-fund tests run each ETF over time; the cross-ETF tests rank the ETFs against each other each day.
The 15 tests with the smallest p-values. The two that passed the per-test false-discovery check (q = 0.10) are highlighted:
| Indicator | Test | Horizon | IC | p-value | FDR q |
|---|---|---|---|---|---|
| RSI (2-day) Passed FDR | Per fund | 5 days | −0.046 | 0.001 | 0.10 |
| 5-day return Passed FDR | Per fund | 5 days | −0.061 | 0.001 | 0.10 |
| 1-day return | Per fund | 5 days | −0.037 | 0.003 | 0.12 |
| distance below 12-month high | Per fund | 1 days | −0.033 | 0.004 | 0.12 |
| Bollinger z-score (20-day) | Per fund | 5 days | −0.053 | 0.005 | 0.12 |
| 14-day average range | Across funds | 5 days | 0.042 | 0.005 | 0.12 |
| 14-day average range | Across funds | 1 days | 0.020 | 0.010 | 0.20 |
| 14-day average range | Across funds | 20 days | 0.070 | 0.012 | 0.20 |
| 20-day volatility | Per fund | 20 days | 0.106 | 0.014 | 0.20 |
| RSI (2-day) | Across funds | 1 days | −0.015 | 0.015 | 0.20 |
| 3-month return per unit of risk | Per fund | 1 days | −0.026 | 0.015 | 0.20 |
| 20-day volatility | Per fund | 5 days | 0.059 | 0.017 | 0.20 |
| RSI (14-day) | Per fund | 5 days | −0.046 | 0.020 | 0.20 |
| 14-day average range | Per fund | 1 days | 0.025 | 0.020 | 0.20 |
| 20-day volatility | Across funds | 5 days | 0.035 | 0.021 | 0.20 |
Why it failed
The false-discovery screen already adjusted for running 144 tests, and two small effects still passed it (q = 0.10). A stricter, family-wide check then asked whether even the largest of the 144 beat what chance would produce, and it did not: the biggest result sat inside the noise (a p-value of 0.22 for the per-fund test, 0.38 for the cross-ETF test). Those are two different standards, not a case of forgetting to count the tests. We treated this only as “weekly reversal is worth a costed test,” and that test has its own page.
Receipts: 144 trials were recorded in our trial registry.