Class OverfitProbability

java.lang.Object

com.quantfinlib.backtest.validation.OverfitProbability

public final class OverfitProbability
extends Object

PROBABILITY OF BACKTEST OVERFITTING via combinatorially symmetric cross-validation — CSCV (Bailey, Borwein, Lopez de Prado & Zhu 2015, "The probability of backtest overfitting").

SharpeValidation asks whether ONE track record is luck. This class asks the prior question: is the SELECTION PROCESS itself broken? When a desk tries N parameter sets and reports the best, the reported Sharpe is a maximum of N draws — and the right diagnostic is: how often does the in-sample winner turn out to be a BELOW-MEDIAN performer out of sample?

The construction: take the T×N matrix of per-period returns (one column per strategy variant), slice time into S equal blocks, and form every one of the C(S, S/2) ways to pick half the blocks as in-sample (IS) and the complementary half as out-of-sample (OOS) — symmetric by construction, so IS and OOS have identical length and no arrow of time bias. For each combination:

1. concatenate the IS blocks and pick the variant with the best IS objective (ties break to the first column — stated, and the tie ranks below make that conservative);
2. rank that winner's OOS objective among all N variants: rank = 1 + #(strictly worse), relative rank w = rank / (N + 1) (never exactly 0 or 1);
3. record the logit lambda = ln(w / (1 - w)) — positive means the IS winner was above the OOS median, negative below.

PBO = the fraction of combinations with lambda <= 0: the probability that the config you would have picked is an out-of-sample loser. Rules of thumb: PBO < 0.1 — selection is finding something real; PBO ≥ 0.5 — the selection is pure noise-mining and the "best" backtest is meaningless regardless of how good it looks.

Trailing periods that don't fill a whole block are dropped (stated: with T = 1007 and S = 8, each block is 125 periods and the last 7 are unused). S is capped at 16 — C(16,8) = 12,870 combinations is already a full re-scoring of every variant 12,870 times; beyond that the cost explodes for no statistical gain. Deterministic (no RNG), research lane.

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static final record	OverfitProbability.Result

Method Summary

Modifier and Type	Method	Description
static OverfitProbability.Result	cscv(double[][] returns, int blocks, ToDoubleFunction<double[]> objective)	CSCV with the caller's objective (applied to a concatenated return sub-series; higher is better).
static OverfitProbability.Result	cscvSharpe(double[][] returns, int blocks)	CSCV with the per-period Sharpe objective mean / stdDev (sample standard deviation; a zero-variance sub-series scores 0 — a flat line has no risk-adjusted evidence either way).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

cscv

public static OverfitProbability.Result cscv(double[][] returns,
 int blocks,
 ToDoubleFunction<double[]> objective)

CSCV with the caller's objective (applied to a concatenated return sub-series; higher is better).

Parameters:

returns - T×N rectangular matrix: returns[t][j] = period-t return of strategy variant j; all finite, N ≥ 2

blocks - S: even, 4 ≤ S ≤ 16; each block needs ≥ 2 periods

objective - score for a variant's return series, e.g. mean or per-period Sharpe; higher is better, must be finite

cscvSharpe

public static OverfitProbability.Result cscvSharpe(double[][] returns,
 int blocks)

CSCV with the per-period Sharpe objective mean / stdDev (sample standard deviation; a zero-variance sub-series scores 0 — a flat line has no risk-adjusted evidence either way).