Full Length Article
DOI: https://doi.org/10.54216/PAMDA.060105
Exact Bias–Variance Decomposition and Degrees of Freedom in Ridge Regression: Theory, Verification, and a Disease-Progression Application
Ridge regression estimates β in the linear model y = Xβ +ε by βˆ (λ) = argminβ ∥y−Xβ∥2+λ∥β∥2, trading bias for variance as λ increases. This paper collects six results about βˆ (λ) into a single self-contained development, each proved and then checked numerically. The estimator is written in closed form through the singular value decomposition of X; its effective degrees of freedom, df(λ)=Σj d2j /(d2j +λ), are shown to be strictly decreasing and convex in λ; its exact bias and variance are derived in closed form; a strictly positive λ is shown always to exist that reduces mean squared estimation error below that of ordinary least squares whenever the noise variance is positive; the estimator is shown to coincide with the posterior mean under a Gaussian prior with precision proportional to λ; and the leave-one-out cross-validation error is shown to admit a closed-form shortcut that generalized cross-validation approximates by averaging its leverage terms. Every derived quantity is verified against data: the leave-one-out shortcut matches brute-force refitting exactly, and a calibrated Monte Carlo simulation confirms the closed-form bias and variance to within simulation error at every tested λ. Applied to a standard diabetes disease-progression dataset (n = 442, ten predictors), the theoretical construction correctly locates a strictly risk-reducing regularization region, and repeated cross-validation shows ridge, lasso, and elastic net all lying within one standard error of ordinary least squares in out-of-sample prediction error—consistent with the closed-form theory, which attributes the available gain to reduced parameter-estimation risk on a well-conditioned design rather than to prediction-error reduction.
Dwi Retnowardani
visibility
105
download
37