Exact Bias–Variance Decomposition and Degrees of Freedom in

Ridge Regression: Theory, Verification, and

a Disease-Progression Application

Dwi Agustin Retnowardani1,*

1 Universitas PGRI Argopuro Jember, Indonesia

Received: September 21, 2025 Revised: November 18, 2025 Accepted: January 08, 2026 ⋆ Corresponding author

ABSTRACT

Ridge regression estimates β in the linear model y = Xβ +ε by βˆ (λ) = argminβ ∥y−Xβ∥2+λ∥β∥2, trading bias

for variance as λ increases. This paper collects six results about βˆ (λ) into a single self-contained development,

each proved and then checked numerically. The estimator is written in closed form through the singular value

decomposition of X; its effective degrees of freedom, df(λ)=Σj d2j

/(d2j

+λ), are shown to be strictly decreasing and

convex in λ; its exact bias and variance are derived in closed form; a strictly positive λ is shown always to exist that

reduces mean squared estimation error below that of ordinary least squares whenever the noise variance is positive;

the estimator is shown to coincide with the posterior mean under a Gaussian prior with precision proportional to λ;

and the leave-one-out cross-validation error is shown to admit a closed-form shortcut that generalized cross-validation

approximates by averaging its leverage terms. Every derived quantity is verified against data: the leave-one-out

shortcut matches brute-force refitting exactly, and a calibrated Monte Carlo simulation confirms the closed-form bias

and variance to within simulation error at every tested λ. Applied to a standard diabetes disease-progression dataset

(n = 442, ten predictors), the theoretical construction correctly locates a strictly risk-reducing regularization region,

and repeated cross-validation shows ridge, lasso, and elastic net all lying within one standard error of ordinary least

squares in out-of-sample prediction error—consistent with the closed-form theory, which attributes the available

gain to reduced parameter-estimation risk on a well-conditioned design rather than to prediction-error reduction.

Keywords: Ridge regression Bias–variance decomposition Degrees of freedom Generalized cross-validation

Bayesian regularization Singular value decomposition

1. INTRODUCTION

Ridge regression regularizes ordinary least squares by adding

an ℓ2 penalty on the coefficient vector, trading an increase

in bias for a reduction in variance. The idea is old, but the

precise mathematical structure of that trade-off – how bias

and variance depend on the regularization parameter, when

a strictly positive parameter is guaranteed to help, what the

resulting estimator’s effective complexity is, and how that

complexity should be chosen from data – is still an active

area of research, particularly as the classical fixed-dimension,

large-sample regime gives way to high-dimensional and even

proportional asymptotics in modern applications [1, 2, 3].

Recent work has substantially sharpened this picture: the risk

of ridge regression and its interpolating (zero-penalty) limit

is now understood precisely in high-dimensional randomdesign

settings [2, 4, 5], the choice of regularization by crossvalidation

has been shown to be asymptotically consistent