Volume 6 • Issue 1 • PP: 29–35 • 2026
Exact Bias–Variance Decomposition and Degrees of Freedom in Ridge Regression: Theory, Verification, and a Disease-Progression Application
Open Access & Copyright
© 2026 The Author(s). Published by ASPG. This article is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0).
View PDF Abstract
Ridge regression estimates β in the linear model y = Xβ +ε by βˆ (λ) = argminβ ∥y−Xβ∥2+λ∥β∥2, trading bias for variance as λ increases. This paper collects six results about βˆ (λ) into a single self-contained development, each proved and then checked numerically. The estimator is written in closed form through the singular value decomposition of X; its effective degrees of freedom, df(λ)=Σj d2j /(d2j +λ), are shown to be strictly decreasing and convex in λ; its exact bias and variance are derived in closed form; a strictly positive λ is shown always to exist that reduces mean squared estimation error below that of ordinary least squares whenever the noise variance is positive; the estimator is shown to coincide with the posterior mean under a Gaussian prior with precision proportional to λ; and the leave-one-out cross-validation error is shown to admit a closed-form shortcut that generalized cross-validation approximates by averaging its leverage terms. Every derived quantity is verified against data: the leave-one-out shortcut matches brute-force refitting exactly, and a calibrated Monte Carlo simulation confirms the closed-form bias and variance to within simulation error at every tested λ. Applied to a standard diabetes disease-progression dataset (n = 442, ten predictors), the theoretical construction correctly locates a strictly risk-reducing regularization region, and repeated cross-validation shows ridge, lasso, and elastic net all lying within one standard error of ordinary least squares in out-of-sample prediction error—consistent with the closed-form theory, which attributes the available gain to reduced parameter-estimation risk on a well-conditioned design rather than to prediction-error reduction.
Keywords
References
[1] P. L. Bartlett, P. M. Long, G. Lugosi, and A. Tsigler, “Benign overfitting in linear regression,” Proceedings of the National Academy of Sciences, vol. 117, no. 48, pp. 30 063–30 070, 2020.
[2] T. Hastie, A. Montanari, S. Rosset, and R. J. Tibshirani, “Surprises in high-dimensional ridgeless least squares interpolation,” The Annals of Statistics, vol. 50, no. 2, pp. 949–986, 2022.
[3] T. Liang and A. Rakhlin, “Just interpolate: Kernel “ridgeless” regression can generalize,” The Annals of Statistics, vol. 48, no. 3, pp. 1329–1347, 2020.
[4] D. Richards, J. Mourtada, and L. Rosasco, “Asymptotics of ridge(less) regression under general source condition,” in Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS), ser. PMLR, vol. 130, 2021, pp. 3889–3897.
[5] A. Tsigler and P. L. Bartlett, “Benign overfitting in ridge regression,” Journal of Machine Learning Research, vol. 24, no. 123, pp. 1–76, 2023.
[6] P. Patil, Y. Wei, A. Rinaldo, and R. Tibshirani, “Uniform consistency of cross-validation estimators for highdimensional ridge regression,” in Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS), ser. PMLR, vol. 130, 2021, pp. 3178–3186.
[7] K. Rahnama Rad and A. Maleki, “A scalable estimate of the out-of-sample prediction error via approximate leave-one-out cross-validation,” Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 82, no. 4, pp. 965–996, 2020.
[8] P. Patil and D. LeJeune, “Asymptotically free sketched ridge ensembles: Risks, cross-validation, and tuning,” arXiv preprint arXiv:2310.04357, 2023.
[9] D. Kobak, J. Lomond, and B. Sanchez, “The optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization,” Journal of Machine Learning Research, vol. 21, no. 169, pp. 1–16, 2020.
[10] P. Patil and J.-H. Du, “Generalized equivalences between subsampling and ridge regularization,” in Advances in Neural Information Processing Systems, vol. 36, 2023.
[11] J.-H. Du, P. Patil, and A. K. Kuchibhotla, “Subsample ridge ensembles: Equivalences and generalized crossvalidation,” in Proceedings of the 40th International Conference on Machine Learning, ser. PMLR, vol. 202, 2023, pp. 8585–8631.
[12] P. Patil, J.-H. Du, and A. K. Kuchibhotla, “Bagging in overparameterized learning: Risk characterization and risk monotonization,” Journal of Machine Learning Research, vol. 24, no. 319, pp. 1–113, 2023.
[13] E. Dobriban and Y. Sheng, “Distributed linear regression by averaging,” The Annals of Statistics, vol. 49, no. 2, pp. 918–943, 2021.
[14] B. Adlam and J. Pennington, “Understanding double descent requires a fine-grained bias-variance decomposition,” Advances in Neural Information Processing Systems, vol. 33, pp. 11 022–11 032, 2020.
[15] B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, “Least angle regression,” The Annals of Statistics, vol. 32, no. 2, pp. 407–499, 2004.
[16] M. Usman, S. Doguwa, and B. Alhaji, “Comparing the prediction accuracy of ridge, lasso and elastic net regression models with linear regression using breast cancer data,” Bayero Journal of Pure and Applied Sciences, vol. 14, no. 2, pp. 134–149, 2022.
[17] N. Herawati, A. Wijayanti, A. Sutrisno, Nusyirwana, and Misgiyatia, “The performance of ridge regression, lasso, and elastic-net in controlling multicollinearity: A simulation and application,” Journal of Modern Applied Statistical Methods, vol. 23, no. 2, 2024.
[18] S. Rosset and R. J. Tibshirani, “From fixed-X to random- X regression: Bias-variance decompositions, covariance penalties, and prediction error estimation,” Journal of the American Statistical Association, vol. 115, no. 529, pp. 138–151, 2020.
[19] G. Tzoumerkas, D. Fouskakis, and I. Ntzoufras, “A comparison of power-expected-posterior priors in shrinkage regression,” Journal of Statistical Theory and Practice, vol. 16, p. 61, 2022.
[20] L. Hucker and M. Wahl, “A note on the prediction error of principal component regression in high dimensions,” Theory of Probability and Mathematical Statistics, vol. 109, pp. 37–53, 2023.
Cite This Article
Choose your preferred format
Publisher's Note
The statements, opinions, and data presented in this article are solely those of the author(s) and do not necessarily represent those of ASPG, the journal, or its editors. ASPG and the editors disclaim responsibility for any harm arising from the use of any ideas, methods, instructions, or products described in this article, to the fullest extent permitted by applicable law.