Fact-checked by Grok 2 weeks ago

Elastic net regularization

Elastic net regularization is a hybrid penalized regression technique that combines the L1 penalty from lasso regression, which induces sparsity by setting some coefficients to zero, and the L2 penalty from ridge regression, which shrinks coefficients toward zero without eliminating them, to simultaneously perform variable selection and handle multicollinearity in high-dimensional data.^[1] Proposed by Hui Zou and Trevor Hastie in 2005, it addresses key shortcomings of its predecessors: unlike lasso, which often arbitrarily selects only one variable from highly correlated groups, elastic net promotes a grouping effect where correlated predictors tend to be included or excluded together; unlike ridge, it produces sparse models suitable for interpretation.^[2] The mathematical formulation of elastic net for linear regression minimizes the residual sum of squares plus a combined penalty:

\hat{\beta} = \arg\min_{\beta} \left\{ \|y - X\beta\|_2^2 + \lambda \left[ \alpha \|\beta\|_1 + \frac{1 - \alpha}{2} \|\beta\|_2^2 \right] \right\},

where \lambda \geq 0 controls the overall regularization strength, \alpha \in [0, 1] balances the L1 and L2 components (\alpha = 1 recovers lasso, \alpha = 0 recovers ridge), y is the response vector, X is the design matrix, and \beta is the coefficient vector.^[2] An efficient algorithm, LARS-EN (Least Angle Regression for Elastic Net), computes the entire regularization path for a sequence of \lambda values in computational time comparable to a single ordinary least squares fit, enabling cross-validation for optimal parameter selection.^[1] Empirical studies demonstrate that elastic net often outperforms lasso and ridge in prediction accuracy, with error reductions of 18% to 27% in simulations involving correlated predictors, and it uniquely handles scenarios where the number of features p exceeds the sample size n by selecting up to all p variables while maintaining stability.^[2] These properties make it particularly valuable in applications such as gene expression analysis for cancer classification, where datasets feature thousands of correlated genetic markers and limited samples, as well as in financial modeling and chemometrics.^[1] Extensions of elastic net beyond ordinary linear regression include generalized linear models (e.g., logistic for binary outcomes, Poisson for count data), Cox proportional hazards for survival analysis, and multinomial regression, all computed via coordinate descent in the widely used R package glmnet, which fits models across a grid of \lambda values and supports family-specific penalties.^[3] This implementation has facilitated its adoption in diverse fields, including bioinformatics and machine learning, where it balances bias-variance trade-offs to enhance model generalizability.^[3]

Background and Motivation

Overview of Regularization in Machine Learning

Regularization in machine learning is a technique that addresses the challenge of model complexity by incorporating a penalty term into the loss function, which discourages overly complex models and promotes simpler, more generalizable solutions. In linear regression, the standard ordinary least squares (OLS) approach minimizes the sum of squared residuals to fit a model to the training data, but this can lead to high variance in parameter estimates, especially when the number of features approaches or exceeds the number of observations. Overfitting occurs when a linear regression model captures noise and idiosyncrasies in the training data rather than the underlying pattern, resulting in excellent performance on training samples but poor generalization to unseen data, while underfitting happens when the model is too simplistic and fails to capture the relevant relationships, leading to high bias and inadequate fit on both training and test data. These issues are particularly pronounced in scenarios with high-dimensional data or correlated predictors, where unregularized models may produce unstable coefficient estimates.^[4] The concept of regularization was pioneered in the late 1960s and early 1970s, with ridge regression introduced by Hoerl and Kennard in 1970 as a method to stabilize estimates in the presence of multicollinearity by adding an L2 penalty term.^[4] This approach marked a shift toward biased estimation techniques that trade some unbiasedness for reduced variance, laying the groundwork for subsequent developments in the 1980s and beyond. Regularization offers several key benefits, including improved generalization by reducing the risk of overfitting, enhanced model stability through shrinkage of coefficients, and effective handling of multicollinearity, where correlated predictors inflate variance in OLS estimates.^[4] Common penalty types include the L0 norm, which counts the number of non-zero coefficients and promotes extreme sparsity but is computationally challenging; the L1 norm (absolute value sum), which induces sparsity by driving some coefficients exactly to zero, aiding feature selection; and the L2 norm (squared sum), which shrinks coefficients toward zero without eliminating them, providing smoother control over model complexity.^[5]^[4] Elastic net regularization combines L1 and L2 penalties to leverage the strengths of both in a hybrid manner.

Limitations of Lasso and Ridge Regression

Lasso regression, introduced by Robert Tibshirani in 1996,^[6] which employs the L1 penalty, achieves sparsity by setting some coefficients to exactly zero, enabling automatic variable selection. However, it exhibits instability when predictors are highly correlated, as it tends to select only one variable from a group of correlated features while ignoring the others, leading to biased and inconsistent predictions.^[7] This limitation is particularly evident in high-dimensional settings where the number of predictors exceeds the sample size (p > n), where Lasso selects at most n variables, potentially overlooking important grouped effects.^[7] Ridge regression, utilizing the L2 penalty, effectively handles multicollinearity by shrinking coefficients toward zero, which stabilizes estimates in the presence of correlated predictors. Despite this, it does not produce sparse models, as no coefficients are driven exactly to zero, resulting in dense solutions that include all variables and reduce interpretability.^[7] Empirical simulations demonstrate that Ridge often yields higher mean squared error compared to methods that incorporate sparsity, especially when true underlying models involve variable groups.^[7] These drawbacks—Lasso's poor handling of correlated groups and Ridge's lack of sparsity—highlight the need for a hybrid approach that combines the strengths of both penalties to achieve consistent variable selection with grouping effects.^[7]

Mathematical Formulation

Objective Function and Parameters

Elastic net regularization addresses the limitations of individual L1 and L2 penalties by combining them in a single objective function for linear regression. The standard formulation minimizes the residual sum of squares augmented with this hybrid penalty:

\hat{\beta} = \arg\min_{\beta} \ \frac{1}{2n} \| y - X \beta \|_2^2 + \lambda \left( \alpha \| \beta \|_1 + \frac{1 - \alpha}{2} \| \beta \|_2^2 \right)

Here, n denotes the number of observations, X \in \mathbb{R}^{n \times p} is the design matrix of predictor variables, y \in \mathbb{R}^n is the response vector, \| \beta \|_1 = \sum_{j=1}^p |\beta_j| is the L1 norm promoting sparsity, and \| \beta \|_2^2 = \sum_{j=1}^p \beta_j^2 is the squared L2 norm encouraging coefficient shrinkage.^[2] The parameters \lambda > 0 and \alpha \in [0, 1] control the penalty's behavior. The regularization strength \lambda scales the overall penalty term and is typically selected via cross-validation to optimize predictive performance by trading off bias and variance. The mixing parameter \alpha determines the relative contribution of the L1 and L2 penalties: when \alpha = 1, the objective recovers Lasso regression, emphasizing variable selection through sparsity; when \alpha = 0, it reduces to Ridge regression, which shrinks all coefficients toward zero without eliminating any but stabilizes estimates for multicollinear predictors.^[2] The initial or "naive" elastic net directly applies the penalty as formulated above, but it induces double shrinkage, where the L1 term selects variables while the L2 term further biases the magnitudes of the retained coefficients downward, leading to attenuated predictions. To mitigate this, the elastic net estimator rescales the naive solution: \hat{\beta}^{\text{elastic net}} = \left(1 + \frac{\lambda (1 - \alpha)}{2}\right) \hat{\beta}^{\text{naive}}. This adjustment corrects the shrinkage bias without altering the variable selection, improving the model's predictive accuracy while preserving sparsity.^[2] Intuitively, the elastic net derives from augmenting the Lasso penalty with an L2 term to incorporate Ridge's benefits: the L1 component drives exact zero coefficients for irrelevant or redundant variables, enabling feature selection, while the L2 component mitigates Lasso's tendency to arbitrarily select one variable from highly correlated groups by shrinking them similarly, thus promoting a grouping effect. This combination yields a more stable and interpretable model, particularly in high-dimensional settings with correlated features.^[2] Beyond linear regression, the elastic net extends to generalized linear models (GLMs) by replacing the least-squares loss with a deviance or negative log-likelihood appropriate to the response distribution. For instance, in logistic regression for binary outcomes, the objective minimizes the binomial deviance plus the elastic net penalty, allowing sparsity in classification tasks while handling correlated predictors.^[8]

Geometric Interpretation

The elastic net penalty combines the L1 norm from Lasso and the L2 norm from Ridge regression, creating a constraint region in the coefficient space that geometrically interpolates between a diamond-shaped L1 ball and a circular L2 ball. Specifically, the level sets of the penalty function P_\alpha(\beta) = \alpha \|\beta\|_1 + \frac{1-\alpha}{2} \|\beta\|_2^2 form a hybrid shape: for \alpha = 1, it reduces to the Lasso's diamond, which touches the coordinate axes at its vertices; for \alpha = 0, it is the Ridge's ellipse (circle in standardized coordinates). For intermediate \alpha (e.g., 0.5), the region becomes a tilted, convex polyhedron-like boundary that is stretched along directions away from the axes, with flat facets derived from the L1 component but smoothed and expanded by the L2 term. This structure avoids axis touches except at corners, similar to Lasso, but the overall form promotes solutions where multiple coefficients are nonzero and balanced.^[2] A useful analogy for understanding solutions involves contour plots of the quadratic loss function, which are elliptical and centered near the ordinary least squares estimate. The optimal elastic net coefficients occur at the point where these loss contours first touch the penalty boundary. When features are highly correlated, the loss ellipses elongate along the line where the corresponding coefficients are equal (e.g., \beta_j = \beta_k for variables j and k). The elastic net's boundary features facets oriented parallel to such lines, allowing the intersection to occur along a face rather than a sharp vertex, thereby setting the coefficients to similar values and encouraging their joint inclusion or exclusion. In contrast, Lasso's diamond-shaped boundary often intersects at a vertex on an axis, arbitrarily selecting one variable from the correlated group while zeroing others.^[2] This grouping effect is a key geometric advantage of the elastic net over Lasso, particularly for datasets with multicollinear predictors. In a 2D visualization of coefficient space for two correlated variables, the penalty constraint appears as a diamond distorted by the L2 term: its sides bow outward, reducing the tendency to hit axis corners and instead favoring contact midway along the sides, where \beta_1 \approx \beta_2. Theoretically, this is formalized by a bound on coefficient differences: for two standardized predictors with correlation \rho, the elastic net solution satisfies |\hat{\beta}_j - \hat{\beta}_k| \leq \frac{1}{\lambda_2 \sqrt{2(1 - \rho)}}, where \lambda_2 is the L2 penalty strength; as \rho \to 1, the bound approaches zero, forcing \hat{\beta}_j \approx \hat{\beta}_k and clustering the coefficients together.^[2]

Optimization and Algorithms

Coordinate Descent Approach

The coordinate descent algorithm serves as a primary optimization method for solving the elastic net problem, particularly effective for high-dimensional datasets where the number of features p exceeds the number of samples n. This approach iteratively optimizes the objective function by updating one coefficient \beta_j at a time while holding all other coefficients \beta_{-j} fixed, thereby reducing the multidimensional optimization to a series of univariate subproblems that can be solved in closed form. The algorithm cycles through the coefficients in a predetermined order, such as sequentially from j = 1 to p, and repeats these cycles until convergence criteria are met, such as a small change in the objective function value or residual norm.^[9] The explicit update rule for each coefficient \beta_j in the Gaussian linear regression case is given by

\beta_j \leftarrow \frac{S\left( \frac{1}{n} \mathbf{X}_j^T (\mathbf{y} - \mathbf{X}_{-j} \beta_{-j}), \lambda \alpha \right)}{1 + \lambda (1 - \alpha)},

where \mathbf{X}_j denotes the j-th column of the design matrix \mathbf{X}, \lambda > 0 is the regularization parameter, \alpha \in [0, 1] balances the \ell_1 and \ell_2 penalties, and S(\cdot, \cdot) is the soft-thresholding operator. This update arises from minimizing the elastic net objective with respect to \beta_j, combining the ridge-like \ell_2 shrinkage in the denominator with the lasso-like sparsity induction via soft-thresholding in the numerator. The soft-thresholding function is defined as

S(z, \gamma) = \operatorname{sign}(z) \left( |z| - \gamma \right)_+,

where \left( \cdot \right)_+ = \max(0, \cdot) and \gamma = \lambda \alpha, which subtracts the penalty threshold from the absolute value of the input and sets the result to zero if below the threshold, promoting sparsity by shrinking small coefficients to exactly zero.^[9] Under standard assumptions such as convexity of the elastic net objective (which holds for the squared-error loss and the penalties used), the coordinate descent algorithm is guaranteed to converge to the global minimum. This convergence is established for nondifferentiable convex minimization problems, making it reliable for the elastic net formulation even in high dimensions where p \gg n. The method's efficiency stems from its ability to exploit the structure of the penalties, often requiring only a few cycles to achieve practical convergence in sparse settings.^[9] A key practical extension involves path algorithms that compute the entire regularization path by solving the elastic net for a decreasing sequence of \lambda values, from an initial \lambda_{\max} (where all coefficients are zero) down to a small \lambda_{\min}, using warm starts from previous solutions to accelerate computation; this is exemplified in implementations like the glmnet package. Each full cycle through all p coordinates has a computational complexity of O(np), as the inner product \mathbf{X}_j^T (\mathbf{y} - \mathbf{X}_{-j} \beta_{-j}) can be updated incrementally using residuals, making the approach scalable for large-scale problems.^[9]

Least Angle Regression for Elastic Net (LARS-EN)

Another important algorithm for elastic net is the Least Angle Regression extension (LARS-EN), proposed in the original formulation. LARS-EN adapts the least angle regression algorithm from lasso to the elastic net penalty by incorporating the L2 term, allowing computation of the full regularization path for a sequence of \lambda values. It proceeds by starting with all coefficients zero and iteratively adding variables in a joint least angle fashion, where the direction of joint addition is adjusted to account for the ridge component, until the path is complete. This method achieves computational efficiency comparable to a single ordinary least squares fit, enabling fast cross-validation for parameter tuning, and is particularly useful for understanding the variable selection process in correlated settings.^[2]^[1]

Reduction to Support Vector Machines

Elastic net regularization admits a constrained formulation that facilitates its reduction to a support vector machine (SVM) problem, enabling the application of established SVM optimization techniques. Specifically, the elastic net can be expressed as minimizing the objective \|X\beta - y\|_2^2 + \lambda_2 \|\beta\|_2^2 subject to the constraint \|\beta\|_1 \leq t, where \lambda_2 > 0 controls the L2 penalty strength and t > 0 enforces the L1 ball constraint.^[2] This setup maps directly to the dual form of a quadratic SVM through variable scaling (\beta := \beta / t) and decomposition of \beta into non-negative components \beta^+ and \beta^-, transforming the problem into minimizing \|[X, -X] \hat{\beta} - y/t\|_2^2 + \lambda_2 \|\hat{\beta}\|_2^2 subject to \sum \hat{\beta}_i = 1, \hat{\beta} \geq 0.^[10] The exact equivalence is achieved by interpreting this as the dual of an SVM with squared hinge loss and L2 regularization on the dual variables \alpha, formulated as \min_{\alpha \geq 0} \|Z \alpha\|_2^2 + \frac{\lambda_2}{2} \sum \alpha_i^2 - 2 \sum \alpha_i subject to \alpha_i \geq 0, where Z is a transformed data matrix incorporating the data and labels; the elastic net coefficients are then recovered from the SVM dual solution \alpha^* as \hat{\beta}_j^* = t \cdot (\alpha_j^* - \alpha_{j+p}^*) / \sum_i \alpha_i^* for j = 1, \dots, p, with t the L1 budget.^[10] Although slack variables are inherent in the soft-margin SVM structure to handle constraint violations, this reduction effectively casts the elastic net's L1 sparsity via an L1-like penalty on the SVM weights in the primal-dual mapping.^[10] This reformulation offers significant benefits by leveraging mature SVM solvers, including interior-point methods and GPU-accelerated implementations, which can solve large-scale elastic net problems up to two orders of magnitude faster than traditional approaches like coordinate descent in certain parallel settings.^[10] For instance, on datasets with thousands of features, this enables efficient handling of high-dimensional regression without custom implementations.^[10] Despite these advantages, the SVM-based reduction is less intuitive for regression contexts, where the classification-oriented SVM framework may introduce unnecessary complexity, and coordinate descent remains preferred for its simplicity and speed in producing sparse solutions.^[10] The connection was formalized to bridge linear regression penalties with SVM optimization, building on the original elastic net proposal that highlighted parallels in variable selection for classification tasks.^[2]^[10]

Properties and Comparisons

Equivalence and Sparsity Properties

The elastic net achieves exact sparsity under a generalized irrepresentable condition (EIC), which requires that the infinity norm of a specific linear combination involving the submatrix of the correlation structure and the true coefficients satisfies ∥C_{21}(C_{11} + λ_2/n I)^{-1}(sign(β^{(1)}) + 2λ_2/λ_1 β^{(1)} )∥∞ ≤ 1 - η for some η > 0, where C{11} and C_{21} are blocks of the correlation matrix corresponding to active and inactive variables, respectively.^[11] This EIC is weaker than the irrepresentable condition for the lasso, as the latter is a special case when λ_2 = 0 and C_{11} is invertible; notably, the lasso's condition implies the existence of suitable λ_1 and λ_2 for the EIC to hold, but the converse is not true, allowing the elastic net to achieve sparsity in scenarios where the lasso fails, such as when the number of variables exceeds the sample size.^[11] A key sparsity property of the elastic net is its grouping effect, whereby highly correlated predictors tend to be selected together with coefficients of similar magnitude.^[12] This behavior arises from the ridge component (L2 penalty), which shrinks correlated coefficients toward each other, while the lasso component (L1 penalty) enforces overall sparsity; formally, for two predictors with correlation ρ, if \hat{\beta}_j(\lambda_1, \lambda_2) \hat{\beta}_k(\lambda_1, \lambda_2) > 0, the difference in their elastic net coefficients satisfies \frac{|\hat{\beta}_j - \hat{\beta}_k|}{||y||_1} \leq \frac{1}{\lambda_2 \sqrt{2(1 - \rho)}}, which approaches zero as λ_2 increases or ρ approaches 1, ensuring that groups of correlated variables are treated similarly unlike the lasso, which arbitrarily selects at most one from such groups.^[12] Under the EIC, the elastic net exhibits asymptotic equivalence to the lasso in variable selection, selecting the same true model with probability approaching 1 as the sample size n → ∞, provided λ_1 and λ_2 scale appropriately (e.g., λ_1 √n → ∞ and λ_1² n / log(p - q) → ∞ where p is the number of variables and q the true sparsity level).^[11] This equivalence holds because the ridge penalty stabilizes the solution without altering the support when correlations are not too strong, allowing the elastic net to mimic lasso sparsity in low-correlation regimes while improving performance elsewhere.^[11] The elastic net possesses oracle properties in its adaptive form, achieving consistency in both variable selection (selecting the true model with probability → 1) and estimation (asymptotic normality of non-zero coefficients at the oracle rate √n), similar to non-convex penalties like SCAD but with the advantage of convex optimization and simpler tuning via a single additional parameter. These properties require mild conditions on the design matrix and penalty weights, diverging with n to emphasize initial consistent estimates. The naive elastic net suffers from double shrinkage, where the L1 penalty selects variables but the preceding L2 shrinkage (equivalent to ridge on the active set) underestimates the magnitudes of the selected coefficients, leading to biased predictions.^[12] This underestimation is corrected by scaling the naive estimates: \hat{\beta}^{\text{(elastic net)}} = (1 + \lambda_2) \hat{\beta}^{\text{(naive elastic net)}}, which unbiasedly recovers the true magnitudes while preserving the sparse support, thereby improving predictive performance without additional computational cost.^[12]

Advantages Over Other Methods

Elastic net regularization excels in scenarios involving highly correlated predictors, where it promotes a grouping effect that selects correlated variables together, thereby preserving prediction accuracy while enhancing model interpretability through collective variable inclusion.^[12] This contrasts with Lasso, which tends to arbitrarily select only one variable from a correlated group, potentially leading to unstable and less interpretable models.^[12] By combining L1 and L2 penalties, elastic net achieves a superior balance in the bias-variance trade-off compared to its predecessors; it mitigates the high variance of Lasso in collinear datasets while avoiding the dense, non-sparse models produced by Ridge regression.^[12] This dual regularization reduces the double shrinkage bias inherent in the naive elastic net formulation, allowing for more accurate coefficient estimates without excessive penalization.^[12] Empirical studies demonstrate elastic net's advantages in high-dimensional settings with correlated features, such as gene expression data. In simulations with grouped variables, elastic net reduced mean squared error (MSE) by 13% to 27% compared to Lasso across various scenarios.^[12] For instance, on leukemia microarray data involving thousands of genes, elastic net achieved zero test error while selecting 45 relevant genes, outperforming Lasso's limitation to at most 38 genes due to its n-variable constraint.^[12] Elastic net is particularly preferable in high-dimensional regimes where the number of predictors p greatly exceeds the sample size n (p >> n) and features exhibit grouping, such as in genomics or finance with multicollinear inputs.^[12] At the extremes of the mixing parameter α, it reduces to Lasso (α=1, emphasizing sparsity) or Ridge (α=0, emphasizing shrinkage), providing a flexible fallback.^[12] However, elastic net still necessitates careful tuning of its two hyperparameters, and it may not outperform simpler methods like Lasso or Ridge when features are purely orthogonal without correlations.^[12]

Implementation and Applications

Software Libraries

Elastic net regularization is implemented in several prominent open-source software libraries across programming languages, facilitating its use in statistical modeling and machine learning workflows. These implementations typically leverage coordinate descent algorithms for efficient computation of regularization paths.^[8] In R, the glmnet package provides a comprehensive implementation for fitting lasso and elastic net regularized generalized linear models, supporting linear, logistic, Poisson, and other response distributions. It computes the full regularization path for a grid of lambda values and includes built-in cross-validation functions like cv.glmnet for model selection. The package is optimized for large datasets, achieving high performance through Fortran code and parallel processing options.^[13]^[8] Python offers multiple libraries for elastic net. The scikit-learn module includes the ElasticNet class, which integrates seamlessly with machine learning pipelines, preprocessing steps, and cross-validation tools like GridSearchCV, enabling easy deployment in predictive modeling tasks. For more statistically oriented analyses, statsmodels provides fit_regularized methods in its OLS and GLM classes, supporting elastic net penalties alongside detailed inference outputs such as standard errors and p-values.^[14]^[15] In other languages, MATLAB's lasso and lassoglm functions incorporate elastic net via the 'Alpha' parameter, which controls the L1/L2 penalty mix, and support cross-validation for lambda selection in linear and generalized linear models. Julia's GLMNet.jl package serves as a wrapper around the R glmnet library, allowing users to fit elastic net models for generalized linear models with similar path computation capabilities.^[16]^[17] Key features common to these libraries include warm starts for efficient computation along the regularization path, parallelization for cross-validation, and extensions to generalized linear models beyond simple linear regression.^[18]^[19] As of 2025, Python libraries have seen enhancements in GPU support, particularly through NVIDIA's cuML in the RAPIDS ecosystem, which accelerates elastic net fitting on massive datasets with up to 50x speedups over CPU-based implementations like scikit-learn, while maintaining API compatibility.^[20]^[21]

Practical Use Cases and Examples

Elastic net regularization has found significant application in genomics, particularly for gene selection in high-dimensional microarray data where genes exhibit strong correlations. In a seminal example, Zou and Hastie applied elastic net to the leukemia classification dataset from Golub et al., which consists of gene expression profiles from 72 patients with acute lymphoblastic or myeloblastic leukemia. The method successfully selected groups of correlated genes relevant to leukemia subtypes, achieving better predictive performance than lasso by incorporating both sparsity and grouping effects, with non-zero coefficients clustered around biologically related genes.^[2] In finance, elastic net is employed to predict stock returns amid multicollinear economic indicators, such as interest rates, inflation metrics, and GDP components, which often move together. For instance, in modeling U.S. stock returns using local and global features like market volatility and macroeconomic variables, elastic net addresses overfitting and multicollinearity by selecting and shrinking coefficients for correlated predictors, leading to more stable forecasts compared to ridge or lasso alone; in one study, it yielded lower mean squared errors in out-of-sample predictions by retaining groups of related indicators. A simple numerical illustration of elastic net's behavior appears in synthetic data generation with correlated features. Consider a dataset with three groups of five nearly perfectly correlated predictors each (within-group correlation approximately 1), plus noise variables, where the true coefficients are equal (β = 3) for the 15 relevant features and zero otherwise; fitting elastic net selects all 15 relevant coefficients (shrunk to approximately 3) while zeroing out noise, demonstrating its grouping effect, whereas lasso selects only about 11 scattered coefficients. This example highlights how elastic net preserves correlated signal without arbitrary exclusion.^[2] In practice, tuning elastic net involves cross-validation to select the regularization parameters λ (overall penalty strength) and α (mixing between L1 and L2 penalties). Typically, a grid search over α ∈ [0,1] and λ via k-fold cross-validation minimizes prediction error, such as mean squared error for regression; the resulting sparse model allows interpretation by focusing on non-zero coefficients, which indicate key predictors while accounting for correlations. Software libraries like glmnet facilitate this process efficiently. A post-2020 case study demonstrates elastic net's utility in COVID-19 prognosis models, where biomarkers like inflammatory cytokines and clinical labs (e.g., C-reactive protein, D-dimer) exhibit multicollinearity. In analyzing serum proteomics from hospitalized patients, elastic net logistic regression identified key inflammatory biomarkers associated with disease severity, selecting 11 variables (9 proteins plus neutrophil/lymphocyte counts) from hundreds while handling correlations, achieving an AUC of 0.91 for classifying severe vs. non-severe cases; this outperformed unregularized models by reducing overfitting in high-dimensional biomarker spaces.^[22]

References

[1]
Regularization and variable selection via the elastic net - Zou - 2005
Mar 9, 2005 · We propose the elastic net, a new regularization and variable selection method. Real world data and a simulation study show that the elastic net often ...
[2]
https://hastie.su.domains/Papers/B67.2%20(2005)%20301-320%20Zou%20&%20Hastie.pdf
[3]
Ridge Regression: Biased Estimation for Nonorthogonal Problems
Introduced is the ridge trace, a method for showing in two dimensions the effects of nonorthogonality. It is then shown how to augment X′X to obtain biased ...
[4]
Regression Shrinkage and Selection Via the Lasso - Oxford Academic
SUMMARY. We propose a new method for estimation in linear models. The 'lasso' minimizes the residual sum of squares subject to the sum of the absolute valu.Missing: original | Show results with:original
[5]
Regularization and Variable Selection Via the Elastic Net
Although ridge regression requires 1/(1+λ2) shrinkage to control the estimation variance effectively, in our new method, we can rely on the lasso shrinkage to ...Introduction and motivation · Naïve elastic net · Elastic net · A simulation study
[6]
Regularization Paths for Generalized Linear Models
Feb 2, 2010 · We develop fast algorithms for estimation of generalized linear models with convex penalties. The models include linear regression, two-class logistic ...
[7]
Regularization Paths for Generalized Linear Models via Coordinate ...
The algorithms use cyclical coordinate descent, computed along a regularization path. The methods can handle large problems and can also deal efficiently with ...
[8]
[PDF] A Reduction of the Elastic Net to Support Vector Machines ... - arXiv
Sep 6, 2014 · Zou and T. Hastie. Regularization and variable selection via the elastic net. Journal of the Royal. Statistical Society: Series B ...
[9]
[PDF] on model selection consistency of the elastic net when
Generally speaking, if the Irrepresentable Condition holds, then there exist λ1 > 0 and λ2 > 0 such that the corresponding Elastic. Irrepresentable Condition ...
[10]
[PDF] Regularization and Variable Selection via the Elastic Net
Abstract. We propose the elastic net, a new regularization and variable se- lection method. Real world data and a simulation study show that the elastic net ...
[11]
Evaluation of the lasso and the elastic net in genome-wide ...
Dec 3, 2013 · Studies have shown that analysis with the elastic net can result in lower mean squared errors than the lasso and ridge regression when predictor ...Missing: advantages evidence
[12]
[PDF] glmnet: Lasso and Elastic-Net Regularized Generalized Linear Models
Jul 17, 2025 · glmnet and assess.glmnet. glmnet.path. Fit a GLM with elastic net regularization for a path of lambda values. Description. Fit a generalized ...
[13]
ElasticNet — scikit-learn 1.7.1 documentation
The parameter l1_ratio corresponds to alpha in the glmnet R package while alpha corresponds to the lambda parameter in glmnet. Specifically, l1_ratio = 1 is the ...
[14]
statsmodels.regression.linear_model.OLS.fit_regularized
The elastic net uses a combination of L1 and L2 penalties. The implementation closely follows the glmnet package in R. The function that is minimized is:.
[15]
lasso - Lasso or elastic net regularization for linear models - MATLAB
This MATLAB function returns fitted least-squares regression coefficients for linear models of the predictor data X and the response y.
[16]
Julia wrapper for fitting Lasso/ElasticNet GLM models using glmnet
jl, an implementation of least angle regression for fitting entire linear (but not generalized linear) Lasso and Elastic Net coordinate paths.
[17]
An Introduction to - glmnet
May 5, 2025 · The regularization path is computed for the lasso or elastic net penalty at a grid of values (on the log scale) for the regularization ...Introduction · Quick Start · Multinomial Regression: family... · Filtering variables
[18]
1.1. Linear Models — scikit-learn 1.7.2 documentation
The following are a set of methods intended for regression in which the target value is expected to be a linear combination of the features.Ordinary Least Squares and... · 1.2. Linear and Quadratic... · LinearRegression
[19]
API Reference — cuml 25.10.00 documentation - RAPIDS Docs
Method to set cuML's single GPU estimators global output type. It will be ... 'elasticnet': Elastic Net regularization, a weighted average of L1 and L2.cuML C++ API: Main Page · Introduction · User Guide · cuML’s documentation!
[20]
NVIDIA's GPU Acceleration in scikit-learn, UMAP, and HDBSCAN
Mar 19, 2025 · NVIDIA's latest cuML update brings GPU acceleration to scikit-learn, UMAP, and HDBSCAN, boosting performance by up to 50x for sklearn—with zero ...
[21]
Combining Deep Phenotyping of Serum Proteomics and Clinical ...
Inflammatory biomarkers associated with COVID-19 severity. (a) Elastic net logistic regression with inflammatory variables. Coefficients of each variable are ...