Fact-checked by Grok 2 weeks ago

Robust regression

Robust regression is a form of regression analysis designed to circumvent the sensitivity of ordinary least squares (OLS) estimation to outliers and violations of classical assumptions, such as normality of errors and homoscedasticity, by downweighting or excluding influential observations to yield more stable parameter estimates and predictions.^[1]^[2] This approach enhances the validity and efficiency of inferences in datasets contaminated by atypical values, which can otherwise distort model fitting and lead to misleading conclusions. Robust methods are particularly valuable in empirical research where data irregularities are common.^[3] The development of robust regression traces its origins to the broader field of robust statistics, pioneered by Peter J. Huber in the mid-20th century. Huber's foundational 1964 paper introduced robust estimation for location parameters, laying the groundwork for handling non-normal distributions, while his 1973 work extended these ideas to regression models through maximum likelihood-type estimators.^[4] Subsequent advancements in the 1980s and 1990s, including contributions from Frank Hampel, Victor Yohai, and Peter Rousseeuw, focused on high-breakdown-point estimators that resist up to nearly 50% contamination in the data.^[1] Robust regression encompasses various estimation techniques, such as M-estimators, S-estimators, MM-estimators, and least trimmed squares (LTS), which minimize robust loss functions or trim outliers to achieve high breakdown points and efficiency. Extensions to high-dimensional settings incorporate regularization methods like Lasso or ridge regression alongside robust losses.^[1] These methods find application in fields with noisy data, such as biomedical sciences, finance, and social sciences.^[5]^[6]^[7] They support outlier detection, model diagnostics, and are implemented in software like R's robustbase package. Despite advantages, robust techniques require careful parameter tuning to balance robustness and efficiency under ideal conditions.^[1]^[3]

Introduction

Definition and purpose

Robust regression encompasses a class of statistical techniques for linear regression that yield reliable parameter estimates even when key assumptions of classical methods, such as the normality and homoscedasticity of error terms, are violated.^[4] These methods prioritize properties including continuity of the estimator with respect to the underlying distribution, qualitative robustness against small perturbations in the data-generating process, and a high breakdown point to withstand substantial contamination before failing. The primary purpose of robust regression is to estimate the regression coefficients \beta in the linear model y = X\beta + \epsilon, where \epsilon may contain outliers or follow heavy-tailed distributions, thereby reducing the undue influence of anomalous observations on the fit.^[4] In the basic setup, for observations y_i = x_i^T \beta + \epsilon_i, robust estimation seeks to solve an optimization problem of the form

\min_{\beta} \sum_i \rho\left( \frac{\epsilon_i}{\sigma} \right),

where \rho is a robust loss function (e.g., bounded or redescending) that downweights large residuals, and \sigma is a scale estimate of the errors.^[4] This approach contrasts with ordinary least squares, which minimizes the sum of squared residuals and is highly sensitive to deviations from its assumptions.^[4] Robust regression goals include achieving high efficiency under the assumed model (e.g., near-maximum likelihood performance when errors are Gaussian), strong resistance to contamination from outliers or model misspecification, and desirable asymptotic properties such as consistency and normality under mild conditions as sample size grows.^[4] These objectives ensure that the estimators remain stable and interpretable in real-world data scenarios where perfect adherence to ideal assumptions is rare.^[4]

Comparison to ordinary least squares

Ordinary least squares (OLS) regression seeks to minimize the sum of squared residuals, formulated as \min_{\beta} \sum_{i=1}^n (y_i - \mathbf{x}_i^T \beta)^2, yielding the closed-form solution \hat{\beta}_{OLS} = (X^T X)^{-1} X^T \mathbf{y}.^[8] This approach assumes homoscedastic, normally distributed errors and is optimal under those conditions, but it exhibits high sensitivity to violations of these assumptions. A primary vulnerability of OLS is its extreme sensitivity to outliers, where even a single aberrant observation can arbitrarily bias the estimates by disproportionately influencing the squared residuals.^[8] For instance, in the presence of leverage points or response outliers, the breakdown point of OLS—the smallest fraction of contaminated data that can cause the estimator to fail—is asymptotically zero, meaning it can collapse with just one outlier in large samples. Additionally, OLS performs poorly under heteroscedasticity or non-normal errors, leading to inefficient and biased inferences as the quadratic loss amplifies deviations from the model. In contrast, robust regression methods provide bounded influence on estimates, limiting the impact of any single observation and preventing arbitrary bias from outliers. These methods achieve consistent efficiency under contamination models, such as Huber's \epsilon-contamination where the error distribution is (1 - \eta) F + \eta [G](/page/G) with contamination fraction \eta (typically small, e.g., 0.05–0.1) and arbitrary contaminating distribution G. This ensures reliable performance even when a fraction of the data deviates from the assumed model F, unlike OLS which loses efficiency rapidly under such mixtures.^[8] To illustrate, consider a hypothetical simple dataset with points (x_i, y_i): (0,0), (1,1), (2,2), (3,3), and an outlier (4,10). The true underlying relationship is y = x + \epsilon with small noise. OLS yields a slope estimate of approximately 2.0, heavily pulled toward the outlier and biasing the fit away from the main cluster. A robust estimator, such as one downweighting large residuals, shifts the slope closer to 1.0, better capturing the bulk of the data and demonstrating reduced bias from contamination.

Fundamental Concepts

Outliers and model deviations

In robust regression, outliers represent observations that deviate substantially from the assumed linear model, potentially distorting parameter estimates. These are broadly classified into three types: vertical outliers, also known as response outliers, which occur when the response variable y for a given predictor x lies far from the expected value under the model but the predictor itself is not extreme; leverage points, or design outliers, where the predictor x is distant from the bulk of the data in the predictor space, exerting strong influence on the fit due to their position; and bad leverage points, which combine elements of both, being extreme in x and having a y value that misaligns with the model's trend, thus influencing the fit in misleading directions. Good leverage points, by contrast, are extreme in x but align well with the model, reinforcing rather than distorting the fit. This classification, introduced by Rousseeuw and Leroy, highlights how different outlier types affect regression diagnostics and necessitates tailored robust approaches. Beyond outliers, model deviations encompass violations of the standard linear regression assumptions regarding error structure. Heteroscedasticity arises when the variance of errors changes with the level of predictors, leading to non-constant spread in residuals across the data range. Heavy-tailed errors occur when the error distribution has thicker tails than the normal distribution, increasing the likelihood of extreme residuals and amplifying the impact of deviations. Correlated errors, meanwhile, violate the independence assumption, where residuals exhibit dependence, such as in time series or spatial data, causing underestimation of standard errors and invalid inference. These deviations collectively challenge the reliability of model fits by introducing systematic biases or inefficiencies in estimation. The presence of outliers and model deviations profoundly impacts the fitting process in linear regression. Vertical outliers primarily inflate the residual variance, pulling the fitted line toward them and increasing overall model uncertainty without strongly biasing slopes. Leverage points, especially bad ones, can drastically alter coefficient estimates by disproportionately weighting extreme predictors; for instance, in a scatterplot of data points clustered around a linear trend, a bad leverage point positioned far in x but offset in y might rotate the regression line away from the main cloud, biasing the slope and intercept. Heteroscedasticity exacerbates this by concentrating influence in regions of higher variance, while heavy-tailed or correlated errors propagate extremes across the fit, leading to unstable predictions. Graphical tools like scatterplots with overlaid regression lines and residual plots visually illustrate these effects: a contaminated point distant from the line in the vertical direction distorts fit quality metrics, whereas a leverage point shifts the line's orientation, highlighting the need for robustness to maintain accurate inference. Ordinary least squares estimation is highly sensitive to such issues, often yielding unreliable results even with small contamination levels.^[9]^[10] To formalize these issues, contamination models describe how outliers and deviations arise in data-generating processes. The gross-error model, pioneered by Huber, posits that a small fraction \epsilon of observations follows a contaminating distribution different from the ideal model F, such as (1 - \epsilon) F + \epsilon G, where G represents arbitrary gross errors; this captures realistic scenarios like measurement mistakes or data entry errors. Contamination can be symmetric, where G is centered around the model's mean (e.g., symmetric heavy-tailed additions), or asymmetric, introducing directional bias (e.g., one-sided shifts in outliers). These models underscore the theoretical motivation for robust methods, emphasizing protection against worst-case deviations within bounded \epsilon, typically up to 10-20% in practice.^[11]

Measures of robustness

In robust regression, measures of robustness provide quantitative assessments of an estimator's resistance to data contamination, such as outliers arising from model deviations. These metrics evaluate the estimator's stability under perturbations to the underlying distribution, balancing global and local robustness properties. Key measures include the breakdown point, which addresses finite contamination fractions, and the influence function, which captures infinitesimal effects, along with derived quantities like gross-error sensitivity and efficiency. The breakdown point quantifies the largest proportion of contaminated data that an estimator can tolerate before its output can be made arbitrarily far from the true value. Introduced as a global robustness measure, it is defined in the finite-sample context for an estimator T at the empirical distribution F_n as \epsilon_n^* = \sup\{\epsilon : \sup_{\|\Delta\| \leq \epsilon} \|T(F_n + \Delta) - T(F_n)\| < \infty\}, where \Delta represents a contamination distribution with total variation norm at most \epsilon.^[12] In regression settings, this measures how many observations can be replaced by arbitrary values (e.g., at infinity) before the estimator breaks down, with the least squares estimator having a breakdown point of $1/n and high-breakdown methods achieving up to nearly 0.5. The maximum attainable breakdown point in regression is 0.5, as exceeding half the sample in contamination allows arbitrary fits by aligning with the outliers.^[13] The influence function assesses the local sensitivity of an estimator to contamination at a specific point, measuring the asymptotic bias from infinitesimal perturbations. For an estimator T at distribution F, it is defined as

IF(z; T, F) = \lim_{\epsilon \to 0} \frac{T((1-\epsilon)F + \epsilon \delta_z) - T(F)}{\epsilon},

where \delta_z is the Dirac measure at point z. This derivative-like quantity indicates how much the estimator shifts when a small fraction \epsilon of the data is contaminated at z, providing insight into the directional impact of outliers on the regression coefficients. In regression, the influence function decomposes into components for residuals and leverage, highlighting leverage points' amplified effects. The gross-error sensitivity extends the influence function to bound worst-case local robustness, defined as \gamma^*(T, F) = \sup_z |IF(z; T, F)|. This supremum over all possible contamination points z gives the maximum asymptotic bias from a single gross error, serving as a finite threshold for B-robustness when bounded. For the ordinary least squares estimator, it is unbounded, reflecting high sensitivity, whereas robust estimators like Huber's proposal limit it to a constant, such as 1. Efficiency measures the precision trade-off in robust estimators under the ideal model, typically the Gaussian error distribution in regression. It is the asymptotic relative efficiency (ARE) to the ordinary least squares estimator, computed as the reciprocal of the ratio of their asymptotic variances. For instance, Huber's M-estimator with tuning constant k=1.345 achieves 95% efficiency at the normal while bounding influence, illustrating the inherent compromise between robustness to contamination and efficiency under normality.

Applications

Handling outliers

Robust regression finds significant application in econometrics, where datasets frequently include outliers arising from measurement errors in economic variables such as GDP estimates or inflation rates, allowing for more reliable inference on relationships like those between policy changes and growth.^[14] In environmental monitoring, it is employed to handle outliers in time-series data from sensors, often caused by temporary malfunctions or extreme weather events, as seen in analyses of river flow or pollutant concentrations where such deviations could otherwise skew trend detection.^[15] A core technique in robust regression for handling outliers involves downweighting deviant observations through robust scales that estimate the spread of the bulk of the data, rather than the entire sample. This approach is particularly effective in models assuming contaminated normality, where the error distribution is viewed as a mixture of a primary normal component and a small proportion of arbitrary outliers, thereby preventing extreme values from dominating the fit.^[1] The breakdown point offers a brief measure of suitability here, indicating the maximum fraction of outliers a method can tolerate before parameter estimates become arbitrary, making it ideal for outlier-heavy datasets.^[5] The benefits of these applications include enhanced parameter stability, especially against high-leverage points that lie far from the data cloud and could otherwise pull regression coefficients toward spurious directions in ordinary least squares.^[16] Additionally, robust regression yields improved prediction intervals by basing uncertainty estimates on the majority of observations, reducing undue widening from outlier-induced variance inflation.^[17] In real-world medical datasets, such as those from vaccine potency tests, robust regression mitigates the impact of outliers in bioassay data, ensuring models better reflect the underlying relationships without distortion.^[18]

Addressing heteroscedasticity

Heteroscedasticity refers to a situation in regression models where the variance of the error terms is not constant across observations, formally expressed as \operatorname{Var}(\varepsilon_i \mid x_i) = \sigma^2(x_i), where \sigma^2(x_i) depends on the covariates x_i. This violation of the ordinary least squares (OLS) assumption of homoscedasticity results in inefficient parameter estimates and biased standard errors, leading to unreliable inference such as invalid t-tests and confidence intervals.^[19] Robust regression addresses heteroscedasticity through methods like iteratively reweighted least squares (IRLS), which incorporates variance-stabilizing weights w_i = 1 / \hat{\sigma}^2(x_i) to downweight observations with higher estimated variances, thereby stabilizing the estimation process and improving efficiency. These weights are typically estimated iteratively by fitting an auxiliary model for the variance structure, often assuming a parametric form such as \log \sigma^2(x_i) = \gamma_0 + \gamma_1 x_i. This approach extends traditional weighted least squares to robust contexts by combining it with bounded influence functions, ensuring resistance to variance outliers while achieving consistency under misspecification of the variance model.^[20]^[21] In finance, robust regression techniques are applied to models exhibiting volatility clustering, where error variances fluctuate over time due to market dynamics, enabling more reliable forecasting of asset returns without assuming constant volatility. For instance, extensions of autoregressive conditional heteroskedasticity (ARCH) models incorporate robust estimators to handle heavy-tailed innovations common in financial data. In biology, these methods are used in dose-response analyses, where precision varies with dose levels due to experimental factors like biological variability, allowing for accurate estimation of potency and efficacy curves in toxicological or pharmacological studies.^[22]^[23]^[24] The primary advantages of robust approaches to heteroscedasticity include consistent inference that does not rely on the homoscedasticity assumption, thereby providing valid p-values and prediction intervals even under variance heterogeneity. Additionally, they enhance model diagnostics by isolating variance-related issues from other deviations, facilitating better model selection and interpretation in complex datasets. Robust loss functions can further adapt these methods to simultaneously handle variance issues alongside other model violations.^[19]

Historical Development

Origins and key milestones

The roots of robust regression trace back to early statistical concerns about the sensitivity of estimators to outliers and model deviations, with precursors in the late 19th and early 20th centuries. Francis Ysidro Edgeworth's work on probable errors, particularly in his 1887 paper "On discordant observations," explored the impact of discrepant observations on probability estimates, laying conceptual groundwork for later robustness ideas by emphasizing the need for methods resilient to data contamination.^[25] However, these early efforts were limited by computational constraints, and robust approaches remained theoretical until the post-World War II era, when advances in computing enabled practical implementation of more sophisticated techniques.^[26] A pivotal shift occurred in 1960 with John W. Tukey's seminal paper "A Survey of Sampling from Contaminated Distributions," which formalized the concept of robustness against contaminated data models and advocated for estimators that perform well under nominal assumptions while resisting gross errors. This work inspired the modern field of robust statistics. Building on this, Peter J. Huber introduced M-estimators in 1964 through his paper "Robust Estimation of a Location Parameter," initially for univariate location problems but soon extended to regression settings, providing a framework for minimizing a robust loss function to downweight outliers. Key milestones in the 1980s advanced robust regression toward higher breakdown points and efficiency. William S. Krasker's 1980 paper "Estimation in Linear Regression Models with Disparate Data Points" proposed bounded-influence estimators to limit the impact of leverage points in regression. Peter J. Rousseeuw's 1984 introduction of least median of squares (LMS) regression in "Least Median of Squares Regression" achieved a maximum 50% breakdown point by minimizing the median of squared residuals, offering superior resistance to outliers.^[27] Victor J. Yohai's 1987 development of MM-estimators in "High Breakdown-Point and High Efficiency Robust Estimates for Regression" combined high breakdown properties with near-maximum efficiency under normality, resolving trade-offs in prior methods.^[28] The 1986 book "Robust Statistics: The Approach Based on Influence Functions" by Frank R. Hampel, Elvezio M. Ronchetti, Peter J. Rousseeuw, and Werner A. Stahel synthesized these developments, extending robustness theory—including Hampel's earlier influence function concept from 1968—to comprehensive regression applications and providing a unified framework for evaluating estimator stability.

Reasons for limited adoption

Despite its theoretical advantages in handling outliers and model deviations, robust regression has experienced limited mainstream adoption in statistical practice and research. Several interconnected barriers—spanning computational demands, interpretability challenges, software availability, and entrenched disciplinary preferences—have contributed to this, though recent developments in data science are beginning to alter the landscape.^[29] A primary obstacle has been the computational complexity of robust regression techniques. Unlike ordinary least squares (OLS), which yields closed-form solutions, robust methods such as M-estimators typically require iterative optimization algorithms like iteratively reweighted least squares (IRLS) to minimize non-quadratic loss functions, leading to higher computational costs, especially in high-dimensional or large-sample settings. These challenges persisted until advances in the 1990s, including improved numerical strategies for IRLS convergence, made robust estimation more feasible for practical applications. Interpretability issues further hinder widespread use. Robust regression produces estimates and standard errors that deviate from the familiar parametric framework of OLS, where inference relies on well-understood p-values and t-statistics under normality assumptions; in contrast, robust inference often involves asymptotic approximations or bootstrap procedures that are less intuitive to explain, particularly to interdisciplinary audiences like biomedical researchers.^[30] This complexity can make robust results appear less transparent, reducing their appeal in fields prioritizing straightforward reporting.^[30] Software limitations exacerbated these problems for decades. Standard statistical packages, such as early versions of SAS and SPSS, defaulted to OLS implementations, relegating robust methods to custom code or specialized routines; for instance, Peter Rousseeuw's introduction of least median of squares (LMS) in 1984 included initial software provisions, but broad accessibility only emerged with tools like the R package robustbase in the early 2000s. Cultural and disciplinary factors have also played a significant role in the reluctance to adopt robust regression. Robust statistics have often been perceived as an "exotic" or advanced topic rather than a core component of statistical training, with many practitioners adhering to classical frequentist paradigms that emphasize parametric efficiency under ideal assumptions, viewing robust downweighting of data as potentially arbitrary or inefficient when outliers are absent.^[29] This skepticism was evident in 1980s discussions, where robust advocates like John Tukey faced resistance from those prioritizing exact inference over resilience to violations.^[31] In recent years, however, adoption has increased post-2010, fueled by the demands of big data environments where outliers and non-normal errors are commonplace, alongside integrations with machine learning frameworks that emphasize predictive robustness over strict parametric inference.

Estimation Methods

M-estimators

M-estimators constitute a primary class of robust regression methods, generalizing the ordinary least squares estimator by solving a system of estimating equations derived from a robust loss function. Specifically, the estimator \hat{\beta} minimizes \sum_{i=1}^n \rho\left( \frac{y_i - x_i^T \hat{\beta}}{\hat{\sigma}} \right), or equivalently, solves the normal equations \sum_{i=1}^n \psi\left( \frac{y_i - x_i^T \beta}{\sigma} \right) x_i = 0, where \psi = \rho' is the derivative of a convex loss function \rho, \sigma is a scale estimate, y_i are the responses, and x_i are the predictors.^[4] This framework, introduced by Huber, replaces the squared loss of maximum likelihood under Gaussian errors with a loss that grows more slowly for large residuals, thereby downweighting outliers.^[11] A canonical example is the Huber M-estimator, where \rho(u) = \frac{u^2}{2} for |u| \leq k and \rho(u) = k|u| - \frac{k^2}{2} otherwise, yielding the monotone \psi(u) = u for |u| \leq k and \psi(u) = k \cdot \sign(u) for |u| > k.^[4] The tuning constant k controls the trade-off between robustness and efficiency; a value of k = 1.345 is commonly used to achieve approximately 95% asymptotic efficiency relative to least squares under Gaussian errors.^[11] Another prominent example is the Tukey biweight estimator, which employs a redescending \psi function to completely ignore large residuals: \rho(u) = \frac{u^2}{6} \left(1 - \left(1 - \left(\frac{u}{c}\right)^2\right)^3 \right) for |u| < c and constant otherwise, with \psi(u) decreasing to zero beyond c.^[32] This redescending behavior enhances outlier rejection compared to monotone functions like Huber's. Under standard regularity conditions, such as bounded \psi and a fixed number of parameters relative to sample size, M-estimators exhibit asymptotic normality: \sqrt{n} (\hat{\beta} - \beta_0) \xrightarrow{d} \mathcal{N}(0, V), where the asymptotic covariance V depends on the design matrix, \psi, and the error distribution.^[4] This property ensures reliable inference in large samples, even under contamination.^[4] M-estimators are typically computed using the iteratively reweighted least squares (IRLS) algorithm, which approximates the nonlinear estimating equations via successive weighted least squares fits. Starting from an initial estimate (e.g., least squares), the update is given by

\hat{\beta}^{t+1} = (X^T W^t X)^{-1} X^T W^t z^t,

where r_i^t = y_i - x_i^T \hat{\beta}^t are the residuals from iteration t, z_i^t = y_i + \frac{\psi(r_i^t / \hat{\sigma})}{\psi'(r_i^t / \hat{\sigma})} \cdot (r_i^t / \hat{\sigma}) \cdot \hat{\sigma}, and W^t is a diagonal matrix with entries w_i^t = \psi'(r_i^t / \hat{\sigma}).^[4] Iterations continue until convergence, often requiring a robust initial scale \hat{\sigma}.^[4] The breakdown point of M-estimators, which measures the fraction of contaminated observations needed to make the estimator arbitrary, can reach up to 0.5 for monotone \psi functions in location models, but is often lower in regression settings due to leverage effects and the need to balance high efficiency. For instance, standard implementations like Huber's achieve a breakdown point near 0 in high-leverage scenarios unless augmented with high-breakdown initials.

L-estimators and regression quantiles

L-estimators in the context of regression are constructed as linear functionals of the ordered residuals, extending the univariate concept of L-estimators—such as trimmed means—to the regression setting through weighted sums that assign coefficients to the sorted absolute residuals.^[33] This approach leverages order statistics to achieve robustness by downweighting extreme residuals, similar to how trimmed means mitigate the influence of outliers in location estimation.^[34] Regression quantiles provide a specific and prominent class of L-estimators for linear models, where the \tau-th regression quantile \boldsymbol{\beta}_\tau is defined as the solution that minimizes the sum \sum_{i=1}^n \rho_\tau (y_i - \mathbf{x}_i^T \boldsymbol{\beta}), with the check function \rho_\tau(u) = u (\tau - \mathbf{1}\{u < 0\}) and \mathbf{1}\{\cdot\} the indicator function.^[35] This formulation generalizes the sample quantile to conditional quantiles, allowing estimation across the distribution of the response variable rather than solely at the mean.^[36] A key robustness property of regression quantiles is their breakdown point, which equals \min(\tau, 1-\tau) and measures the smallest fraction of contaminated data that can cause the estimator to break down; for instance, median regression at \tau=0.5 achieves a breakdown point of 0.5, making it highly resistant to outliers. This property holds independently of the regression model's dimensionality, providing consistent robustness even in high-dimensional settings.^[37] In applications, regression quantiles are particularly valuable for handling non-symmetric error distributions, such as in economics where they estimate conditional medians to analyze heterogeneous effects across the wage distribution or income inequality.^[38] For example, they enable modeling how predictors influence different points of the response distribution, revealing insights into tail behaviors that ordinary least squares overlooks.^[39] Computationally, regression quantiles are obtained by solving the corresponding linear programming problem, traditionally using the simplex method for efficiency and stability, though modern implementations often employ interior-point algorithms to handle larger datasets.^[40] The least absolute deviations estimator, a special case at \tau=0.5, aligns with this framework as a robust baseline.^[20]

Other robust techniques

S-estimators represent a class of robust regression estimators that simultaneously estimate the regression coefficients and a scale parameter by minimizing a robust measure of scale, specifically the value of σ(β) such that the sum over i of ρ((r_i(β)/σ)/c) equals b n, where r_i(β) are the residuals, ρ is a bounded redescending loss function, c is a tuning constant, and b is a fixed proportion typically set around 0.5 to achieve high breakdown point.^[41] These estimators possess a maximum breakdown point of 0.5, meaning they can withstand up to half the observations being arbitrary outliers without the estimate diverging, a property that makes them particularly suitable for contaminated datasets.^[28] Proposals for their implementation, including considerations for high-dimensional settings, have been advanced to enhance computational feasibility while preserving robustness. MM-estimators build on S-estimators through a two-step procedure: an initial high-breakdown S-estimator provides a robust starting point, followed by an M-estimator refinement tuned for high efficiency at the model distribution, such as achieving 95% relative efficiency to least squares under normality.^[28] This hybrid approach maintains the 0.5 breakdown point of the initial S-estimator while improving asymptotic efficiency, making MM-estimators a balanced choice for practical applications where both robustness and precision are required.^[28] Least trimmed squares (LTS) estimators minimize the sum of the h smallest squared residuals, Σ_{i=1}^h r_{(i)}^2(β), where h = floor((n + p + 1)/2) + 1 and p is the number of predictors, yielding a breakdown point of approximately 0.5. This method trims the largest residuals, effectively ignoring potential outliers, and is computationally intensive but highly effective for affine-equivariant robustness in regression settings. Unit weights offer a simple, non-iterative robust scaling alternative in early robust regression frameworks, assigning equal weights to predictors rather than differential weights derived from least squares, which reduces sensitivity to multicollinearity and outliers in prediction tasks. This approach, while less sophisticated than modern estimators, provides a baseline robustness comparable to ordinary least squares in certain low-contamination scenarios without requiring scale estimation iterations. Parametric alternatives extend robustness to generalized linear models (GLMs) by adjusting the deviance function to incorporate bounded influence, such as through robust quasi-deviance measures that downweight outliers in the estimation of coefficients for non-normal responses like binary or count data. These adjustments preserve the GLM structure while achieving high breakdown points and efficiency, enabling robust inference in exponential family models.

Examples

BUPA liver dataset

The BUPA liver dataset, sourced from the BUPA Medical Research Ltd. and available through the UCI Machine Learning Repository, comprises 345 observations from male patients undergoing blood tests for liver disorders potentially linked to alcohol consumption.^[42] The dataset includes six continuous variables: mean corpuscular volume (mcv), alkaline phosphatase (alkphos), alamine aminotransferase (sgpt), aspartate aminotransferase (sgot), gamma-glutamyl transpeptidase (gammagt), and daily alcohol consumption in half-pint equivalents (drinks). A common misinterpretation treats the 'selector' field (a train/test split indicator) as a binary response for liver disorder presence; however, the dataset is suited for regression tasks, such as predicting the continuous 'drinks' variable using the five blood test predictors. Note that the UCI documentation warns against using the selector as a class label.^[42] In applying linear regression to this dataset, ordinary least squares (OLS) estimation of drinks on the five blood test predictors can be biased by outliers, particularly in elevated enzyme levels like gammagt and sgot, which are common in heavy drinkers and distort the fit toward extreme values. Robust regression methods, such as MM-estimators—which combine high breakdown-point initial estimates (e.g., S-estimators) with high-efficiency M-estimation iterations—effectively downweight these influential observations, yielding more stable parameter estimates compared to OLS by mitigating the leverage of outliers. For instance, analyses show that OLS tends to overestimate coefficients for gammagt (a key indicator of alcohol-induced liver damage) due to a cluster of high-response outliers, while MM-estimation adjusts these downward, improving overall model efficiency and predictive accuracy. Representative analyses of the BUPA dataset illustrate the downweighting effect, with MM-estimators showing smaller magnitudes for outlier-sensitive predictors like gammagt, alongside tighter standard errors that enhance inference reliability. Residual plots further highlight this: OLS residuals exhibit large deviations for high-enzyme outliers, violating normality assumptions, whereas MM-estimator residuals cluster more tightly around zero, with weights assigned below 0.5 to influential observations. Interpretation of the robust model underscores its value in biomedical contexts, reliably identifying key predictors such as gammagt levels and the ratio of sgot to sgpt (indicating differential enzyme elevation from alcohol stress) as strong signals of alcohol consumption, less affected by outliers. This approach supports more trustworthy clinical insights for early intervention in alcohol-related liver disease.

Outlier detection procedures

Outlier detection in robust regression relies on diagnostics that mitigate the influence of anomalous observations during estimation. Robust residuals are defined as e_i = \frac{y_i - \mathbf{x}_i^T \hat{\beta}}{\hat{\sigma}}, where \hat{\beta} is a robust estimate of the regression coefficients and \hat{\sigma} is a robust scale estimate of the errors, such as the median absolute deviation. These residuals provide a standardized measure less sensitive to outliers than ordinary least squares residuals. Studentized versions of these residuals further adjust for the variability associated with each observation, computed by excluding the i-th case from the scale estimation to yield externally studentized residuals r_i^* = \frac{e_i}{\hat{\sigma}_{(i)}}, where \hat{\sigma}_{(i)} is the robust scale without the i-th observation; values exceeding thresholds like |r_i^*| > 2.5 flag potential outliers. Key methods for detection include the forward search algorithm, which iteratively builds subsets of increasing size starting from a clean initial sample, monitoring trajectories of statistics like residuals to identify outliers as they enter later subsets. Deletion diagnostics, such as a robustified Cook's distance, adapt the classical influence measure by substituting robust estimates into the formula D_i = \frac{e_i^2 h_{ii}}{p \hat{\sigma}^2 (1 - h_{ii})}, where h_{ii} is the leverage from the robust fit and p is the number of parameters; large D_i indicates observations whose removal substantially alters the fit. QQ-plots of robust residuals can reveal outliers by showing deviations from the expected normal quantiles, particularly in the tails, which helps unmask masking effects where outliers conceal each other in ordinary diagnostics. Practical procedures often involve iterative re-estimation: after identifying suspects via diagnostics, temporarily remove them, refit the robust model, and reassess residuals to confirm; this process repeats until stability is achieved, ensuring detected outliers do not unduly influence subsequent steps. Envelope tests complement this by constructing confidence bands around forward search trajectories of monitoring statistics, such as minimum deletion residuals; excursions beyond these envelopes signal structural changes or outliers at specific subset sizes. Breakdown point concepts guide threshold selection in these tests, ensuring detection robustness to contamination levels up to 50%.

References

[1]
Robust linear regression for high‐dimensional data: An overview
Jul 8, 2020 · Robust regression methods aim at assigning appropriate weights to observations that deviate from the model. While robust regression techniques ...
[2]
T.1.1 - Robust Regression Methods | STAT 501
Robust regression methods provide an alternative to least squares regression by requiring less restrictive assumptions.Missing: key applications
[3]
[PDF] Robust Regression
Robust regression provides useful information even when assumptions are not applicable, and is less vulnerable to unusual data than least squares. M-estimation ...
[4]
Robust Regression: Asymptotics, Conjectures and Monte Carlo
Maximum likelihood type robust estimates of regression are defined and their asymptotic properties are investigated both theoretically and empirically.Missing: original | Show results with:original
[5]
A review of robust regression in biomedical science research - PMC
In this article four robust regression techniques that combine high breakdown points and high efficiency are presented.
[6]
Applications of Robust Statistical Methods in Quantitative Finance
Robust methods are used for outlier detection in asset returns and for testing asset pricing models, addressing multivariate outliers.Missing: key | Show results with:key
[7]
[PDF] Robust Linear Regression: A Review and Comparison - arXiv
Apr 24, 2014 · Ordinary least-squares (OLS) estimators for a linear model are very sensitive to unusual values in the design space or outliers among y values.
[8]
9.1 - Distinction Between Outliers and High Leverage Observations
An outlier is a data point whose response y does not follow the general trend of the rest of the data. A data point has high leverage if it has "extreme" ...Missing: robust | Show results with:robust
[9]
The Impact of Outliers on Linear Regression Models: Detection and ...
Jun 7, 2025 · Outliers can significantly distort the results of linear regression models, leading to misleading conclusions and reduced predictive ...
[10]
Robust Estimation of a Location Parameter - Project Euclid
This model arises for instance if the observations are assumed to be normal with variance 1, but a fraction ϵ ϵ of them is affected by gross errors. Later ...
[11]
A General Qualitative Definition of Robustness - jstor
The concept of the "breakdown point" of a sequence of estimators is defined, and some examples are given. 1. Introduction and motivation. The setup of robust ...<|control11|><|separator|>
[12]
HIGH BREAKDOWN-POINT AND HIGH EFFICIENCY - Project Euclid
Rousseeuw. (1984) proposed the least median of squares (LMS) and the least trimmed squares (LTS) which are defined by the minimization of the median or the.
[13]
Applications of Robust Regression Techniques: An Econometric ...
May 29, 2021 · A robust regression is an iterative procedure that is designed to overcome the problem of outliers and influential observations in the data and ...
[14]
Combining statistical methods for detecting potential outliers in ...
Nov 8, 2022 · The methodology for outlier detection described in this paper uses robust regression on order statistics (ROS) to deal with measured values ...
[15]
Robust Regression | R Data Analysis Examples - OARC Stats - UCLA
Robust regression is an alternative to least squares when data has outliers, weighing observations differently based on how well-behaved they are.Missing: key | Show results with:key
[16]
Robust prediction intervals in a regression setting - ScienceDirect.com
In this study we intend to examine methods of predicting a future observation that are robust across a variety of situations involving errors that are ...
[17]
Appropriateness of Robust Regression in Addressing Outliers in an ...
Nov 25, 2015 · Abstract. Outliers within a bioassay are not uncommon, especially with animal models. Ordinary linear regression is sensitive to outliers; ...
[18]
[2311.02822] Robust estimation of heteroscedastic regression models
Nov 6, 2023 · We collect robust proposals given in the field of regression models with heteroscedastic errors. Our motivation stems from the fact that the practitioner ...
[19]
[PDF] Robust Regression in the Presence of Heteroscedasticity
Robust Regression in the Presence of Heteroscedasticity 91 least absolute deviations (Hoaglin et al., 1983). In the location case, the median has a breakdown ...
[20]
Robust methods for heteroskedastic regression - ScienceDirect.com
We provide a new robust method for the analysis of heteroskedastic data with the linear regression model which is both efficient and has high breakdown point.
[21]
Heteroscedasticity-Robust Inference in Linear Regression Models ...
Abstract. We consider inference in linear regression models that is robust to heteroscedasticity and the presence of many control variables.
[22]
A Robust Proposal for Heteroscedastic Dose–Response Models ...
May 30, 2025 · This article proposes a robust approach to dose–response analysis with inhomogeneous variance observations, as it often arises in practice ...
[23]
Robust and Efficient Assessment of Potency (REAP) as a ...
To improve the quantitative estimation of the dose-response relationship, we introduce a novel approach based on robust beta regression.
[24]
John Tukey and Robustness - Project Euclid
In this article, I review some of this early work, discuss in particular one inspiring article that was published in 1960 (Tukey, 1960a), describe some of the ...
[25]
Least Median of Squares Regression - Taylor & Francis Online
In this article a different approach is introduced in which the sum is replaced by the median of the squared residuals.
[26]
High Breakdown-Point and High Efficiency Robust Estimates for ...
The MM-estimates are defined by a three-stage procedure. In the first stage an initial regression estimate is computed which is consistent robust and with high ...
[27]
(PDF) The historical development of robust statistics - ResearchGate
econometrics, and biostatistics. f) The breakdown point. The breakdown point introduced by Hampel (1968, 1971) is a measure of global stability. for a ...
[28]
https://projecteuclid.org/journals/annals-of-statistics/volume-15/issue-2/High-Breakdown-Point-and-High-Efficiency-Robust-Estimates-for-Regression/10.1214/aos/1176350366.full
[29]
https://www.researchgate.net/publication/228434224_The_historical_development_of_robust_statistics
[30]
The Fitting of Power Series, Meaning Polynomials, Illustrated on ...
Apr 9, 2012 · Technometrics Volume 16, 1974 - Issue 2 · Submit an article Journal ... View PDF (open in a new window) PDF (open in a new window) · Share.
[31]
9 On some L-estimation in linear regression models - ScienceDirect
Linear combination of order statistics or L-estimators plays an extremely important role in the development of robust methods for location parameters.Missing: seminal papers
[32]
4. robust regression for the linear model
important role in calculating MM-estimates, which are far more efficient. MM-Estimators. First proposed by Yohai (1987), MM-estimators have become increasingly.
[33]
Regression Quantiles - jstor
BY ROGER KOENKER AND GILBERT BASSETT, JR. A simple minimization problem yielding the ordinary sample quantiles in the location model is shown to generalize ...
[34]
Regression Quantiles - Econometrica - The Econometric Society
Jan 1, 1978 · A simple minimization problem yielding the ordinary sample quantiles in the location model is shown to generalize naturally to the linear model.
[35]
Robust regression quantiles | Request PDF - ResearchGate
Aug 6, 2025 · The robustness of these procedures is independent of the complexity of the regression model and is proportional to min{τ, 1 − τ }, where τ ...
[36]
Economic applications of quantile regression 2.0
Dec 24, 2021 · This special issue of Empirical Economics combines eight empirical applications of quantile regression and related methods focusing on modeling ...
[37]
Quantile Regression - American Economic Association
Quantile regression, as introduced by Koenker and Bassett (1978), may be viewed as an extension of classical least squares estimation of conditional mean models ...
[38]
[PDF] Computational Methods for Quantile Regression
Aug 3, 2016 · Linear programming and the associated simplex solution method emerged out of the fog of World War II, as did many other important statistical ...
[39]
Robust Regression by Means of S-Estimators - SpringerLink
In this paper we shall develop a class of methods for robust regression, and briefly comment on their use in time series.
[40]
UCI Machine Learning Repository
### Summary of Liver Disorders Dataset