Fact-checked by Grok 2 weeks ago

Likelihood-ratio test

The likelihood-ratio test (LRT) is a fundamental statistical procedure in hypothesis testing that assesses the goodness-of-fit between observed and two competing models—one corresponding to the and the other to the alternative—by computing the ratio of their maximized likelihood functions. The test rejects the if this ratio is below a critical , indicating that the provides a substantially better explanation of the . Developed by and in their seminal 1933 paper, the LRT emerged as a response to the need for optimal tests in the Neyman-Pearson framework, which emphasizes controlling the type I error rate while maximizing power against specific alternatives. The Neyman-Pearson lemma establishes that, for simple null and alternative hypotheses (where both specify exact parameter values), the LRT yields the most powerful test of a given size, meaning it achieves the highest probability of correctly rejecting the null when it is false. This optimality property underpins its foundational role in classical statistics, distinguishing it from earlier approaches like those of , which focused more on p-values without explicit power considerations. For more general composite hypotheses (where hypotheses involve ranges of parameters), the test is extended to the generalized likelihood-ratio test, which compares the maximum likelihood under the full model to that under the restricted null model. Under standard regularity conditions—such as the models being identifiable and the data being independent and identically distributed—the test statistic, defined as -2 times the log of the likelihood ratio, converges asymptotically to a chi-squared distribution with degrees of freedom equal to the difference in the number of free parameters between the unrestricted and restricted models. This asymptotic property enables practical computation of p-values and critical values even for large samples, making the LRT robust and versatile across diverse applications.

Introduction

Overview

The (LRT) is a fundamental method in for comparing the relative fit of two competing models to a , particularly when one model is nested within the other, by evaluating the ratio of their maximized likelihood values. This approach assesses whether the additional parameters in the more complex model significantly improve the fit beyond what can be attributed to chance, aiding in and hypothesis evaluation. In the context of hypothesis testing, the LRT evaluates a H_0 (often corresponding to the simpler, nested model) against an H_A (the more general model), using the -2 \log \Lambda, where \Lambda is the defined as the supremum of the likelihood under H_0 divided by the supremum under H_A. The test rejects H_0 if this statistic exceeds a determined by the desired level. The LRT derives its appeal from optimality properties under regularity conditions; for simple hypotheses (point and alternative), the Neyman-Pearson lemma establishes it as the uniformly most powerful test of a given size, maximizing power while controlling the type I error rate. The LRT relies on key assumptions, including that the data consist of independent and identically distributed (i.i.d.) observations from the specified parametric family, and that the is correctly specified for both models. These ensure the validity of and the test's inferential properties. Under large sample sizes, the asymptotically follows a with equal to the difference in the number of parameters between the models.

Historical Development

The emerged from early 20th-century developments in , with precursors in Pearson's work on goodness-of-fit testing. In 1900, Pearson introduced the chi-squared statistic as a measure for assessing whether observed frequencies in categorical data deviated significantly from expected values under a hypothesized distribution, laying foundational groundwork for ratio-based criteria in hypothesis testing that were later refined into likelihood frameworks. Building on this, Ronald A. Fisher formalized the method of maximum likelihood estimation in 1922, providing the probabilistic foundation essential for constructing likelihood ratios by maximizing the probability of observed data given parameters. Fisher's approach emphasized estimating parameters that render the data most probable, which directly influenced subsequent tests comparing likelihoods under competing hypotheses. The test itself was developed in the 1930s by Jerzy Neyman and Egon S. Pearson as a core component of their unified theory of hypothesis testing. In their seminal 1933 paper, they proposed the likelihood ratio as the optimal criterion for deciding between two simple hypotheses, rejecting the null when the ratio of likelihoods under the alternative versus null exceeds a threshold calibrated for a fixed significance level, thus establishing it as a uniformly most powerful test in that setting. Extensions to composite hypotheses followed soon after, with Samuel S. Wilks contributing key results on the asymptotic behavior of the in 1938. Wilks demonstrated that, under regularity conditions, minus twice the log-likelihood ratio converges in distribution to a chi-squared with equal to the difference in the number of free parameters between the full and restricted models, enabling practical for more complex scenarios. The likelihood-ratio test gained broader influence in modern statistics through its integration into generalized linear models, as outlined by John A. Nelder and Robert W. M. Wedderburn in 1972, where it underpins deviance-based assessments of model fit within distributions. Post-1980s advancements in further propelled its adoption, with software packages like GLIM implementing likelihood-ratio procedures for iterative model fitting and testing, facilitating widespread use in applied analyses across disciplines.

Mathematical Formulation

Likelihood Function and Hypotheses

The likelihood function serves as the foundational measure in the likelihood-ratio test, quantifying how well a parametric model explains observed data. For an independent and identically distributed (i.i.d.) sample \mathbf{x} = (x_1, \dots, x_n) drawn from a probability density or mass function f(x_i \mid \theta), where \theta is the parameter in the parameter space \Theta, the likelihood function is defined as L(\theta \mid \mathbf{x}) = \prod_{i=1}^n f(x_i \mid \theta). This formulation treats the data as fixed and views the function as varying with respect to \theta, distinguishing it from the joint probability density of the sample. The assumption of i.i.d. observations ensures the product form of the likelihood, which simplifies subsequent maximizations and is a prerequisite for the test's theoretical properties. In the context of hypothesis testing, the likelihood-ratio test compares a H_0: \theta \in \Theta_0 against an H_A: \theta \in \Theta_A, where \Theta_0 \subset \Theta_A ensures the models are nested, with \Theta_0 representing the restricted under the null. The is simple if \Theta_0 consists of a single point (e.g., a specific value of \theta) and composite otherwise, allowing for a range of parameter values. These hypotheses are formulated within identifiable parametric models, where distinct parameters yield distinct distributions, and the likelihood is well-defined and positive for all \theta \in \Theta. Additionally, differentiability of the log-likelihood \ell(\theta \mid \mathbf{x}) = \log L(\theta \mid \mathbf{x}) with respect to \theta is assumed to facilitate and asymptotic approximations in later analyses. The test relies on the maximized likelihoods under each hypothesis: \sup_{\theta \in \Theta_0} L(\theta \mid \mathbf{x}) for the and \sup_{\theta \in \Theta_A} L(\theta \mid \mathbf{x}) for the , obtained by evaluating the likelihood at their respective maximum likelihood estimators. These suprema quantify the relative support for the hypotheses given the , assuming the models satisfy regularity conditions such as of the parameter spaces or interior maxima. For non-nested models where \Theta_0 \not\subset \Theta_A, the standard likelihood-ratio test is invalid, as the ratio lacks a meaningful interpretation under the ; alternative procedures, such as the Vuong test, should be used instead. These elements form the basis for constructing the in subsequent formulations.

Test Statistic

The likelihood-ratio test statistic is defined as the ratio of the maximum likelihood under the to the maximum likelihood under the , given by \Lambda = \frac{\sup_{\theta \in \Theta_0} L(\theta \mid x)}{\sup_{\theta \in \Theta_A} L(\theta \mid x)}, where L(\theta \mid x) denotes the , \Theta_0 is the parameter space under the H_0, and \Theta_A encompasses \Theta_0 under the H_A. This formulation, introduced by Neyman and Pearson, measures the relative support for H_0 versus H_A based on the observed data x. Under H_0, \Lambda \leq 1 because the supremum over the restricted space \Theta_0 cannot exceed that over the larger space \Theta_A, leading to a rejection region of the form \Lambda < k for some critical value k < 1. For computational convenience, the statistic is often transformed to \lambda = -2 \log \Lambda, which satisfies \lambda \geq 0 under H_0 with rejection when \lambda > c for a suitable threshold c. Equivalently, \lambda can be expressed in terms of the log-likelihood l(\theta) = \log L(\theta \mid x) as \lambda = 2 \left[ l(\hat{\theta}_A) - l(\hat{\theta}_0) \right], where \hat{\theta}_0 = \arg\max_{\theta \in \Theta_0} l(\theta) and \hat{\theta}_A = \arg\max_{\theta \in \Theta_A} l(\theta). The maximizations required to compute \lambda involve maximum likelihood estimation, which for complex models typically relies on numerical optimization techniques such as the Newton-Raphson method or expectation-maximization algorithms. However, challenges arise when \Theta_0 lies on the boundary of \Theta_A, as in cases of testing for the presence of parameters (e.g., variance components equal to zero); here, the standard asymptotic chi-squared distribution for \lambda under H_0 may not hold, requiring instead a mixture of chi-squared distributions or other adjustments for valid inference. In large samples, \lambda facilitates inference by approximating known distributions when boundary issues are absent.

Distribution and Inference

Exact Distribution for Simple Hypotheses

When both the H_0 and the H_A are simple—meaning each fully specifies a single for the data—the likelihood-ratio test (LRT) possesses optimal properties. Specifically, the establishes that the LRT is the uniformly most powerful (UMP) test of any given significance level \alpha for distinguishing between these two distributions. Under H_0, the exact of the likelihood ratio statistic \Lambda = \frac{L(\theta_0 | \mathbf{x})}{L(\theta_A | \mathbf{x})}, where \theta_0 and \theta_A are the fixed parameters under H_0 and H_A, respectively, is model-dependent and derived directly from the assumed data-generating process. This distribution governs the exact critical values and p-values for the test, but its form varies with the parametric family; for instance, in models from the , it can involve ratios of sufficient statistics whose distributions are known under the null (e.g., gamma or chi-squared for scale parameters in exponential distributions). Exact tests are particularly feasible in discrete models with small sample sizes, such as the binomial distribution for testing simple hypotheses about a success probability p. Here, \Lambda simplifies to a monotone function of the number of successes k, and the null distribution is binomial with parameters n (trials) and p = p_0, enabling precise computation of rejection regions by enumerating probabilities of outcomes yielding \Lambda \leq c for a chosen c that controls the type I error at \alpha. To obtain p-values, one calculates P(\Lambda \leq \Lambda_{\text{obs}} \mid H_0), where \Lambda_{\text{obs}} is the observed value; in discrete cases like the , this is the sum of null probabilities over the critical region, often using precomputed tables for very small n. For continuous or higher-dimensional models where analytical summation or integration is intractable, simulation approximates the by generating many samples from the null model, computing \Lambda for each, and estimating the tail probability empirically. However, deriving the exact distribution remains challenging and highly specific to the model, often rendering it impractical for composite hypotheses, large samples, or non-standard families, which motivates reliance on asymptotic approximations for broader applicability.

Asymptotic Distribution

Under the H_0 and appropriate regularity conditions, states that the likelihood ratio test statistic -2 \log \lambda_n, where \lambda_n is the likelihood ratio, converges in distribution to a with d as the sample size n \to \infty, with d = \dim(\Theta_A) - \dim(\Theta_0) representing the difference in the dimensions of the parameter spaces under the alternative hypothesis \Theta_A and the \Theta_0. This asymptotic facilitates the computation of p-values and critical values for the test in large samples. The requires several regularity conditions to hold, including that the true value lies in the interior of the parameter space \Theta_0, ensuring of the parameters under H_0; that the log-likelihood function is twice continuously differentiable with respect to the parameters in an open neighborhood of the true value; that the of the outer product of the score function (or equivalently, the matrix) is positive definite; and that the of the does not depend on the parameters. These conditions guarantee the and asymptotic of the maximum likelihood estimators under both hypotheses. A sketch of the proof begins with a second-order Taylor expansion of the log-likelihood around the maximum likelihood \hat{\theta}_0 under H_0, evaluated at the unrestricted \hat{\theta}_A, yielding -2 \log \lambda_n \approx (\hat{\theta}_A - \hat{\theta}_0)^T I(\hat{\theta}_0) (\hat{\theta}_A - \hat{\theta}_0), where I(\cdot) is the observed information matrix. Asymptotically, under H_0, \hat{\theta}_A - \hat{\theta}_0 behaves like a normal random vector with covariance involving the inverse Fisher information restricted to the tangent space orthogonal to \Theta_0, resulting in the quadratic form converging to \chi^2_d. For small samples or when regularity conditions are mildly violated, the chi-squared approximation can exhibit bias, leading to inflated type I error rates. Bartlett corrections address this by scaling the test statistic by a factor c (typically c \approx 1 - \frac{d(d+2)}{6n}) derived from higher-order expansions of the log-likelihood, improving the agreement with the . Alternatively, bootstrap methods, such as parametric or nonparametric resampling of the data under H_0, provide empirical approximations to the of -2 \log \lambda_n, enhancing accuracy without relying on asymptotic .

Examples and Applications

Binomial Proportion Test

The likelihood ratio test is commonly applied to assess hypotheses concerning a single proportion p, the probability of success in independent Bernoulli trials. Consider testing the null hypothesis H_0: p = p_0 against the two-sided alternative H_A: p \neq p_0, based on n trials yielding k observed successes. The likelihood function under this model is given by L(p) = \binom{n}{k} p^k (1-p)^{n-k}. Under H_0, the maximized likelihood is L(p_0), while under the full model (including H_A), the maximum likelihood estimator (MLE) is \hat{p} = k/n, yielding L(\hat{p}). The likelihood ratio test statistic is then \lambda = 2 \left[ \log L(\hat{p}) - \log L(p_0) \right] = 2 \left[ k \log \left( \frac{\hat{p}}{p_0} \right) + (n - k) \log \left( \frac{1 - \hat{p}}{1 - p_0} \right) \right], where the binomial coefficient cancels out in the difference. To interpret the test, \lambda is compared to a critical value from the \chi^2_1 distribution (referenced asymptotically for decision-making). For a numerical illustration, suppose n = 100, k = 45, and p_0 = 0.5, so \hat{p} = 0.45. Substituting into the formula gives \lambda \approx 1.00. The corresponding p-value is approximately 0.32, indicating insufficient evidence to reject H_0 at common significance levels like \alpha = 0.05. This step-by-step computation highlights the LRT's practicality for binomial data. A key advantage of the LRT in this binomial setting is its exact equivalence to Pearson's chi-squared goodness-of-fit test when framed as comparing observed frequencies (k successes, n-k failures) to expected frequencies under H_0 (n p_0, n (1-p_0)). Furthermore, it aligns precisely with the statistic here, as all three tests reduce to similar forms under the simple for the proportion.

Linear Regression Model

In the model, the likelihood-ratio test (LRT) is applied to compare a full model, specified as \mathbf{Y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\epsilon} where \boldsymbol{\epsilon} \sim N(\mathbf{0}, \sigma^2 \mathbf{I}), against a reduced model under the H_0: \mathbf{Y} = \mathbf{X}_0 \boldsymbol{\beta}_0 + \boldsymbol{\epsilon} with the same error assumption, where \mathbf{X}_0 is a subset of columns from \mathbf{X} corresponding to the parameters restricted under H_0. This setup tests whether the additional predictors in the full model significantly improve the fit, such as when evaluating the inclusion of one or more regressors. The test statistic is computed as \lambda = n \log\left(\frac{\text{RSS}_\text{reduced}}{\text{RSS}_\text{full}}\right), where n is the number of observations and RSS denotes the residual sum of squares for each model; under the assumption of normally distributed errors, this LRT is equivalent to the F-test for nested models. For instance, consider a dataset with n=20 observations where a simple linear regression model (one predictor) is compared to a multiple regression model adding a second predictor to test the significance of that coefficient; the resulting \lambda \approx 5.4 is then compared to a \chi^2_1 distribution for inference at a 5% significance level, where the p-value indicates whether the added predictor is warranted. The LRT in connects directly to analysis of variance (ANOVA) frameworks, as the comparison of RSS between nested models mirrors the partitioning of variance in ANOVA tables for testing model adequacy. Regarding , the LRT remains applicable even when predictors in the full model are correlated, though high multicollinearity can inflate the variance of estimates and reduce the test's to detect true differences between models. While the LRT leverages the full likelihood for generality and can extend beyond strict assumptions through asymptotic approximations, its reliability for non-normal errors depends on large-sample behavior. Under normality, the test relates exactly to the for finite samples.

Related Concepts and Extensions

Comparison with Other Tests

The likelihood-ratio test (LRT) is one of three classical large-sample tests for composite hypotheses, alongside the and the (also known as the Lagrange multiplier test). The is based on the maximum likelihood estimator (MLE) under the () and its estimated , effectively testing whether the MLE is sufficiently far from the () values. In contrast, the LRT compares the likelihoods under both H0 and HA by evaluating the ratio of the maximized likelihood under H0 to that under HA, utilizing information from both restricted and unrestricted models. The , meanwhile, relies solely on the score function (the of the log-likelihood) and observed information matrix evaluated under H0, making it computationally advantageous when the null model is simpler to fit than the alternative. Asymptotically, under H0, all three tests are equivalent and follow a with equal to the difference in the number of parameters between HA and H0, as established by for the LRT and corresponding results for the others. However, in finite samples, differences emerge: the can exhibit poorer performance, such as inflated Type I error rates or reduced , particularly when the MLE under HA is near the boundary or under model misspecification, whereas the LRT tends to be more robust due to its balanced use of both models' information. Additionally, while the LRT and are invariant to reparameterization of the model, the is not, meaning its results can change depending on how parameters are scaled or transformed. The choice among these tests often depends on computational feasibility and model structure. The LRT is particularly preferred for nested models, where HA encompasses H0, as it directly measures the improvement in fit via the likelihood ratio and is the natural extension of exact tests for simple hypotheses. The score test is ideal when evaluating perturbations from H0 without needing to estimate the full model, such as in preliminary screening, while the Wald test suits scenarios where the unrestricted MLE is already available. In generalized linear models (GLMs), the LRT is operationalized through the deviance statistic, defined as twice the difference in log-likelihoods between the full and reduced models, which serves as the standard measure for model comparison and testing nested hypotheses.

Generalized Likelihood Ratio Test

The generalized likelihood ratio test (GLRT) is the extension of the classical likelihood ratio test to composite hypotheses within nested models, where the restricts the parameter space of the alternative. Further extensions and modifications of the GLRT address more complex scenarios, such as non-nested hypotheses, high-dimensional settings, and situations where asymptotic approximations are unreliable. For instance, when comparing non-nested models, Vuong's test (1989) provides a framework by approximating the Kullback-Leibler divergence between models using a derived from normalized log-likelihood differences, enabling selection between competing specifications without assuming nesting. In high-dimensional contexts with numerous nuisance parameters, the profile likelihood approach adjusts the GLRT by maximizing the likelihood over nuisance parameters under each , often incorporating penalty terms to mitigate and improve finite-sample performance. This adjusted LRT, sometimes combined with regularization like in sparse settings, ensures the test remains valid even when the number of parameters approaches or exceeds the sample size. For cases where asymptotic chi-squared distributions fail, such as small sample sizes or constraints on parameters, the bootstrap GLRT resamples the to empirically estimate p-values, offering a distribution-free alternative that enhances accuracy in problems or non-regular cases. Applications of the GLRT in frequently involve for , where it evaluates the number of components by testing nested or generalized structures, serving as a foundation for information criteria like AIC and that approximate the test for computational efficiency. In fitting, the GLRT helps detect the presence of multiple subpopulations, though it requires adjustments for issues.