Fact-checked by Grok 2 weeks ago

Sufficient statistic

In statistics, a sufficient statistic is a function of a sample that captures all the information about an unknown parameter contained in the sample, such that no other statistic derived from the same sample provides additional information regarding the value of that parameter.^[1] This concept, introduced by R. A. Fisher in his seminal 1922 paper, allows for data reduction without loss of inferential value, making it a cornerstone of parametric inference.^[1] The formal identification of sufficient statistics is facilitated by the Fisher–Neyman factorization theorem, which states that a statistic T(\mathbf{X}) is sufficient for a parameter \theta if the joint probability density (or mass) function of the sample \mathbf{X} can be expressed as f(\mathbf{x} \mid \theta) = g(T(\mathbf{x}), \theta) \cdot h(\mathbf{x}), where g depends on \theta only through T and h does not depend on \theta.^[2] This theorem, originally sketched by Fisher for discrete cases and generalized by Jerzy Neyman in 1935, provides a constructive criterion for verifying sufficiency in many parametric families.^[2] Sufficiency is particularly valuable in estimation and hypothesis testing, as it enables the use of lower-dimensional summaries for maximum likelihood estimation while preserving the properties of the full sample. Common examples illustrate the utility of sufficient statistics across distributions. For independent Bernoulli trials with success probability \theta, the total number of successes \sum X_i is sufficient for \theta, as it condenses the binary outcomes into a single informative value.^[3] Similarly, for a sample from a normal distribution N(\mu, \sigma^2) with known \sigma^2, the sample mean \bar{X} is sufficient for \mu.^[3] In the uniform distribution on [0, \theta], the maximum observation X_{(n)} serves as a sufficient statistic for \theta.^[3] These examples highlight how sufficiency often aligns with natural summaries like sums or order statistics, aiding in efficient statistical procedures.^[3] Further developments include the notions of minimal sufficient and complete sufficient statistics, which refine the concept for optimal inference; a minimal sufficient statistic is a coarsest reduction that retains all information, while completeness ensures unbiased estimators based on it are unique.^[2] Sufficiency underpins exponential families, where fixed-dimensional statistics often suffice regardless of sample size, and extends to Bayesian contexts via the sufficiency principle, which posits that inferences should depend only on the sufficient statistic.

Fundamentals

Historical Background

The concept of a sufficient statistic originated in the early 20th century amid the foundational developments in frequentist statistical inference, driven primarily by Ronald A. Fisher's efforts to formalize efficient estimation methods. In his seminal 1922 paper, Fisher introduced maximum likelihood estimation as a principle for parameter inference, highlighting the need for data summaries that preserved all relevant information about the parameters without redundancy. This laid the groundwork for sufficiency by emphasizing likelihood functions as carriers of evidential content from the data. Fisher further developed these ideas in his 1925 paper, where he explicitly coined the term "sufficient statistic" and proposed an early version of the factorization criterion as a sufficient condition for sufficiency, applicable to specific distributions like the normal and Poisson. Building on Fisher's heuristic insights, Jerzy Neyman extended and generalized the concept in the 1930s, integrating it into the broader framework of hypothesis testing and efficient estimation. Neyman's 1934 work on sampling theory discussed representative methods, including purposive selection, with R.A. Fisher introducing the idea of sufficient statistics in his response to the paper. He formalized the factorization theorem in 1935, providing a necessary and sufficient condition for sufficiency across more general parametric families, thus resolving limitations in Fisher's earlier criterion. This advancement was detailed in their 1936 publication with Egon S. Pearson, which linked sufficiency to uniformly most powerful tests, solidifying its role in reducing data dimensionality while maintaining inferential power.^[4] The development of sufficient statistics addressed key inefficiencies in pre-20th-century estimation practices, where full datasets were often retained despite much of the information being extraneous for parameter inference. By enabling data reduction without information loss, sufficiency aligned with the emerging likelihood-based paradigm in frequentist statistics, facilitating practical computations and influencing subsequent theories of estimation and testing. This historical progression, as chronicled in Lehmann's analysis, marked a pivotal shift toward modern statistical efficiency.^[5]

Mathematical Definition

In probability theory and statistics, a statistic T = T(X_1, \dots, X_n), where X = (X_1, \dots, X_n) is a random sample from a distribution parameterized by \theta, is defined as sufficient for \theta if the conditional distribution of X given T(X) = t is independent of \theta for every value of t.^[6] This condition, originally articulated by R. A. Fisher, ensures that the value of T(X) fully accounts for the sample's relevance to \theta, rendering further details of X ancillary to inference about the parameter.^[7] An equivalent characterization of sufficiency arises through the factorization of the likelihood function: the joint density (or probability mass function) of the sample can be expressed as f(x_1, \dots, x_n \mid \theta) = g(T(x_1, \dots, x_n), \theta) \cdot h(x_1, \dots, x_n), where g depends on the data only through T and on \theta, while h is free of \theta.^[7] This formulation highlights how sufficiency partitions the likelihood into a component tied to the parameter via the statistic and a residual component unrelated to inference. Sufficiency implies that T(X) captures all the information about \theta available in the full sample X, allowing statistical procedures—such as estimation or hypothesis testing—to proceed using T(X) alone with no loss of inferential efficiency.^[6] In this sense, the sufficient statistic achieves maximal data reduction while preserving the evidential content for parameter assessment. Unlike ancillary statistics, whose distributions do not depend on \theta and thus provide no information about the parameter, sufficient statistics explicitly incorporate the dependence on \theta through the data structure.^[3]

Basic Example

A simple example of a sufficient statistic arises in the context of independent and identically distributed observations from a Bernoulli distribution with success probability \theta \in (0,1). Consider a random sample X = (X_1, \dots, X_n) where each X_i equals 1 with probability \theta and 0 otherwise. The sample sum T(X) = \sum_{i=1}^n X_i, which counts the total number of successes, serves as a sufficient statistic for \theta. To verify sufficiency, recall that a statistic T is sufficient if the conditional distribution of the sample given T = t does not depend on \theta. The joint probability mass function of X is P(X = x \mid \theta) = \theta^{\sum x_i} (1-\theta)^{n - \sum x_i} for x_i \in \{0,1\}. The marginal distribution of T is binomial: P(T = t \mid \theta) = \binom{n}{t} \theta^t (1-\theta)^{n-t}. Thus, the conditional probability is

P(X = x \mid T = t, \theta) = \frac{P(X = x \mid \theta)}{P(T = t \mid \theta)} = \begin{cases} \frac{1}{\binom{n}{t}} & \text{if } \sum x_i = t, \\ 0 & \text{otherwise}. \end{cases}

This distribution, uniform over all sequences with exactly t ones, is free of \theta, confirming sufficiency. Intuitively, T(X) encapsulates all relevant information about \theta because the likelihood depends solely on the total number of successes, rendering the specific order of outcomes irrelevant for inference about \theta.^[8] In contrast, a single observation such as X_1 is not sufficient, as the conditional distribution of the remaining sample (X_2, \dots, X_n) given X_1 = x_1 retains dependence on \theta through the unchanged Bernoulli probabilities for the other variables.^[8]

Characterization of Sufficiency

Fisher-Neyman Factorization Theorem

The Fisher-Neyman factorization theorem establishes a necessary and sufficient condition for a statistic to be sufficient in the context of independent and identically distributed (i.i.d.) observations. Specifically, for a sample \mathbf{X} = (X_1, \dots, X_n) drawn from a parametric family with joint probability density function (pdf) or probability mass function (pmf) f(\mathbf{x}; \theta), where \theta is the unknown parameter, a statistic T(\mathbf{X}) is sufficient for \theta if and only if there exist functions g: \mathbb{R}^k \times \Theta \to [0, \infty) and h: \mathbb{R}^n \to [0, \infty) such that

f(\mathbf{x}; \theta) = g(T(\mathbf{x}), \theta) \cdot h(\mathbf{x})

for all \mathbf{x} \in \mathbb{R}^n and \theta \in \Theta.^[3] This criterion applies equally to both discrete and continuous distributions, without imposing regularity conditions such as differentiability of the density or the existence of moments.^[3] The theorem's formulation in terms of the joint pdf/pmf factorization directly characterizes the mathematical definition of sufficiency, as it ensures that the conditional distribution of \mathbf{X} given T(\mathbf{X}) does not depend on \theta.^[9] The theorem derives its name from the independent contributions of Ronald A. Fisher and Jerzy Neyman; Fisher first presented a version in 1922, while Neyman provided the general form in 1935.^[9]^[10] A key practical advantage of the factorization theorem is its ability to verify sufficiency by inspecting the structure of the likelihood function, avoiding the more computationally intensive task of explicitly deriving and examining conditional distributions.^[3] This makes it an essential tool for applied statisticians in identifying sufficient reductions of data in parametric inference problems.

Proof of the Factorization Theorem

The Fisher-Neyman factorization theorem is proved under the assumption that the observed sample X = (X_1, \dots, X_n) consists of independent and identically distributed (i.i.d.) random variables from a parametric family with parameter \theta, where the joint probability mass function (p.m.f.) or probability density function (p.d.f.) f_X(x \mid \theta) exists.^[11] The proof establishes equivalence between sufficiency of a statistic T(X) and the factorization form f_X(x \mid \theta) = g(T(x), \theta) h(x), where g depends on the data only through T(x) and \theta, and h is independent of \theta. It proceeds in two directions, first for the discrete case and then analogously for the continuous case.^[2]

Direct Part: Sufficiency Implies Factorization

Assume T(X) is sufficient for \theta, meaning the conditional distribution of X given T(X) = t is independent of \theta. By definition, the joint distribution factors as

f_X(x \mid \theta) = f_{T}(t \mid \theta) \cdot f_{X \mid T}(x \mid t),

where t = T(x). Since sufficiency implies f_{X \mid T}(x \mid t) does not depend on \theta, define h(x) = f_{X \mid T}(x \mid T(x)) (with h(x) = 0 if f_X(x \mid \theta) = 0) and g(t, \theta) = f_{T}(t \mid \theta). Thus,

f_X(x \mid \theta) = g(T(x), \theta) h(x).

This holds for both discrete and continuous cases, as the conditional form arises directly from the definition of sufficiency.^[11]^[2]

Converse Part: Factorization Implies Sufficiency

Assume the factorization f_X(x \mid \theta) = g(T(x), \theta) h(x) holds. To show sufficiency, verify that the conditional distribution f_{X \mid T}(x \mid t, \theta) is independent of \theta. For the discrete case, the marginal p.m.f. of T at t is

f_T(t \mid \theta) = \sum_{\{x' : T(x') = t\}} g(t, \theta) h(x') = g(t, \theta) \sum_{\{x' : T(x') = t\}} h(x'),

where the sum is over the support of X restricted to the level set \{x' : T(x') = t\}, which can be expressed using the indicator function I_{\{T(x') = t\}}(x') as

f_T(t \mid \theta) = g(t, \theta) \sum_{x'} h(x') I_{\{T(x') = t\}}(x').

The conditional p.m.f. is then

f_{X \mid T}(x \mid t, \theta) = \frac{f_X(x \mid \theta) I_{\{T(x) = t\}}(x)}{f_T(t \mid \theta)} = \frac{g(t, \theta) h(x) I_{\{T(x) = t\}}(x)}{g(t, \theta) \sum_{x' : T(x') = t} h(x')} = \frac{h(x)}{\sum_{x' : T(x') = t} h(x')},

provided T(x) = t; otherwise, it is zero. This expression does not depend on \theta, confirming sufficiency.^[2]^[11] For the continuous case, the proof proceeds analogously, with the marginal p.d.f. of T at t given by

f_T(t \mid \theta) = g(t, \theta) \int_{\{x : T(x) = t\}} h(x) \, dx,

where the integral is over the level set \{x : T(x) = t\}, handled via the indicator function I_{\{T(x) = t\}}(x). The conditional p.d.f. is

f_{X \mid T}(x \mid t, \theta) = \frac{f_X(x \mid \theta) I_{\{T(x) = t\}}(x)}{f_T(t \mid \theta)} = \frac{g(t, \theta) h(x) I_{\{T(x) = t\}}(x)}{g(t, \theta) \int_{\{x' : T(x') = t\}} h(x') \, dx'} = \frac{h(x)}{\int_{\{x' : T(x') = t\}} h(x') \, dx'},

which is independent of \theta when T(x) = t. This establishes sufficiency in the continuous setting.^[12]^[2] The theorem extends to non-i.i.d. samples if the joint distribution satisfies similar factorization conditions, though additional regularity assumptions may be required.

Likelihood Principle Interpretation

The concept of sufficiency aligns closely with the likelihood principle in statistical inference, as a sufficient statistic T ensures that all inferences about the parameter \theta depend solely on the likelihood function, which remains fully preserved through the form L(\theta \mid x) \propto g(T(x), \theta). This preservation implies that the evidential content regarding \theta in the original data is captured entirely by T, without alteration to the relative support for different \theta values. The Fisher-Neyman factorization theorem enables this interpretation by decomposing the likelihood in a way that isolates the parameter-dependent component to the sufficient statistic. Allan Birnbaum formalized the likelihood principle in 1962, asserting that two experiments are equivalent for inference about \theta if their likelihood functions are proportional (i.e., if the likelihood ratios L(\theta_1 \mid x)/L(\theta_2 \mid x) are identical for all \theta_1, \theta_2). Under this principle, Birnbaum showed that the sufficiency principle—stating that inferences should be identical for samples yielding the same T value—follows directly, as the sufficient statistic encapsulates the entire likelihood structure relevant to \theta. This connection has profound implications for data reduction: ancillary information or details in the data beyond T become irrelevant for \theta-based inferences, justifying the use of sufficient statistics to simplify analysis while retaining full inferential power. Such reduction supports efficient statistical procedures without compromising evidential integrity. However, while the sufficiency principle enjoys broad acceptance across frequentist and Bayesian paradigms, the full likelihood principle has drawn critiques from some frequentists, who argue it overlooks error rates and long-run performance, even as they endorse sufficiency for its data-summarizing utility.

Minimal Sufficiency

Definition of Minimal Sufficiency

A sufficient statistic T for a family of distributions parameterized by \theta is minimal if it is a function of every other sufficient statistic, meaning that for any other sufficient statistic S, there exists a measurable function g such that T = g(S) with probability 1 for all \theta. This property ensures that T achieves the greatest possible reduction in data dimensionality while preserving all information about \theta, refining the general concept of sufficiency introduced earlier. The notion of minimal sufficiency was formalized to identify the essential summary of the data that cannot be further coarsened without loss of inferential content. An equivalent definition characterizes minimal sufficiency through partitions of the sample space \mathcal{X}. Specifically, T is minimal sufficient if and only if the partition induced by the level sets of T (i.e., the sets \{ x \in \mathcal{X} : T(x) = t \} for each t in the range of T) is the coarsest partition such that, within each block, the likelihood ratio f(x|\theta_1)/f(x|\theta_2) is constant in x for all \theta_1, \theta_2. This equivalence highlights how minimal sufficiency corresponds to the finest discrimination needed between different parameter values based on the observed data. Minimal sufficient statistics possess the property that the conditional distribution of the observation X given T(X) = t is independent of \theta, and uniform over the fiber \{ x : T(x) = t \} in models where the likelihood function is constant within each fiber. Additionally, any two minimal sufficient statistics are equivalent up to one-to-one measurable transformations, ensuring their essential uniqueness for a given statistical model.

Properties and Identification

A minimal sufficient statistic is always sufficient, as it retains all information from the sample relevant to the parameter of interest, but the converse does not hold: there exist sufficient statistics that are not minimal, such as the full sample data itself, which contains redundant information beyond what is needed for inference. This distinction highlights the dimension reduction potential of minimal sufficient statistics, allowing for the coarsest possible partitioning of the sample space while preserving sufficiency.^[13] One practical method to identify a minimal sufficient statistic involves applying the Fisher-Neyman factorization theorem to the joint likelihood function, which yields a candidate sufficient statistic that is often minimal; for instance, in the case of independent uniform random variables on [0, \theta], the maximum order statistic serves as the minimal sufficient statistic for \theta. A precise characterization for minimal sufficiency uses likelihood ratios: a statistic T is minimal sufficient if and only if the ratio L(\theta_1; X)/L(\theta_2; X) depends on the data X only through T(X) for all \theta_1 \neq \theta_2, ensuring that T induces the finest equivalence classes where likelihoods are proportional.^[14] In exponential families, the vector of sufficient statistics in a minimal (full-rank) representation is a minimal sufficient statistic, offering a straightforward computational approach for identification (as explored in subsequent sections on exponential families).^[13]

Examples of Sufficient Statistics

Bernoulli Distribution

In the Bernoulli model, consider a random sample X_1, \dots, X_n where each X_i is independently and identically distributed as Bernoulli(\theta), with \theta \in (0,1) denoting the success probability. The probability mass function for each X_i is P(X_i = x_i \mid \theta) = \theta^{x_i} (1 - \theta)^{1 - x_i} for x_i \in \{0, 1\}. The joint probability mass function of the sample is thus f(\mathbf{x} \mid \theta) = \theta^{\sum_{i=1}^n x_i} (1 - \theta)^{n - \sum_{i=1}^n x_i}.^[11] By the Fisher-Neyman factorization theorem, the statistic T(\mathbf{X}) = \sum_{i=1}^n X_i, representing the total number of successes, is sufficient for \theta. This follows because the joint pmf factors as f(\mathbf{x} \mid \theta) = g(T(\mathbf{x}); \theta) \cdot h(\mathbf{x}), where g(t; \theta) = \theta^t (1 - \theta)^{n - t} and h(\mathbf{x}) = 1.^[15] The statistic T is minimal sufficient, as it is a one-dimensional function of any other sufficient statistic for \theta and induces the coarsest partition of the sample space that preserves the likelihood ratios. Additionally, since the Bernoulli distribution forms a one-parameter exponential family, T is boundedly complete: if E_\theta[g(T)] = 0 for all \theta \in (0,1), then g(t) = 0 almost surely.^[15]^[16] For n > 1, no individual X_i is sufficient for \theta, because the conditional distribution of the full sample given X_i = x_i depends on \theta.^[17]

Poisson Distribution

The Poisson distribution models the number of events occurring in a fixed interval of time or space, assuming events happen independently at a constant average rate \lambda > 0. Consider an independent and identically distributed (i.i.d.) sample X_1, X_2, \dots, X_n from a Poisson(\lambda) distribution, where each X_i takes non-negative integer values. The probability mass function (pmf) for a single observation is

P(X_i = x_i) = \frac{\lambda^{x_i} e^{-\lambda}}{x_i!} , \quad x_i = 0, 1, 2, \dots

The joint pmf of the sample is therefore

f(\mathbf{x} \mid \lambda) = \prod_{i=1}^n \frac{\lambda^{x_i} e^{-\lambda}}{x_i!} = \lambda^{\sum_{i=1}^n x_i} e^{-n\lambda} \left( \prod_{i=1}^n x_i! \right)^{-1} .

^[3] Applying the Fisher-Neyman factorization theorem, the joint pmf factors into a part depending on the data only through the statistic T(\mathbf{x}) = \sum_{i=1}^n x_i and a part independent of \lambda. Specifically,

f(\mathbf{x} \mid \lambda) = g(T, \lambda) \cdot h(\mathbf{x}) ,

where g(T, \lambda) = \lambda^T e^{-n\lambda} and h(\mathbf{x}) = \left( \prod_{i=1}^n x_i! \right)^{-1}. Thus, T, the total number of events across the n intervals (or total counts), is a sufficient statistic for \lambda.^[3]^[18] The statistic T is minimal sufficient for \lambda. This follows because the likelihood ratio f(\mathbf{x} \mid \lambda_1) / f(\mathbf{x} \mid \lambda_2) simplifies to a function solely of T(\mathbf{x}), indicating that T captures all information about \lambda without reducible components. Equivalently, since the Poisson distribution is a member of the regular exponential family with natural parameter related to \log \lambda, the sufficient statistic T achieves minimal dimension.^[19]^[20] When the sample size n is known, the sample mean \bar{X} = T / n provides an alternative sufficient statistic for \lambda, as it is a one-to-one function of T and thus preserves all information about the parameter. This form is particularly useful for estimating the mean rate \lambda directly.^[21]

Normal Distribution

In the context of the normal distribution, consider a random sample X_1, \dots, X_n drawn independently and identically distributed (i.i.d.) from N(\mu, \sigma^2), where \mu is the mean and \sigma^2 is the variance.^[7] When \sigma^2 is known and \mu is unknown, the sum T = \sum_{i=1}^n X_i (or equivalently, the sample mean \bar{X}) is a sufficient statistic for \mu. This follows from the Fisher-Neyman factorization theorem applied to the joint density, which can be expressed as f(\mathbf{x} \mid \mu) = (2\pi \sigma^2)^{-n/2} \exp\left( -\frac{1}{2\sigma^2} \sum_{i=1}^n (x_i - \mu)^2 \right) = g\left( \sum x_i, \mu \right) h(\mathbf{x}), where g depends on the data only through \sum x_i and \mu, and h(\mathbf{x}) is independent of \mu.^[3] When \mu is known and \sigma^2 is unknown, the statistic T = \sum_{i=1}^n (X_i - \mu)^2 is sufficient for \sigma^2. The joint density factors as f(\mathbf{x} \mid \sigma^2) = (2\pi \sigma^2)^{-n/2} \exp\left( -\frac{1}{2\sigma^2} \sum_{i=1}^n (x_i - \mu)^2 \right) = g\left( \sum (x_i - \mu)^2, \sigma^2 \right) h(\mathbf{x}), with g depending on the data solely through \sum (x_i - \mu)^2 and h(\mathbf{x}) = 1.^[22] When both \mu and \sigma^2 are unknown, the statistics T_1 = \sum_{i=1}^n X_i and T_2 = \sum_{i=1}^n X_i^2 are jointly sufficient for (\mu, \sigma^2). The joint density is

f(\mathbf{x} \mid \mu, \sigma^2) \propto \exp\left( -\frac{n}{2} \log(2\pi \sigma^2) - \frac{1}{2\sigma^2} \sum_{i=1}^n (x_i - \mu)^2 \right),

which expands to a form that factors as g(T_1, T_2, \mu, \sigma^2) h(\mathbf{x}), where \sum (x_i - \mu)^2 = T_2 - \frac{T_1^2}{n} and h(\mathbf{x}) = 1, so g depends on the data only through (T_1, T_2) (or equivalently, through (\bar{X}, s^2), the sample mean and sample variance).^[7] The pair (\bar{X}, s^2) is minimal sufficient for (\mu, \sigma^2), as it is a one-to-one function of the jointly sufficient statistics and captures all information about the parameters without redundancy.^[7] Notably, this minimal sufficient statistic has dimension 2, matching the number of unknown parameters.^[7]

Exponential Family Distributions

In the canonical form of a one-parameter exponential family, the probability density function (or mass function) for an observation x is expressed as

f(x; \theta) = h(x) \exp\left\{ \eta(\theta) T(x) - A(\theta) \right\},

where h(x) is a base measure, \eta(\theta) is the natural parameter, T(x) is the sufficient statistic, and A(\theta) is the log-partition function ensuring normalization.^[23] This structure implies that the statistic T(x) captures all information about \theta relevant for inference, as the joint density of a sample x_1, \dots, x_n factors such that the likelihood depends on the data only through T = \sum_{i=1}^n T(x_i).^[24] For multiparameter exponential families, the form generalizes to vector-valued natural parameters and sufficient statistics, with the same sufficiency property holding for the natural sufficient statistic.^[25] The exponential distribution with rate parameter \lambda > 0 provides a concrete illustration, where the density is f(x; \lambda) = \lambda e^{-\lambda x} for x > 0. This belongs to the exponential family in canonical form with natural parameter \eta(\theta) = -\lambda, sufficient statistic T(x) = x, base measure h(x) = 1 for x > 0, and log-partition function A(\lambda) = -\log \lambda.^[26] For an independent sample X_1, \dots, X_n, the sum T = \sum_{i=1}^n X_i is sufficient for \lambda, as the joint likelihood factors accordingly via the Fisher-Neyman theorem.^[7] Similarly, the gamma distribution with known shape parameter \alpha > 0 and unknown scale parameter \beta > 0 has density f(x; \beta) = \frac{\beta^\alpha}{\Gamma(\alpha)} x^{\alpha-1} e^{-\beta x} for x > 0. In canonical exponential family form, the natural parameter is \eta(\beta) = -\beta, the sufficient statistic is T(x) = x, the base measure is h(x) = \frac{x^{\alpha-1}}{\Gamma(\alpha)} for x > 0, and A(\beta) = -\alpha \log \beta.^[27] For a sample, the sum T = \sum_{i=1}^n X_i suffices for \beta, highlighting the pattern where the natural sufficient statistic aggregates the data to encapsulate parameter information.^[7] The uniform distribution on (0, \theta) with \theta > 0 is not a member of the exponential family because the support depends on \theta. Nonetheless, for a sample, the maximum T = \max\{X_1, \dots, X_n\} is sufficient for \theta. The joint density is \theta^{-n} if $0 < x_i \leq \theta for all i (i.e., T \leq \theta) and 0 otherwise, which factors as g(T, \theta) \cdot h(\mathbf{x}) with g(t, \theta) = \theta^{-n} I(t \leq \theta) and h(\mathbf{x}) = 1.^[19] In contrast, for the two-parameter uniform on (\theta_1, \theta_2) with \theta_1 < \theta_2, the joint statistic (T_1 = \min\{X_i\}, T_2 = \max\{X_i\}) is sufficient, as both endpoints inform the interval parameters.^[19] In regular exponential families, where the parameter space has full dimension equal to the number of sufficient statistics and support is independent of parameters, the natural sufficient statistic is minimal sufficient, meaning no further reduction preserves all information about the parameter.^[13]

Rao-Blackwell Theorem

The Rao–Blackwell theorem establishes a fundamental connection between sufficiency and the improvement of estimators in statistical inference. Let \hat{\theta} be an unbiased estimator of a parameter \theta based on a random sample X, and let T = T(X) be a sufficient statistic for \theta. Then, the conditional expectation \hat{\theta}^* = E[\hat{\theta} \mid T] is also an unbiased estimator of \theta, and its variance satisfies \mathrm{Var}(\hat{\theta}^*) \leq \mathrm{Var}(\hat{\theta}) for every value of \theta. This result implies that any unbiased estimator can be refined by projecting it onto the sigma-algebra generated by the sufficient statistic, yielding a more efficient alternative without sacrificing unbiasedness. The theorem underscores the value of sufficient statistics in data reduction, as estimators that depend only on T cannot be improved further in this manner.^[28] The theorem was independently derived by C. Radhakrishna Rao in 1945 and David Blackwell in 1947, marking a key advancement in estimation theory. Rao's contribution appeared in his seminal paper on the accuracy of statistical parameters, where he linked information bounds to estimator efficiency under sufficiency. Blackwell extended the idea to sequential estimation contexts, emphasizing conditional expectations in unbiased settings. These works laid the groundwork for modern approaches to finding minimum-variance unbiased estimators.^[28] A sketch of the proof begins with verifying unbiasedness: by the law of total expectation, E[\hat{\theta}^*] = E[E[\hat{\theta} \mid T]] = E[\hat{\theta}] = \theta. For the variance inequality, apply the law of total variance:

\mathrm{Var}(\hat{\theta}) = E[\mathrm{Var}(\hat{\theta} \mid T)] + \mathrm{Var}(E[\hat{\theta} \mid T]) = E[\mathrm{Var}(\hat{\theta} \mid T)] + \mathrm{Var}(\hat{\theta}^*).

Since E[\mathrm{Var}(\hat{\theta} \mid T)] \geq 0, it follows that \mathrm{Var}(\hat{\theta}) \geq \mathrm{Var}(\hat{\theta}^*), with equality if and only if \mathrm{Var}(\hat{\theta} \mid T) = 0 almost surely, meaning \hat{\theta} is a function of T. This conditional variance decomposition highlights how sufficiency captures all relevant information about \theta, allowing the extraneous variability in \hat{\theta} to be averaged out.^[29] In application, the Rao–Blackwell theorem guides the construction of better estimators by performing Rao–Blackwellization: starting from a crude unbiased estimator like a single observation or a simple average, one conditions on an available sufficient statistic to reduce variance. For instance, in problems where a complete sufficient statistic exists, repeated application alongside the Lehmann–Scheffé theorem can yield the uniformly minimum-variance unbiased estimator. The theorem's utility extends beyond point estimation, influencing methods in survey sampling and computational statistics where variance reduction is critical, though it assumes the existence of a sufficient statistic and finite second moments. Equality in the variance bound occurs precisely when the original estimator is already sufficient, emphasizing that functions of sufficient statistics are optimal in this class.^[29]

Sufficiency in Exponential Families

Exponential families offer a structured approach to sufficiency, where the canonical parameterization directly identifies low-dimensional sufficient statistics. For a k-parameter exponential family in canonical form, the density is given by

p(x \mid \theta) = h(x) \exp\left( \sum_{i=1}^k \theta_i T_i(x) - A(\theta) \right),

where \theta = (\theta_1, \dots, \theta_k) denotes the natural parameter vector, T(x) = (T_1(x), \dots, T_k(x)) is the corresponding sufficient statistic vector, h(x) is the base measure, and A(\theta) is the log-partition function ensuring normalization. This representation implies that T(x) encapsulates all information about \theta from the data x, making it sufficient by the factorization theorem. In the multiparameter case, the joint sufficiency of the vector components arises naturally from the additive structure in the exponent.^[13] In regular full-rank exponential families—where the parameter space contains an open set and the sufficient statistics are affinely independent—the minimal sufficient statistic has dimension exactly equal to the number of parameters k. This minimal dimension ensures efficient data reduction without loss of information, a property that holds for families satisfying standard regularity conditions such as differentiability of A(\theta).^[23] Illustrative examples within exponential families highlight this structure. For the normal distribution N(\mu, \sigma^2) with both parameters unknown, the canonical form yields the sufficient statistic T(\mathbf{x}) = \left( \sum x_i, \sum x_i^2 \right), a two-dimensional vector matching the parameter count. The Poisson distribution \text{Poisson}(\lambda) has one-dimensional sufficient statistic T(x) = x (or sum for i.i.d. samples), aligning with its single parameter. Similarly, the exponential distribution \text{Exp}(\lambda) features T(x) = x as its sufficient statistic in canonical form \exp(\log \lambda - \lambda x). These cases demonstrate how the exponential family parameterization explicitly reveals T.^[24] A key property in full-rank exponential families is the completeness of the minimal sufficient statistic T(x), meaning that if E_\theta[g(T(x))] = 0 for all \theta, then g(T(x)) = 0 almost surely. This completeness, combined with sufficiency, implies uniqueness for minimum-variance unbiased estimators based on T(x). By Basu's theorem, the complete sufficient statistic T(x) is independent of any ancillary statistic, facilitating conditional inference and simplifying the analysis of sampling distributions in these models.^[13]

Other Forms of Sufficiency

Bayesian Sufficiency

In Bayesian statistics, a statistic T is sufficient for the parameter \theta if the posterior distribution \pi(\theta \mid X) depends on the observed data X only through T, that is, \pi(\theta \mid X) = \pi(\theta \mid T).^[30] This condition ensures that all information about \theta contained in the data is captured by the posterior based on T, allowing for data reduction without loss of inferential content in the Bayesian framework.^[31] Equivalently, X and \theta are conditionally independent given T.^[2] Frequentist sufficiency, based on the Neyman-Fisher factorization theorem, implies Bayesian sufficiency unconditionally, as the conditional distribution of the data given the statistic does not depend on \theta, preserving the posterior structure.^[30] However, the converse does not hold in general; Bayesian sufficiency for a specific prior may fail to imply frequentist sufficiency unless the prior distribution is independent of the statistic T.^[32] This dependence arises because Bayesian sufficiency is tied to the chosen prior, potentially incorporating subjective beliefs that affect the posterior in ways not captured by frequentist criteria.^[31] A Bayesian analog of Basu's theorem, which in the frequentist setting establishes independence between a complete sufficient statistic and an ancillary statistic, has been developed for scenarios involving conjugate priors within exponential families.^[33] In this framework, the theorem extends to show that under conjugate priors, the posterior distribution of the parameter given the complete sufficient statistic is independent of ancillary statistics, facilitating sharper Bayesian inferences by separating location and scale information.^[33] This result underscores the role of exponential family structures in aligning Bayesian and frequentist insights on independence.^[33] Bayesian sufficiency differs from its frequentist counterpart by explicitly incorporating subjective prior probabilities, which reflect the analyst's beliefs before observing the data, thus addressing uncertainties in a more personalized manner.^[30] In contrast, frequentist sufficiency focuses on objective data reduction independent of priors, but Bayesian approaches critique this as potentially overlooking prior knowledge, leading to less efficient inferences in small-sample or subjective contexts.^[32] This integration of priors enables Bayesian sufficiency to handle complex models where frequentist methods may struggle with non-informative data reduction.^[31]

Linear Sufficiency

In linear models, particularly within the Gauss-Markov framework, linear sufficiency provides a distribution-free criterion for identifying linear statistics that capture all relevant information for estimating parametric functions via best linear unbiased estimators (BLUEs). Introduced by Barnard in 1963, the concept applies to models of the form Y = X\beta + \epsilon, where Y is the response vector, X is the design matrix, \beta is the parameter vector, and \epsilon has zero mean and known or unknown covariance matrix V. A linear statistic T = A Y is linearly sufficient for an estimable function q^T \beta if there exists a matrix B such that B T equals the BLUE of q^T \beta, ensuring that T spans the linear information subspace necessary for optimal estimation.^[34] For multivariate normal distributions, linear sufficiency is particularly relevant, as the normality assumption makes the BLUE coincide with the maximum likelihood estimator for the mean parameters. Consider independent observations Y_i \sim N_p(\mu, \Sigma), i=1,\dots,n, where \mu lies in a linear subspace defined by the model; here, the sample mean vector \bar{Y} = \frac{1}{n} \sum Y_i and the sample covariance matrix serve as jointly sufficient statistics, with \bar{Y} being linearly sufficient for \mu by projecting the data onto the parameter space. This projection property ensures that any linear unbiased estimator of \mu can be recovered from \bar{Y}, aligning linear sufficiency with full sufficiency under normality while facilitating dimension reduction in high-dimensional settings.^[2]^[35] In applications to analysis of variance (ANOVA) and linear regression, linear sufficient statistics enable efficient inference by condensing the data into forms like sums or cross-products. For instance, in a balanced one-way ANOVA model under normality, the group totals \sum_{j \in g} Y_j for each group g are linearly sufficient for the group mean effects, allowing the BLUEs for contrasts to be computed directly from these totals without retaining individual observations. Similarly, in ordinary least squares regression Y = X\beta + \epsilon with \epsilon \sim N(0, \sigma^2 I), the statistics X^T Y and X^T X are linearly sufficient for X\beta, as they yield the BLUE \hat{\beta} = (X^T X)^{-1} X^T Y, supporting tests and confidence intervals with reduced computational burden. These examples highlight linear sufficiency's role in simplifying model fitting while preserving estimability.^[36]^[37] As a specialized variant of general sufficiency, linear sufficiency focuses on linear transformations and is weaker in non-normal cases but equivalent under normality for mean parameters; it proves especially useful for computational efficiency in large-scale regression problems by avoiding full data storage.^[38]

References

[1]
On the mathematical foundations of theoretical statistics - Journals
On the mathematical foundations of theoretical statistics. R. A. Fisher. Google Scholar · Find this author on PubMed · Search for more papers by this author.
[2]
[PDF] Mathematical Statistics, Lecture 6 Sufficiency - MIT OpenCourseWare
Sufficiency: Factorization Theorem. Theorem 1.5.1 (Factorization Theorem Due to Fisher and. Neyman). In a regular model, a statistic T (X ) with range T is.
[3]
24.2 - Factorization Theorem | STAT 415
The Factorization Theorem states a statistic is sufficient if the joint probability function can be factored into two components, one depending on data and one ...
[4]
IX. On the problem of the most efficient tests of statistical hypotheses
On the problem of the most efficient tests of statistical hypotheses. Jerzy Neyman ... 5_1935, (13013-13038), . Berger J (2018) Statistical Decision Theory ...
[5]
[PDF] “On the Theoretical Foundations of Mathematical Statistics”
Feb 10, 2003 · Def. A statistic is sufficient if it summarizes the whole of the relevant information supplied by the data. If θ is to be estimated and T1.
[6]
[PDF] 1 Sufficient statistics - Arizona Math
In particular we can multiply a sufficient statistic by a nonzero constant and get another sufficient statistic. We now apply the theorem to some examples.
[7]
Sufficiency - Stat 210a
Sufficiency is a central concept in statistics that allows us to focus on the essential aspects of the data set while ignoring details that are irrelevant to ...
[8]
Theory of Statistical Estimation | Mathematical Proceedings of the ...
Sufficient statistics and intrinsic accuracy. Mathematical Proceedings of the Cambridge Philosophical Society, Vol. 32, Issue. 4, p. 567 ...
[9]
(PDF) The Factorization Theorem for Sufficiency - ResearchGate
Sep 7, 2022 · Fisher (1925) and Neyman (1935) characterized sufficiency through the factorization theorem for special and more general cases respectively.
[10]
[PDF] Sufficient Statistics - Arizona Math
Feb 21, 2008 · How we find sufficient statistics is given by the Neyman-Fisher factorization theorem. 1 Neyman-Fisher Factorization Theorem. Theorem 2. The ...Missing: 1935 paper
[11]
[PDF] 4. Sufficiency 4.1. Sufficient statistics. Definition 4.1. A statistic T = T ...
Proof of Theorem 4.2. I'll discuss the case of a discrete distribution; the continuous case is similar. If T is sufficient, then, as we saw above,. Page 4 ...<|control11|><|separator|>
[12]
Monte Carlo goodness-of-fit tests for degree corrected and related ...
That this conditional distribution is uniform is stated, for example, in ... conditional distribution on the fiber. This set is called a Markov basis ...
[13]
[PDF] Sufficiency, Minimal Sufficiency and the Exponential Family of ...
Although a minimal sufficient statistic provides in some sense an optimal degree of data compression it is still possible for it to contain much \extra" or.Missing: seminal paper
[14]
[PDF] Sufficient Statistics and Extreme Points - EB Dynkin
Nov 17, 2005 · BY E. B. Dynkin ... If a convex separable class M has an H-sufficient statistic, then there exists an H-sufficient statistic Q, such that.Missing: 1951 | Show results with:1951
[15]
[PDF] all-of-statistics.pdf
Part I of the text is concerned with probability theory, the formal language of uncertainty which is the basis of statistical inference. The basic problem that ...
[16]
[PDF] STAT 713 MATHEMATICAL STATISTICS II
is a sufficient statistic for the Bernoulli family. By the previous result ... minimal sufficient statistic does not match the dimension of the parameter.
[17]
[PDF] Lecture 11 Likelihood, MLE and sufficiency
Sep 25, 2019 · A statistic T is said to be minimal sufficient if any other sufficient statistic S has the property that S = f(T) for some. (non-random) ...
[18]
[PDF] Lecture 4: Sufficient Statistics
Theorem 1 (Fisher-Neyman Factorization) Let X be a random variable with density P(x|θ) for some θ ∈ Θ. The statistic t(X) is sufficient for θ iff the ...
[19]
[PDF] Chapter 4 Sufficient Statistics
A statistic T(Y 1, ..., Y n) is a sufficient statistic for θ if the condi- tional distribution of (Y 1, ..., Y n) given T = t does not depend on θ for any value of ...
[20]
[PDF] Lecture Notes 11 36-705 1 Minimal sufficiency
There is a strong sense in which estimators which do not depend only on sufficient statistics can be improved. This is known as the Rao-Blackwell theorem. Let ...Missing: Bernoulli | Show results with:Bernoulli
[21]
[PDF] Lecture 4 slides: Sufficient statistics and factorization theorem
The following theorem gives a characterization of minimal sufficient statistics: Theorem 4. Let f(x|θ) be the pdf of X and T(X) be such that, for any x, y ...
[22]
[PDF] 1 Sufficient statistic theorem (1)
If X is a Bernoulli random variable (with probability θ of success) then, by the ... ¯X)2 is the sum of squares about the average of a sample of size k from a ...<|control11|><|separator|>
[23]
[PDF] Chapter 8 The exponential family: Basics - People @EECS
The natural parameter space N is convex (as a set) and the cumulant function. A(η) is convex (as a function). If the family is minimal then A(η) is strictly ...<|control11|><|separator|>
[24]
[PDF] 18 The Exponential Family and Statistical Applications
A sufficient statistic is supposed to contain by itself all of the information about the unknown parameters of the underlying distribution that the entire ...<|control11|><|separator|>
[25]
[PDF] Statistical exponential families - arXiv
May 13, 2011 · The canonical exponential family decomposition yields: – t(x) = x is the sufficient statistic,. – θ = log λ are the natural parameters,. – F ...
[26]
Lesson 24: Sufficient Statistics - STAT ONLINE
In this lesson, we'll learn how to find statistics that summarize all of the information in a sample about the desired parameter.
[27]
https://www.randomservices.org/random/point/Sufficient.html
[28]
[PDF] Information and the Accuracy Attainable in the Estimation of ... - Gwern
The object of the paper is to derive certain inequality relations connecting the elements of the Information Matrix as defined by Fisher (1921) and the.
[29]
Rao-Blackwell theorem - Scholarpedia
Jul 21, 2008 · Rao-Blackwell Theorem provides a process by which a possible improvement in efficiency of an estimator can be obtained by taking its conditional expectation
[30]
[PDF] Chapter 6 Principle of Data Deduction - Arizona Math
Definition. A sufficient statistic T is called a minimal sufficient statistic provided that any sufficient statistic U, T is a function c(U) of U. • T is a ...Missing: seminal paper
[31]
[PDF] Bayesian sufficient statistics and invariance - Numdam
An essential sufficient statistic is defined and conditions are given under which it is equivalent to a Bayesian sufficient statistic.
[32]
How does Bayesian Sufficiency relate to Frequentist Sufficiency?
Apr 6, 2018 · However, I recently came across in a Bayesian book, with the definition P(θ|x,t)=P(θ|t). It's stated in the link that both are equivalent, but I ...Bayesian definition of sufficient statistics [duplicate] - Cross ValidatedIs there a difference between Bayesian and Classical sufficiency?More results from stats.stackexchange.com
[33]
A Bayesian Variation of Basu's Theorem and its Ramification in ...
Dec 22, 2023 · A Bayesian version of this result, where the parameter is treated as a random variable, is developed in this note, along with other extensions of the related ...
[34]
[PDF] sufficiency in linear models - Biblioteka Nauki
2.3. DEFINITION. A linear statistic Ay is said to be linearly sufficient for Y₁ if there is linear transformation B such that BAy is BLUE of Ey in Y₁.
[35]
Sufficiency and completeness in the linear model - ScienceDirect
This paper provides further contributions to the theory of linear sufficiency and linear completeness.
[36]
Linear sufficiency and completeness in the context of estimating the ...
The concept of linear sufficiency was introduced by Barnard (1963), Baksalary and Kala (1981), and Drygas (1983) while investigating those linear statistics ...
[37]
[PDF] Lecture 20: Linear model, the LSE, and UMVUE
The linear model is Xi = βτZi +εi. A least squares estimator (LSE) of β is any bβ ∈ B such that kX −Z bβk2 = min. If Z is full rank, there is a unique LSE.<|control11|><|separator|>
[38]
Linear sufficiency and completeness in the context of estimating the ...
In this paper we consider linear sufficiency and linear completeness in the context of estimating the estimable parametric function K′β under the general ...