Fact-checked by Grok 2 weeks ago

Estimation statistics

Estimation statistics is a data analysis framework that emphasizes estimating the magnitude and precision of effects using effect sizes and confidence intervals (CIs), rather than dichotomous decisions from null hypothesis significance testing (NHST).^[1] Developed as part of "the new statistics," it addresses limitations of traditional inferential statistics by focusing on quantification of uncertainty and practical significance.^[2] Originating in the late 20th century and gaining prominence in the 21st with proponents like Geoff Cumming, it promotes tools such as precision planning for study design and visualizations like the Gardner–Altman plot to aid interpretation. At its core, estimation statistics relies on point and interval estimation to approximate population parameters from sample data, but prioritizes effect size measures (e.g., Cohen's d) alongside CIs to convey the plausibility of different effect magnitudes.^[2] For example, a 95% CI for an effect size provides a range of values compatible with the data, highlighting uncertainty without binary rejection or acceptance of a null hypothesis. This approach aligns with evidence-based decision making in fields like psychology, medicine, and social sciences, where understanding effect precision informs real-world applications.^[3] While rooted in classical estimation methods, estimation statistics critiques the overreliance on p-values and advocates for reporting full estimation results to avoid misinterpretations.^[1] It encourages Bayesian perspectives in some contexts but primarily uses frequentist CIs, balancing bias and variance to enhance reliability in high-stakes research. As of 2025, it continues to influence open science practices through software like esci (Estimation Statistics with Confidence Intervals).^[4]

Introduction

Core Concepts

Estimation statistics represents a paradigm shift in statistical inference, emphasizing the direct estimation of population parameters, effect sizes, and associated uncertainties over the traditional reliance on null hypothesis significance testing (NHST). This approach seeks to provide more informative and nuanced insights into data by focusing on the magnitude and precision of effects rather than binary decisions about whether to reject a null hypothesis. By prioritizing estimation, researchers can better quantify the strength of relationships or differences in a study, facilitating practical interpretation and decision-making in fields such as psychology, medicine, and social sciences. Central to estimation statistics are principles that promote the assessment of estimate precision and the avoidance of dichotomous outcomes. Precision is evaluated through measures that indicate how closely a sample-based estimate approximates the true population value, often visualized via intervals that capture a range of plausible values. Compatibility intervals, proposed as an alternative framing for confidence intervals, represent the set of parameter values deemed compatible with the observed data at a specified level (e.g., 95%), shifting emphasis from long-run frequency properties to direct evidential support. This principle discourages interpretations like "significant" or "not significant," which can oversimplify evidence and lead to misleading conclusions, instead encouraging a continuous evaluation of uncertainty and effect magnitude. Key terminology in estimation statistics includes point estimates and interval estimates. A point estimate is a single value derived from sample data that serves as the best guess for an unknown population parameter; for instance, the sample mean \bar{x} estimates the population mean \mu. Interval estimates extend this by providing a range around the point estimate to quantify uncertainty, such as a confidence interval, which helps assess the reliability of the evidence supporting the estimate. These tools enable researchers to report not just a central tendency but also the degree of precision, promoting a more comprehensive understanding of the data's implications. Consider an example where a clinical trial estimates the effect size of a new treatment compared to a control, yielding a point estimate of 0.45 with a 95% confidence interval of [-0.2, 1.1]. This interval overlaps with zero, indicating that the data are compatible with no effect, but rather than declaring the result "non-significant," estimation statistics highlights the potential for a positive effect while underscoring the imprecision due to the wide range. A foundational component in constructing interval estimates is the standard error of the mean (SE), which measures the variability of the sample mean as an estimate of the population mean. The formula is given by

\text{SE} = \frac{\sigma}{\sqrt{n}},

where \sigma is the population standard deviation and n is the sample size.^[5] In practice, \sigma is often replaced by the sample standard deviation s when the population value is unknown. The SE decreases with larger n, reflecting improved precision, and is used to build intervals, such as \bar{x} \pm z \cdot \text{SE} for a 95% confidence interval where z \approx 1.96.^[6] This quantifies how sample data inform the likely range for the population parameter, central to the estimation paradigm.

Relation to Inferential Statistics

Inferential statistics encompasses methods for drawing conclusions about a population from a sample, with estimation forming one of its primary pillars alongside hypothesis testing.^[7] Point estimation yields a single value approximating an unknown population parameter, such as the sample mean as an estimate of the population mean, while interval estimation provides a range likely containing the parameter, incorporating uncertainty through measures like standard errors.^[7] This dual approach enables researchers to quantify both central tendencies and the precision of inferences, distinguishing estimation from mere data summarization by extending results beyond the observed sample to the broader population.^[8] Estimation statistics is firmly rooted in the frequentist framework, where confidence intervals are defined by their long-run frequency properties: across repeated samples from the same population, a specified proportion—such as 95%—of these intervals will contain the true parameter value.^[9] This interpretation emphasizes the procedure's reliability over any probability statement about a particular interval, aligning with Jerzy Neyman's foundational work on statistical estimation.^[9] In contrast, Bayesian credible intervals offer a subjective probability that the parameter falls within the interval given the data and prior beliefs, though without delving into their computational details here.^[10] A key application of estimation lies in evidence synthesis, particularly meta-analysis, where effect sizes from individual studies are pooled to derive a more precise overall estimate of an intervention's impact.^[11] For instance, in analyses of continuous outcomes, the standardized mean difference—Cohen's d—serves as a common effect size metric, computed as
d = \frac{M_1 - M_2}{SD_{\text{pooled}}}
where M_1 and M_2 are the means of the two groups, and SD_{\text{pooled}} is the pooled standard deviation; these d values are then weighted and combined across studies to assess cumulative evidence. This pooling enhances statistical power and generalizability, allowing estimation to integrate diverse findings into a cohesive summary.^[11] Estimation differs fundamentally from descriptive statistics by inferring population characteristics rather than merely describing sample features, relying on variability metrics like confidence intervals to gauge the reliability of extrapolations.^[8] Although it shares the frequentist foundations of Neyman-Pearson theory—developed for optimal hypothesis testing—estimation shifts emphasis from rejecting null hypotheses via test statistics to directly reporting intervals that capture parameter uncertainty.^[12] This focus promotes a more nuanced view of evidence, prioritizing effect magnitudes and precision over binary decisions.^[12]

Historical Development

Origins in the 20th Century

In the early 20th century, estimation practices were integral to biometric and agricultural statistics, where researchers grappled with small datasets from field experiments. William Sealy Gosset, publishing under the pseudonym "Student," introduced the t-distribution in 1908 to enable reliable estimation of means and their uncertainties in such limited samples, addressing the limitations of the normal distribution for small agricultural trials at the Guinness brewery.^[13] This work underscored estimation's centrality in practical sciences before formalized hypothesis testing paradigms emerged. Ronald A. Fisher advanced estimation further in his 1925 book Statistical Methods for Research Workers, which emphasized maximum likelihood as a method for obtaining precise parameter estimates in experimental data, particularly in agricultural research. Fisher viewed estimation as fundamental to inference, integrating it with his developing ideas on significance testing to provide researchers with tools for both point estimates and assessments of reliability.^[14] The 1930s saw intense debates between Fisher and Jerzy Neyman on inferential methods, elevating estimation's theoretical rigor. Fisher proposed fiducial inference around 1930 as a way to derive interval estimates from probability statements about parameters, aiming to bridge estimation and inference without Bayesian priors. In response, Neyman developed confidence intervals in his 1937 paper, offering a frequentist framework for interval estimation that quantified long-run coverage probabilities, distinguishing it from Fisher's approach and solidifying estimation's role in hypothesis evaluation.^[15] These exchanges highlighted estimation's potential for nuanced uncertainty quantification amid growing interest in decision-oriented testing. By the World War II era, estimation approaches were increasingly sidelined in applied statistics, as the demand for rapid, binary decisions in military and industrial contexts favored p-value-based significance testing for its simplicity and standardized tables.^[16] Fisher's accessible methods proliferated postwar, overshadowing detailed estimation in favor of quick hypothesis assessments, though the foundational debates had already embedded interval-based estimation in statistical theory.^[16]

Key Proponents and Shifts in the 21st Century

In the early 21st century, the replication crisis in psychology, which gained prominence in the mid-2000s through high-profile failed replications of seminal studies, highlighted the limitations of null hypothesis significance testing (NHST) and catalyzed a shift toward estimation statistics as a more reliable approach for quantifying effects and uncertainty. This crisis, exemplified by the Open Science Collaboration's 2015 project replicating only 36% of 100 psychological studies with significant results,^[17] prompted widespread calls for emphasizing effect sizes and confidence intervals over binary p-value decisions to enhance reproducibility.^[18] Geoffrey Cumming emerged as a leading advocate for this paradigm shift through his 2012 book, Understanding The New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis, which critiques NHST's focus on dichotomous outcomes and promotes estimation practices for providing nuanced insights into effect magnitudes and precision in behavioral sciences.^[19] Cumming's work, grounded in meta-analytic examples from psychology, argues that confidence intervals offer a superior framework for inference by visualizing the range of plausible effect sizes, influencing educational curricula and research guidelines in the social sciences.^[20] The American Statistical Association reinforced this movement with its 2016 statement on p-values and statistical significance, which explicitly warns against the misuse of p-values for causal claims or probability assessments and endorses estimation approaches, such as confidence intervals and effect sizes, to foster more informative statistical communication across disciplines.^[21] This influential document, endorsed by over 60 statisticians, underscores that proper inference relies on evaluating compatibility with data models through estimation rather than arbitrary significance thresholds.^[22] In 2021, the ASA's task force further clarified and reinforced these principles, emphasizing the valid role of p-values alongside estimation methods while addressing common misinterpretations to promote better statistical practice.^[23] In response to the replication crisis, journals like Advances in Methods and Practices in Psychological Science (launched in 2018) have mandated the reporting of effect sizes accompanied by confidence intervals in submissions, promoting estimation as a core practice to improve methodological rigor and transparency in psychological research.^[24] Key proponents in education statistics, such as Lisa Harlow, have further advanced this shift; as editor of the 1997 volume What If There Were No Significance Tests?, Harlow advocates for alternatives like confidence intervals and Bayesian methods to replace NHST in quantitative psychology and educational research.^[25] Post-2010, the rise of open science practices, including pre-registration of studies on platforms like the Open Science Framework, has integrated estimation statistics by requiring explicit reporting of uncertainty through confidence intervals and effect sizes, thereby reducing selective reporting and enhancing the credibility of findings in social and behavioral sciences.^[26] This integration, as seen in Registered Reports formats adopted by journals since 2013, ensures that estimation-focused analyses are planned and transparently documented upfront, addressing replication issues by prioritizing effect quantification over significance.^[27] By 2023, over 300 journals across disciplines had adopted Registered Reports, reflecting the format's widespread impact on promoting estimation-based research.^[28] Such practices have extended to fields like medicine, where estimation aids in robust clinical inference amid similar reproducibility concerns.^[29]

Methodological Foundations

Point and Interval Estimation

Point estimation involves selecting a single value from a sample to serve as the best approximation of an unknown population parameter. Common methods include the method of moments, introduced by Karl Pearson, which equates population moments to corresponding sample moments to solve for parameters. For example, in estimating the proportion p of a binomial distribution, the point estimate is the sample proportion \hat{p} = k/n, where k is the number of successes in n trials. Another prominent approach is maximum likelihood estimation, developed by Ronald A. Fisher, which selects the parameter value that maximizes the likelihood function of observing the sample data.^[30] Interval estimation extends point estimates by constructing a range of plausible values for the parameter, typically as confidence intervals that incorporate uncertainty. For the population mean \mu from a normally distributed sample with unknown variance, a (1 - \alpha) \times 100\% confidence interval is given by

\bar{x} \pm t_{\alpha/2, n-1} \cdot \frac{s}{\sqrt{n}},

where \bar{x} is the sample mean, s is the sample standard deviation, n is the sample size, and t_{\alpha/2, n-1} is the critical value from the t-distribution with n-1 degrees of freedom.^[31] These intervals rely on assumptions such as normality of the population for small samples (n < 30); for larger samples, the central limit theorem justifies approximate normality of the sampling distribution of \bar{x} under conditions of independent and identically distributed observations with finite variance.^[32] The coverage probability of such an interval is $1 - \alpha, meaning that in repeated sampling, 95% of intervals (for \alpha = 0.05) will contain the true parameter value.^[33] Desirable properties of point estimators include unbiasedness and efficiency, evaluated via metrics like mean squared error (MSE). An unbiased estimator has an expected value equal to the true parameter, such as the sample variance s^2 = \frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})^2, which corrects for bias in the population variance estimate by dividing by n-1 rather than n. MSE quantifies overall performance as \text{MSE}(\hat{\theta}) = \text{Var}(\hat{\theta}) + [\text{Bias}(\hat{\theta})]^2, balancing variance and bias.^[34] For non-normal data, standard intervals may be asymmetric due to skewed sampling distributions, prompting robust alternatives like bootstrap methods, pioneered by Bradley Efron, which resample the data with replacement to empirically approximate the distribution and construct percentile-based intervals.^[35]

Effect Size Measures

Effect size measures provide standardized, scale-free quantifications of the magnitude of an observed phenomenon, enabling comparisons across studies and contexts while emphasizing practical significance over mere statistical detection. In estimation statistics, these measures complement point and interval estimates by focusing on the substantive importance of effects, facilitating better interpretation of uncertainty and replicability without reliance on arbitrary significance thresholds. Unlike raw differences, effect sizes normalize variations to a common metric, such as standard deviations or proportions, which is crucial for assessing real-world relevance in diverse fields like psychology and medicine. Among the most widely used effect size measures is Cohen's d, which quantifies the standardized difference between two group means: d = \frac{\mu_1 - \mu_2}{\sigma}, where \mu_1 and \mu_2 are the population means of the two groups, and \sigma is the pooled population standard deviation. For correlations, Pearson's r serves as a common effect size, representing the strength and direction of the linear relationship between two continuous variables, ranging from -1 to 1. In analyses of binary outcomes, the odds ratio (OR) measures association in 2x2 contingency tables as OR = \frac{a/b}{c/d}, where a and b are the frequencies in the exposed group (event and non-event), and c and d are those in the unexposed group; an OR of 1 indicates no association, values greater than 1 suggest a positive association, and values less than 1 indicate a negative one. Interpretation of effect sizes often follows Jacob Cohen's conventional benchmarks for Cohen's d: small (0.2), medium (0.5), and large (0.8), intended as rough guides for behavioral sciences where these magnitudes correspond to noticeable differences in everyday observations. However, these thresholds are arbitrary and context-dependent, varying by research domain— for instance, smaller effects may be meaningful in large-scale public health studies, while larger ones are expected in controlled lab settings—necessitating domain-specific judgment over rigid application. Similar guidelines apply to r (small: 0.1, medium: 0.3, large: 0.5) and OR (though less standardized, values around 1.5–2.0 often denote moderate effects in epidemiology). Confidence intervals for effect sizes enhance precision by conveying both the estimate and its uncertainty; for Cohen's d, these are commonly constructed using the non-central t-distribution to account for the sampling variability of the t-statistic, where the interval is derived by solving for the effect size that matches the observed t-value's percentile in the non-central distribution with degrees of freedom equal to the sample size minus 2. This approach yields asymmetric intervals that better reflect the true sampling distribution, particularly for small samples, and can be implemented via software or approximation formulas.^[36] A bias-corrected variant of Cohen's d is Hedges' g, designed to adjust for positive bias in small samples: g = d \left(1 - \frac{3}{4(df) - 1}\right), where df = n_1 + n_2 - 2 is the degrees of freedom; this correction is negligible for large samples but reduces overestimation when n < 20. Hedges' g is particularly valuable in meta-analyses, where unbiased pooling across studies improves overall effect size estimates.

Visualization and Interpretation Techniques

Gardner–Altman Plot

The Gardner–Altman plot is a visualization technique designed to display confidence intervals for group comparisons, emphasizing the estimation of effect sizes over null hypothesis significance testing. Introduced by Michael J. Gardner and Douglas G. Altman in 1986, it facilitates the assessment of evidence by showing the compatibility of confidence intervals with zero, thereby highlighting the precision and uncertainty of differences between groups without relying on p-values.^[37] This approach aligns with broader estimation statistics principles, where confidence intervals provide a range of plausible values for the true effect, as opposed to binary decisions from hypothesis tests. The plot consists of two vertically aligned panels: the left panel shows the raw data or summary statistics (such as means) for each group along a shared y-axis, often with error bars representing confidence intervals for individual group means; the right panel, on a floating axis, depicts the effect size—typically the mean difference between groups—along with its confidence interval. To construct the plot, first calculate the group means and their confidence intervals using standard parametric methods or non-parametric alternatives like bootstrapping for robustness against distributional assumptions; then, compute the difference in means and its confidence interval via bootstrapping, which involves resampling the data 5000–10000 times to generate the interval; finally, align the panels so the right panel's y-axis scale matches the left but is shifted to center the effect size estimate.^[37]^[38] This alignment allows visual inspection of overlap between the effect size confidence interval and zero, indicating the strength of evidence for a meaningful difference. One key advantage of the Gardner–Altman plot is its ability to handle multiple comparisons by displaying several effect sizes on the right panel, enabling direct visual evaluation of compatibility across contrasts without inflating error rates associated with post-hoc tests. For instance, in a clinical trial comparing blood pressure reductions between two antihypertensive treatments (e.g., a new drug versus placebo), the left panel might show individual patient reductions with means and 95% confidence intervals, while the right panel illustrates the mean difference (e.g., -8 mm Hg) with its interval (-13 to -3 mm Hg), demonstrating a likely beneficial effect.^[37] This format promotes intuitive understanding of effect magnitude and precision, aiding decisions in fields like medicine and psychology. An update to the original design incorporates swarm or strip plots of raw data points on the left panel to enhance intuition about data distribution and variability, alongside bootstrap confidence intervals for the effect size to accommodate non-normal data. This modern variant, popularized through open-source software implementations, improves transparency by revealing the full dataset alongside summaries, making it particularly useful for small samples or exploratory analyses.^[38]

Cumming Plot

The Cumming plot is a visualization technique in estimation statistics that employs a metaphorical representation to aid in the interpretation of confidence intervals (CIs) for effect sizes or other parameters. In this approach, the CI is depicted as a "dance floor," a horizontal line segment indicating the range of plausible values for the true population parameter, while the point estimate—such as a sample mean difference—is portrayed as a "dancer" positioned at its center, symbolizing the observed value amid potential variability. This analogy highlights the uncertainty inherent in estimation, where the true parameter is likely to lie somewhere within the bounds of the dance floor, rather than at a single fixed point.^[39] The primary purpose of the Cumming plot is to facilitate intuitive understanding of estimation results by emphasizing compatibility and precision over dichotomous significance testing. For instance, if the vertical line representing the null value (typically zero for effect sizes) intersects the dance floor, the plot illustrates that the null hypothesis remains compatible with the data, indicating uncertainty about the effect rather than a definitive rejection. This method promotes a focus on the magnitude and reliability of effects, encouraging researchers to consider the full range of plausible outcomes. Developed by statistician Geoffrey Cumming, the technique was introduced around 2012 as part of a broader advocacy for estimation-based approaches in the social and behavioral sciences.^[39] In terms of construction, a basic Cumming plot consists of a horizontal line denoting the CI endpoints, with the point estimate marked at the midpoint, and a vertical line drawn at zero to assess overlap. For multiple studies or replications, the plot can be extended by representing each CI as an adjacent or overlapping dance floor, visually demonstrating consistency or divergence across datasets—such as in meta-analyses where compatible intervals reinforce evidence for an effect. This simple yet effective design avoids clutter while conveying key inferential insights.^[39] A representative example involves a psychological experiment examining the effect size of a cognitive intervention, yielding a 95% CI of [0.1, 0.9] for the standardized mean difference. In the Cumming plot, the dance floor spans from 0.1 to 0.9, with the point estimate (e.g., 0.5) as the dancer; since zero falls within this interval, the visualization underscores weak evidence against the null, suggesting the true effect could plausibly include no difference, thereby guiding cautious interpretation.^[39] The Cumming plot has been integrated into interactive software tools to enhance its practical application, notably in the Exploratory Software for Confidence Intervals (ESCI), which allows users to dynamically adjust parameters and simulate CI variability for educational and analytical purposes.^[4]

Other Visualization Methods

Forest plots are widely used in meta-analysis to summarize effect sizes and their associated confidence intervals across multiple studies, with each study's estimate represented by a point and its interval as a horizontal line segment, often culminating in a diamond-shaped summary for the pooled effect. This visualization facilitates the assessment of consistency and overall uncertainty in estimation results by allowing direct comparison of intervals' overlaps and widths.^[40] Raincloud plots integrate raw data points, kernel density estimates (via half-violin shapes), and summary statistics including box plots with medians and confidence intervals, providing a comprehensive view of data distribution and estimation precision in a single graphic. Proposed as a robust alternative to traditional box or bar plots, these visualizations emphasize the full dataset alongside interval estimates to better convey variability and avoid over-reliance on aggregates.^[41] In software implementations, R's ggplot2 package enables flexible customization of confidence interval visualizations, such as through geom_errorbar() for error bars or geom_ribbon() for shaded bands around fitted lines, supporting layered plots that incorporate estimation results from models like linear regressions. Similarly, the jamovi statistical software includes the esci module, which generates interactive plots for effect sizes, confidence intervals, and meta-analytic summaries, streamlining the depiction of estimation outcomes for users without advanced coding. Adoption of such estimation-focused visualizations has been encouraged in the American Psychological Association's 7th edition style guidelines (published 2020), which prioritize reporting intervals and effect sizes over p-values to enhance transparency in scientific communication.^[42]^[43] Bootstrap clouds visualize the variability of confidence intervals by plotting multiple resampled distributions or interval endpoints as a scattered "cloud" of points, illustrating the sampling distribution's spread and potential range of estimates without assuming normality. This approach, derived from bootstrap resampling techniques, helps quantify uncertainty in non-parametric settings by showing how intervals might shift across repeated samplings.^[44] TOST equivalence plots depict the regions of practical equivalence for effect sizes using two one-sided tests, with horizontal lines or shaded bands representing the predefined equivalence bounds and vertical lines for observed confidence intervals to indicate whether the estimate falls within the non-inferiority margins. These plots are particularly useful for visualizing decisions on practical equivalence in estimation contexts, such as clinical trials, by overlaying the interval against the equivalence region to assess overlap.^[45]

Critiques of Null Hypothesis Significance Testing

Inherent Limitations

Null hypothesis significance testing (NHST) fundamentally operates through a dichotomous decision framework, where results are deemed either "statistically significant" or "not significant" based on whether the p-value falls below an arbitrary threshold, conventionally set at α = 0.05. The p-value itself is defined as the probability of observing data at least as extreme as that obtained, assuming the null hypothesis (H₀) is true:

p = P(T > t_{\text{obs}} \mid H_0)

where T is the test statistic and t_{\text{obs}} is the observed value. This threshold, originally proposed by Ronald Fisher as a convenient benchmark rather than a strict boundary, introduces inherent risks of type I errors (false positives, controlled at rate α) and type II errors (false negatives, controlled at rate β), without balancing the two in a way that reflects true uncertainty.^[46] A core weakness exacerbating type II errors is the prevalence of low statistical power in many studies, which represents the probability (1 - β) of correctly rejecting H₀ when it is false. In fields like psychology, typical studies conducted before 2010 often operated at around 50% power for detecting medium effect sizes, meaning half of genuine effects went undetected and contributed to false negatives. This underpowering stems from small sample sizes and optimistic effect size assumptions, systematically inflating the rate of non-rejections without providing meaningful insight into effect existence.^[47] Furthermore, NHST yields non-informative outcomes when H₀ is not rejected, as this merely indicates insufficient evidence to dismiss the null rather than affirmative support for it or quantification of effect magnitude. Unlike estimation approaches that provide intervals or point estimates to gauge uncertainty, a non-significant result offers no probabilistic statement about the absence or size of an effect, leaving researchers unable to distinguish between true nulls and underpowered tests.^[48] Compounding these issues is the "statistical significance filter," where publication practices prioritize significant results (p < 0.05), systematically biasing the literature toward exaggerated effect sizes from low-powered studies. This filter selects for inflated estimates—often by factors of 2-4 times the true value—creating an overoptimistic view of replicability and effect robustness, while suppressing null or small effects that might better represent reality.^[49]

Misinterpretations and Practical Issues

One common misinterpretation in null hypothesis significance testing (NHST) involves p-hacking, where researchers selectively analyze data—through practices like optional stopping, exclusion of outliers, or multiple analytic paths—until a statistically significant p-value is obtained, often without disclosing these decisions. This data dredging inflates the type I error rate, as demonstrated by simulations showing that even conservative combinations of such flexible strategies can raise false positives from the nominal 5% to over 60%. Such practices undermine the reliability of published findings, particularly in fields like psychology where analytic flexibility is high.^[50] Another prevalent issue is HARKing (hypothesizing after the results are known), in which post-hoc interpretations are retroactively framed as pre-registered predictions, masking exploratory analyses as confirmatory ones. This approach systematically increases type I errors by capitalizing on chance patterns in the data without accounting for the multiplicity of hypotheses tested. HARKing distorts the scientific record, as it discourages replication of genuine effects while promoting spurious ones as theoretically grounded.^[51] Publication bias exacerbates these problems through the file drawer problem, where studies yielding non-significant results are disproportionately withheld from publication, leaving journals filled primarily with positive findings and biasing meta-analyses toward inflated effect sizes. Rosenthal quantified this tolerance for null results, estimating that for a meta-analytic effect to withstand the "file drawer" of unpublished studies, tens to hundreds of suppressed non-significant reports would need to exist depending on the observed significance level.^[52] This selective reporting skews cumulative evidence, as seen in systematic reviews where apparent effects diminish upon including gray literature.^[53] A notable illustration of p-value manipulation emerged from a 2011 study by Bakker and Wicherts, who examined statistical reporting in 281 articles from psychology journals and identified inconsistencies—such as p-values incompatible with reported test statistics—in approximately 54% of articles with exactly reported p-values, with an average of about one error per article and some containing over a dozen. These discrepancies often suggested selective rounding or adjustment to achieve significance thresholds, highlighting systemic issues in reporting integrity.^[54] The multiple comparisons problem further compounds misinterpretations when researchers perform numerous tests without correction, erroneously treating each p-value in isolation and ignoring the cumulative risk of false positives across the family of tests. This elevates the family-wise error rate (FWER), the probability of at least one type I error in the set; for instance, 20 independent tests at α = 0.05 yield an FWER of approximately 64% without adjustment.^[55] The Bonferroni correction mitigates this by setting the per-test α to 0.05 divided by the number of comparisons, ensuring the overall FWER remains at 0.05, though it can be conservative for large test families.^[56] Failure to apply such controls is widespread in exploratory research, leading to overconfident claims of significance.^[57]

Advantages of Estimation Approaches

Enhanced Quantification of Uncertainty

Confidence intervals (CIs) provide a more comprehensive assessment of uncertainty than point estimates alone by delineating a range of plausible values for the population parameter, allowing researchers to evaluate the precision and potential variability of their findings. Unlike a single point estimate, which offers only a best guess without context for reliability, a narrow CI indicates high precision, suggesting that the true value is likely close to the estimate, whereas a wide CI signals greater uncertainty and the need for caution in interpretation. This range-based approach facilitates better decision-making in fields like medicine and psychology, where understanding the span of possible effects is crucial for practical application. The compatibility interpretation of CIs emphasizes that the interval represents the set of parameter values compatible with the observed data at a specified level, such as 95%, rather than a probabilistic statement about the true value. For instance, if a 95% CI excludes zero for an effect size, the data are incompatible with a null effect of zero at that level, but this does not prove the effect's existence or magnitude, avoiding the overreach common in significance testing. This perspective shifts focus from binary decisions to the evidential support for various hypotheses within the interval, promoting a nuanced view of results. The width of a CI serves as a key precision metric and is approximately proportional to \frac{1}{\sqrt{n}}, where n is the sample size, meaning that doubling the sample size roughly halves the interval width and thus reduces uncertainty. This relationship underscores the importance of adequate sample sizes in study design to achieve desired precision levels. Researchers can use CI width to quantify how informative their data are, with narrower intervals providing stronger evidence for the estimated effect. In the frequentist framework, a 95% CI does not imply a 95% probability that the true parameter lies within the specific calculated interval; instead, it means that if the sampling procedure were repeated many times, 95% of the resulting intervals would contain the true parameter. This interpretation guards against overconfidence by reminding analysts that the observed interval is just one realization, and the true value could plausibly fall outside it, encouraging humility in conclusions drawn from data. For example, a 95% CI of [0.3, 1.2] for a risk ratio in an epidemiological study indicates that the true risk ratio is compatible with values suggesting a reduced risk (below 1) up to a slight increase (above 1), reflecting a positive but uncertain effect overall with moderate precision depending on the sample size.

Precision in Study Design and Planning

In estimation statistics, sample size planning emphasizes achieving a desired level of precision in parameter estimates, rather than solely powering a study to detect a specific effect under null hypothesis significance testing (NHST). This approach determines the required sample size n to obtain a confidence interval (CI) of a specified width w, ensuring the estimate is informative regardless of the true effect size. For estimating a population mean with known standard deviation \sigma, the formula is n \approx \left( \frac{2 z \sigma}{w} \right)^2, where z is the z-score corresponding to the desired confidence level (e.g., z = 1.96 for a 95% CI). This method prioritizes the half-width of the CI as the margin of error, allowing researchers to plan for practical utility in decision-making.^[58] Precision-based power analysis extends this by focusing on attaining a CI of a target width, such as \pm 0.5 effect units, to quantify uncertainty adequately without assuming a particular effect size. For instance, in planning a study on treatment effects, researchers might specify a desired precision around the standardized mean difference, calculating n to ensure the 95% CI falls within that range based on anticipated variability. This contrasts with traditional NHST power calculations, which require a hypothesized effect size and risk underpowering if the assumption is incorrect; precision planning instead guarantees informativeness by targeting the expected CI width directly. Tools like the R package presize facilitate these computations for various parameters, including means and proportions. Sequential analysis in estimation-oriented designs incorporates adaptive stopping rules based on achieved precision, allowing trials to halt early if the CI narrows sufficiently to inform decisions. In such adaptive designs, interim analyses monitor the CI width after collecting portions of the data (e.g., 50% of planned n), stopping if precision meets the predefined criterion. Since these designs focus on estimation without formal hypothesis testing, they avoid Type I error concerns.^[20] This approach enhances efficiency in resource-limited settings, such as clinical trials, by focusing on estimation accuracy rather than fixed power thresholds. Compared to NHST power planning, precision-based methods offer key advantages: they avoid reliance on potentially optimistic effect size guesses, promote studies that are always informative by ensuring narrow CIs, and align better with the replication crisis by emphasizing estimation over dichotomous decisions.

Alignment with Evidence-Based Decision Making

Estimation statistics supports evidence-based decision making by enabling the synthesis of research findings across studies, particularly through meta-analytic integration. In meta-analysis, confidence intervals (CIs) from individual studies are pooled using inverse-variance weighting, where each study's effect estimate is weighted by the inverse of its variance to produce an overall effect size with associated uncertainty. This method prioritizes more precise studies, yielding a robust summary estimate that reflects the cumulative evidence and facilitates informed decisions in cumulative science. In clinical and policy contexts, estimation approaches enhance decision frameworks by incorporating CIs into risk-benefit analyses. For example, in medicine, an odds ratio (OR) of 1.1 with a 95% CI of [0.8, 1.5] indicates uncertain benefit, as the interval spans values compatible with no effect or potential harm, guiding clinicians to weigh practical implications rather than binary significance. This focus on interval-based uncertainty promotes nuanced assessments in fields like pharmacology and public health, where decisions must account for the precision and magnitude of effects.^[59]^[60] Policy implications of estimation statistics include a shift away from overreliance on statistical significance toward effect sizes and CIs in guideline development. The PRISMA 2020 statement for reporting systematic reviews emphasizes presenting effect estimates with CIs to better inform health policies, reducing misinterpretation of p-values and supporting decisions based on practical relevance. This aligns with broader calls in organizations like the American Statistical Association to prioritize estimation for transparent, reproducible policy advice.^[61] Educational reforms in psychology and education increasingly emphasize—as of 2024—training in estimation statistics to cultivate evidence-based practice. Curricula now incorporate interval estimation and meta-analytic thinking, moving beyond null hypothesis significance testing to equip practitioners with tools for interpreting uncertainty in real-world applications. For instance, recent discussions highlight the ongoing need for dedicated estimation modules to address gaps in statistical education.^[62]^[63]^[64] Recent trends, including open science practices, further promote estimation over NHST in teaching and research to enhance reproducibility and practical inference.^[65] Reporting guidelines, such as those advocating for clear presentation of sample details, uncertainty measures, effect ranges, and point estimates, further support this training by standardizing evidence communication.^[66]

References

[1]
STAT340 Lecture 06: Estimation
Estimation refers to the task of giving a value or range of values that are a “good guess” about some quantity out there in the world. Often this quantity is ...
[2]
[PDF] Estimation theory
An “estimator” is any decision rule, that is, any function from the data space XN into the parameter space Θ.
[3]
[PDF] Overview of Estimation - Arizona Math
The estimate is a statistic. ˆθ : data → Θ. Introduction to estimation in the classical approach to statistics is based on two fundamental questions: • How ...
[4]
1.3.5.3. Two-Sample <i>t</i>-Test for Equal Means
... STANDARD ERROR OF THE MEAN = 0.40652 SAMPLE 2: NUMBER OF OBSERVATIONS = 79 MEAN = 30.48101 STANDARD DEVIATION = 6.10771 STANDARD ERROR OF THE MEAN = 0.68717.
[5]
What Is Standard Error? | How to Calculate (Guide with Examples)
Dec 11, 2020 · Standard error formula The standard error of the mean is calculated using the standard deviation and the sample size.
[6]
Statistical Inference and Estimation | STAT 504
Estimation represents ways or a process of learning and determining the population parameter based on the model fitted to the data. Point estimation and ...
[7]
[PDF] 5: Introduction to Estimation
Both estimation and NHTS are used to infer parameters. A parameter is a statistical constant that describes a feature about a phenomena, population, pmf, or pdf ...
[8]
Statistical tests, P values, confidence intervals, and power: a guide ...
Confidence intervals are examples of interval estimates. Neyman [76] proposed the construction of confidence intervals in this way because they have the ...
[9]
Understanding and interpreting confidence and credible intervals ...
Dec 31, 2018 · Frequentist 95% CI: we can be 95% confident that the true estimate would lie within the interval. •. Bayesian 95% CI: there is a 95% probability ...
[10]
Estimation in meta‐analyses of mean difference and standardized ...
Arguably, the main purpose of a meta‐analysis is to provide point and interval estimates of an overall effect. Usually, after estimating the between‐study ...
[11]
[PDF] The Fisher, Neyman-Pearson Theories of Testing Hypotheses
The Fisher and Neyman-Pearson approaches to testing statistical hypotheses are compared with respect to their attitudes to the interpretation.
[12]
The Probable Error of a Mean - Biometrika - jstor
The probable error of a mean is the uncertainty in the mean of a sample due to random sampling, where the sample mean deviates from the population mean.<|control11|><|separator|>
[13]
Fisher (1925) Chapter 1 - Classics in the History of Psychology
The following example exhibits in a relatively simple case the application of the method of maximum likelihood to discover a statistic capable of giving an ...
[14]
Outline of a Theory of Statistical Estimation Based on the Classical ...
Abstract. We shall distinguish two aspects of the problems of estimation . (i) the practical and (ii) the theoretical. The practical aspect may be described as ...
[15]
Using History to Contextualize p-Values and Significance Testing
The history helps clarify today's debates, adding a long-term dimension to modern discussions.
[16]
The replication crisis has led to positive structural, procedural, and ...
Jul 25, 2023 · Pooling the Open Science Collaboration replications with 207 other replications from recent years resulted in a higher estimate; 64% of effects ...
[17]
Understanding The New Statistics: Effect Sizes, Confidence Intervals,
Understanding The New Statistics Effect Sizes, Confidence Intervals, and Meta-Analysis. By Geoff Cumming Copyright 2012. Paperback $71.99. Hardback $140.00.
[18]
Understanding The New Statistics | Effect Sizes, Confidence Intervals,
Jun 19, 2013 · Understanding The New Statistics ; Edition 1st Edition ; First Published 2012 ; eBook Published 19 June 2013 ; Pub. Location New York ; Imprint ...
[19]
[PDF] p-valuestatement.pdf - American Statistical Association
Mar 7, 2016 · The American Statistical Association (ASA) has released a “Statement on Statistical Significance and P-Values” with six principles ...
[20]
The ASA Statement on p-Values: Context, Process, and Purpose
Jun 9, 2016 · ASA Statement on Statistical Significance and P-Values. Ronald L. Wasserstein American Statistical Association. Pages 129-133. 1.
[21]
Advances in Methods and Practices in Psychological Science ...
Apr 29, 2024 · Advances in Methods and Practices in Psychological Science (AMPPS) welcomes submissions that communicate advances in methods, practices, and ...
[22]
What If There Were No Significance Tests? - 1st Edition - Routledge
Free delivery"What If There Were No Significance Tests? is a thought-provoking book and worthy of the attention of anyone who is interested in the question of whether ...
[23]
Preregistration - Center for Open Science
When you preregister your research, you're simply specifying your research plan in advance of your study and submitting it to a registry.Missing: estimation uncertainty
[24]
The preregistration revolution - PNAS
Progress in science relies in part on generating hypotheses with existing observations and testing hypotheses with new observations.
[25]
What the replication crisis means for intervention science - PMC
The “replication crisis” not only highlights the limitations of traditional statistical approaches and the circumscribed requirements for scientific publication ...
[26]
On the mathematical foundations of theoretical statistics - Journals
A recent paper entitled "The Fundamental Problem of Practical Statistics," in which one of the most eminent of modern statisticians presents what purports to ...
[27]
2.5 - A t-Interval for a Mean | STAT 415
So far, we have shown that the formula: x ¯ ± z α / 2 ( σ n ). is appropriate for finding a confidence interval for a population mean if two conditions are ...
[28]
The 6 Confidence Interval Assumptions to Check - Statology
In order to apply the Central Limit Theorem, our sample size must be sufficiently large. In general, we consider “sufficiently large” to be 30 or larger.
[29]
Confidence Intervals | Introduction to Data Science
This probability is called the coverage probability, and is most often taken to be 95%. Thus, we aim to construct an interval that is just wide enough so that ...
[30]
[PDF] Evaluating the Performance of Estimators (Section 7.3)
Historically, estimators have been most frequently compared using Mean Squared Error: MSE(θ) = Eθ(ˆθ− θ)2. This is because the MSE can often be calculated or ...
[31]
Confidence Intervals for the Mean of Non-Normal Distribution
In this paper, we compare various methods for constructing confidence intervals when data are non-normally distributed. Three of the most popular and commonly ...Missing: asymmetric | Show results with:asymmetric
[32]
[PDF] A review of effect sizes and their confidence intervals, Part I
There are, at least, three different methods to build confi- dence intervals for Cohen's d effect sizes. The three meth- ods involve: (1) a noncentral t ...
[33]
Confidence intervals rather than P values - The BMJ
Mar 15, 1986 · Confidence intervals present a range of values, on the basis of the sample data, in which the population value for such a difference may lie.Missing: plot | Show results with:plot
[34]
Moving beyond P values: data analysis with estimation graphics - Nature Methods
### Gardner-Altman Estimation Plot Summary
[35]
https://www.scirp.org/journal/paperinformation?paperid=76758
[36]
esci (Estimation Statistics with Confidence Intervals)
Pronounced 'ESS-key'; Free, open-source, no textbook purchase required. esci has two sides. Simulations and explorations (including “Dance of the Means”) ...
[37]
How to Interpret a Meta-Analysis Forest Plot - PMC - NIH
May 3, 2021 · The width of the diamond shows the confidence interval for the overall effect. Each forest plot contains a vertical line, the line of 'no effect ...
[38]
Raincloud plots: a multi-platform tool for robust data visualization
Apr 1, 2019 · These “raincloud plots” can visualize raw data, probability density, and key summary statistics such as median, mean, and relevant confidence intervals.Missing: Hart | Show results with:Hart
[39]
Linear model and confidence interval in ggplot2 - The R Graph Gallery
Display the result of a linear model and its confidence interval on top of a scatterplot. A ggplot2 implementation with reproducible code.
[40]
The esci module for jamovi
Jun 9, 2020 · The esci module in jamovi focuses on effect sizes, confidence intervals, and meta-analysis, providing estimates and confidence intervals for ...Tl;Dr · What Is Estimation... · An Example With Esci
[41]
Calibrating and Visualizing Some Bootstrap Confidence Regions
Sep 25, 2024 · A new bootstrap confidence region uses a simple prediction region calibration technique to improve the coverage.
[42]
16.2 Two One-Sided Tests Equivalence Testing | A Guide on Data ...
The Two One-Sided Tests (TOST) procedure is a method used in equivalence testing to determine whether a population effect size falls within a range of ...
[43]
Problems and alternatives of testing significance using null ... - NIH
May 30, 2023 · The P-value does not measure the probability that the research hypothesis is true or the probability that the data were created by chance;. One ...
[44]
https://www.mdpi.com/2075-1680/13/10/659
[45]
Null hypothesis significance testing: a short tutorial - PMC - NIH
By failing to reject, we simply continue to assume that H0 is true, which implies that one cannot argue against a theory from a non-significant result (absence ...
[46]
The statistical significance filter leads to overoptimistic expectations ...
First, in the NHST way of thinking, a parameter in the frequentist paradigm is an unknown point value. Once one assumes that a parameter can only have a fixed ...
[47]
The Extent and Consequences of P-Hacking in Science - PMC - NIH
Mar 13, 2015 · One type of bias, known as “p-hacking,” occurs when researchers collect or select data or statistical analyses until nonsignificant results become significant.
[48]
HARKing: hypothesizing after the results are known - PubMed
HARKing is defined as presenting a post hoc hypothesis (ie, one based on or informed by one's results) in one's research report as if it were, in fact, an a ...
[49]
[PDF] The "File Drawer Problem" and Tolerance for Null Results
Both behavioral researchers and statisti- cians have long suspected that the studies published in the behavioral sciences are a biased sample of the studies ...
[50]
Publication Bias in Meta-Analysis: Confidence Intervals for ...
This issue was memorably termed as the file-drawer problem by Rosenthal [11]; nonsignificant results are stored in file drawers without ever being published.
[51]
The (mis)reporting of statistical results in psychology - ResearchGate
Aug 7, 2025 · Typically, this involves checking whether reported p-values are consistent with their accompanying test statistics and degrees of freedom ( ...Missing: audit manipulation
[52]
Zen and the Art of Multiple Comparisons - PMC - NIH
We focus on two such metrics, namely the family-wise error rate (FWER), defined as the probability of obtaining at least one false positive in a family of tests ...3.1 Fwer · Bonferroni Correction · 3.3. Comparing Fwer And Fdr...<|control11|><|separator|>
[53]
4.2 - Controlling Family-wise Error Rate | STAT 555
The most commonly used method which controls FWER at level α α is called Bonferroni's method. It rejects the null hypothesis when p<α/m.
[54]
The fallacy of using family-based error rates to make inferences ...
During multiple testing, researchers often adjust their alpha level to control the familywise error rate for a statistical inference about a joint union ...
[55]
[PDF] Statistics: An introduction to sample size calculations - Statstutor
To summarise, in order to carry out any precision-based sample size calculation you need to decide how wide you want your confidence interval to be and you need ...
[56]
Power, Precision, Sample Size in Sport Science
Jun 19, 2020 · ... planning a study based on obtaining a given precision in the parameter estimate has some advantages over the use of power. Sequential ...Missing: NHST | Show results with:NHST
[57]
Odds Ratio - StatPearls - NCBI Bookshelf
May 22, 2023 · The odds ratio (OR) is a measure of how strongly an event is associated with exposure. The odds ratio is a ratio of two sets of odds: the odds ...
[58]
What's the Risk: Differentiating Risk Ratios, Odds Ratios, and ...
Aug 26, 2020 · Having a confidence interval between 1.5 and 4.1 for the risk ratio indicates that patients with a prolonged QTc interval were 1.5-4.1 times ...
[59]
PRISMA 2020 explanation and elaboration: updated guidance and ...
Mar 29, 2021 · The PRISMA 2020 statement has been designed primarily for systematic reviews of studies that evaluate the effects of health interventions.
[60]
[PDF] to effect estimation: - statistical reform in psychology, medicine and ...
in the psychology department I work in. The time has come—is perhaps well over- due—to provide an evidence base for statistical reform in psychology and other.
[61]
Statistics Education in Undergraduate Psychology: A Survey of UK ...
Sep 14, 2022 · We found that only 19% of universities had publicly available curricula describing the statistical content taught in their undergraduate psychology programme.
[62]
Reporting guidelines | EQUATOR Network
Search for reporting guidelines. Browse for reporting guidelines by selecting one or more of these drop-downs.Observational Studies: STROBE · The AGREE Reporting Checklist · Study protocolsMissing: shift | Show results with:shift<|separator|>