Fact-checked by Grok 2 weeks ago

Randomized experiment

A randomized experiment is a scientific study design in which participants or subjects are randomly assigned to different groups—typically an experimental group receiving a treatment or intervention and a control group receiving no treatment or a placebo—to assess the causal effect of the intervention on a specified outcome.^[1] This random assignment process ensures that, on average, the groups are balanced in terms of both observed and unobserved characteristics, thereby minimizing selection bias and confounding variables that could otherwise distort the results.^[2] The foundations of randomized experiments were laid in the early 20th century by British statistician Ronald A. Fisher, who advocated randomization as essential for valid statistical inference while working on agricultural trials at the Rothamsted Experimental Station.^[3] Fisher's seminal works, including Statistical Methods for Research Workers (1925) and The Design of Experiments (1935), formalized randomization as a tool to control experimental error and enable the analysis of variance, revolutionizing experimental design in fields like agriculture, biology, and medicine.^[4] Earlier precursors existed, such as James Lind's 1747 controlled trial on scurvy treatments aboard a British ship, but these lacked true randomization and systematic statistical analysis.^[5] Randomized experiments, often implemented as randomized controlled trials (RCTs) in clinical settings, are considered the gold standard for establishing causality because they allow researchers to isolate the effect of the intervention under controlled conditions.^[6] Their reliability stems from the probabilistic nature of randomization, which supports both Fisher's exact null hypothesis testing and Jerzy Neyman's potential outcomes framework for estimating average treatment effects.^[7] Widely applied across disciplines—from evaluating medical therapies and educational programs to testing policy interventions—these experiments underpin evidence-based decision-making, though they require careful consideration of ethical issues, sample size, and generalizability to real-world settings.^[8]

Fundamentals

Definition and Purpose

A randomized experiment is a research design in which subjects or units are randomly assigned to either a treatment group, which receives an intervention, or a control group, which does not, to minimize selection bias and enable valid causal inference about the intervention's effects.^[9] This random assignment process ensures that the groups are comparable on average, except for the intervention, thereby isolating its impact on outcomes.^[2] The primary purpose of randomized experiments is to test hypotheses regarding the causal effects of interventions by establishing that observed differences in outcomes between groups are attributable to the treatment rather than confounding factors.^[10] Unlike observational studies, where associations may be confounded by self-selection or other variables, randomization provides a rigorous method for distinguishing causation from correlation, supporting reliable statistical estimates of treatment efficacy.^[2] Key components include the treatment, typically an intervention such as a drug or policy change; the control, often involving no intervention, a placebo, or standard care; the randomization procedure, which uses methods like random number generation to assign participants; and outcome measurement, which assesses the effects on relevant variables post-assignment.^[9] For instance, in medicine, a simple two-arm randomized trial might compare a new drug against a placebo to evaluate its therapeutic benefits, while in marketing, A/B testing randomly assigns website visitors to different versions of a page to determine which drives higher engagement.^[11]^[12]

Key Principles

Random assignment is the core mechanism in randomized experiments, involving the allocation of experimental units—such as participants, plots, or samples—to treatment and control groups through probabilistic methods like coin flips, dice rolls, or computer-generated random number sequences.^[13] This process ensures that the assignment to groups is independent of any pre-existing characteristics or covariates of the units, thereby preventing systematic differences between groups that could arise from deliberate selection.^[3] By making all possible assignments equally likely, randomization establishes a foundation for valid causal comparisons.^[14] A key theoretical consequence of randomization is exchangeability, which arises because the randomization procedure treats all units symmetrically, rendering the joint distribution of potential outcomes invariant to permutations of the assignment vector.^[15] Under this principle, the observed outcomes under any specific randomization are representative of the full randomization distribution, where each possible assignment has equal probability, leading to identical marginal distributions of potential outcomes across treatment arms.^[16] This symmetry underpins the ability to draw inferences about treatment effects without assuming specific models for the data-generating process. Randomization promotes balance across groups with respect to both known and unknown confounding factors, thereby yielding unbiased estimates of treatment effects on average over repeated randomizations.^[9] While perfect balance is not guaranteed in any single experiment, the probabilistic nature of assignment minimizes selection bias by distributing covariates evenly in expectation, reducing the influence of extraneous variables on group differences.^[17] This unbiasedness is a direct result of the independence introduced by random allocation, ensuring that systematic errors from non-random selection are eliminated.^[3] In analysis, randomization supports the intention-to-treat (ITT) approach, which evaluates outcomes based on initial group assignment regardless of adherence or protocol deviations, preserving the original randomization and thus maintaining group comparability.^[18] In contrast, per-protocol (PP) analysis restricts the sample to units that fully comply with the assigned treatment, potentially reintroducing bias by violating the initial random allocation.^[19] The ITT principle aligns with randomization's design by estimating the pragmatic effect of offering the treatment, while PP focuses on the effect among compliers but risks undermining the experiment's validity.^[20]

Historical Development

Early Concepts

The foundations of randomization in experimental design emerged in the 18th century amid a broader shift toward inductive and probabilistic approaches in scientific inquiry. Francis Bacon's Novum Organum (1620) laid early groundwork by advocating an inductive method that prioritized systematic observation and experimentation to derive general principles from particulars, emphasizing the need to eliminate biases through methodical procedures like tables of presence, absence, and degrees of phenomena.^[21] This empirical framework influenced later experimental philosophy by promoting cooperative, bias-free investigations, though it predated explicit probabilistic elements. A pivotal advancement came with Jacob Bernoulli's Ars Conjectandi (1713), which formalized probability theory and introduced the law of large numbers, demonstrating that repeated trials—such as coin tosses—would converge observed outcomes to their true probabilities, providing a mathematical basis for interpreting chance in empirical trials.^[22] Bernoulli's work, developed between 1684 and 1689, extended probability from games of chance to real-world applications like judicial and political decisions, establishing a precursor to randomization by quantifying certainty through observational data in controlled repetitions.^[23] In the 19th century, these ideas evolved through biometric applications and growing recognition of the need for controls, particularly in agricultural and biological experiments. Karl Pearson, building on Francis Galton's legacy, advanced biometrics as a statistical approach to heredity and variation, founding the journal Biometrika in 1901 to apply quantitative methods to biological problems, which highlighted the importance of controlled comparisons to isolate effects amid natural variability.^[24] In agricultural contexts, 19th-century experiments focused on yield improvements through mineral and organic additions, with early designs incorporating principles of balanced comparisons that foreshadowed randomization to address soil heterogeneity and environmental confounders, though systematic random allocation remained undeveloped.^[25] These efforts underscored the limitations of ad hoc controls, setting the stage for more rigorous designs. Philosophical debates during this period contrasted deterministic worldviews—rooted in 18th-century mechanics—with emerging probabilistic interpretations, emphasizing chance to mitigate confounding factors in natural sciences. Thinkers like John Venn and Robert Ellis developed frequency theories of probability in the late 19th century, framing randomness as a tool for inferring reliable patterns from trials rather than assuming universal causation, as seen in early psychological experiments on telepathy that used probabilistic scoring to counter deterministic biases.^[26] This shift validated randomization's role in establishing circumstantial causality through repeated, chance-based assignments, challenging strict determinism by allowing for evidential inference in uncertain domains. Early non-randomized attempts further illustrated the risks of bias, as exemplified by Francis Galton's 1886 studies on heritability using parental and offspring heights from 205 families (928 adult children), which revealed "regression toward mediocrity" where extreme parental traits did not fully transmit to children, due to selection biases in non-random samples that skewed toward extremes and ignored population variability.^[27] Such observational work exposed how unrandomized designs amplified confounding, prompting calls for chance-based allocation to ensure representative comparisons and reduce systematic errors in experimental outcomes.

Modern Milestones

In the early 20th century, Ronald Fisher advanced the formalization of randomized experiments through his work at Rothamsted Experimental Station, where he developed randomized block designs to control for soil variability in agricultural trials.^[28] These designs involved dividing experimental plots into homogeneous blocks and randomly assigning treatments within each to minimize bias and enhance statistical precision.^[29] Fisher's seminal book, The Design of Experiments (1935), codified these principles, emphasizing randomization as essential for valid inference in experimental agriculture.^[28] The application of randomized controlled trials (RCTs) expanded significantly in medicine during the mid-20th century, with the 1948 British Medical Research Council (MRC) trial of streptomycin for pulmonary tuberculosis marking the first large-scale RCT.^[30] This double-blind study randomly allocated 107 patients to streptomycin treatment and 107 to bed rest alone, demonstrating a dramatic reduction in mortality (from 29% in controls to 7% in the treatment group during the first six months) and establishing RCTs as the gold standard for evaluating drug efficacy.^[31] Building on this, the 1950s saw RCTs applied to vaccine development, notably in the 1954 Salk polio vaccine field trial involving over 1.8 million children randomly assigned to vaccine or placebo groups across the United States.^[32] The trial confirmed the vaccine's 60-90% efficacy against paralytic polio, accelerating its widespread adoption and influencing global public health strategies.^[33] In the technology sector, randomized experiments gained prominence in the 1990s with the rise of online A/B testing at companies like Microsoft and Google, enabling rapid evaluation of user interface changes on vast digital platforms.^[34] Microsoft's experimentation platform, formalized in the early 2000s, conducted thousands of controlled tests on products like Bing search, revealing subtle impacts such as a 1-2% lift in user engagement from minor design tweaks.^[35] Similarly, Google's 2000 "50 shades of blue" A/B test optimized ad button colors, increasing click-through rates by 0.2% and generating millions in additional revenue, underscoring the scalability of randomization in tech product development.^[36] Econometric applications of field experiments proliferated in the 2000s, particularly through the work of economists Abhijit Banerjee and Esther Duflo, who integrated randomization into development economics to test antipoverty interventions.^[37] Their randomized evaluations, such as a 2000s study in India on remedial tutoring for low-performing students, showed learning gains equivalent to two years of regular schooling, informing scalable education policies.^[38] Earlier policy experiments in the 1960s-1970s, like the U.S. negative income tax (NIT) trials in cities such as Seattle and Denver, randomly assigned families to guaranteed income supplements, revealing modest work disincentives (e.g., 5-10% reduction in hours for secondary earners) but positive effects on health and schooling.^[39] These findings influenced welfare debates, though NIT was not adopted.^[40] In the 21st century, the Abdul Latif Jameel Poverty Action Lab (J-PAL), founded in 2003 by Banerjee, Duflo, and others at MIT, has driven the expansion of randomized experiments in development economics, conducting more than 2,300 evaluations across over 90 countries (as of 2025) to assess interventions like microfinance and deworming programs.^[41] J-PAL's work, including a 2000s Kenyan study demonstrating deworming's 25% increase in school attendance, has shaped global aid priorities by providing rigorous evidence on cost-effective poverty alleviation.^[42] This institutionalization has elevated randomized experiments as a cornerstone of evidence-based policymaking in low-income settings.^[43]

Design and Implementation

Types of Designs

Randomized experiments encompass a variety of designs tailored to different research contexts, structures, and scales, allowing researchers to balance simplicity, precision, and efficiency in testing hypotheses. These designs vary in how randomization is applied—whether to individuals, groups, or adaptively over time—and are chosen based on factors like population heterogeneity, logistical constraints, and the need to detect interactions or minimize bias. Core variants include completely randomized designs for straightforward assignments, blocked designs for controlling variability, factorial setups for multifaceted interventions, cluster approaches for group-level effects, and sequential methods for dynamic online environments.^[44]^[45] The completely randomized design (CRD) represents the simplest form of randomization, where experimental units are assigned to treatment groups entirely at random without any predefined structure or stratification. This approach assumes relative homogeneity among units, making it ideal for settings where external variability is minimal and the primary goal is to ensure unbiased allocation. Developed as a foundational principle by Ronald A. Fisher in the 1920s and 1930s during his work at the Rothamsted Experimental Station, the CRD facilitates straightforward statistical analysis via methods like analysis of variance (ANOVA), as all units are treated equivalently except for the random assignment. It is particularly suitable for laboratory or controlled agricultural trials with uniform conditions, such as testing fertilizer effects on identical soil plots.^[46]^[44]^[47] To address potential imbalances from unaccounted covariates, the randomized block design (RBD), also known as the randomized complete block design, stratifies experimental units into homogeneous blocks based on known sources of variation before randomizing treatments within each block. This stratification enhances precision by reducing error variance, allowing smaller sample sizes to achieve the same power as a CRD while controlling for factors like soil fertility gradients in field trials or patient age in clinical studies. Fisher introduced blocking as one of his three core principles of experimental design in the 1920s, emphasizing its role in agricultural experiments where environmental heterogeneity could otherwise confound results; for instance, in a trial blocking by age might group elderly and young participants separately to isolate treatment effects more accurately. The design is widely applied in preclinical research and agronomy, where blocks ensure that treatment comparisons occur within similar subgroups.^[45]^[48]^[49] Factorial designs extend randomization to multiple factors simultaneously, enabling the evaluation of main effects from each factor as well as their interactions within a single experiment, thereby increasing efficiency over one-factor-at-a-time approaches. In a full factorial setup, all possible combinations of factor levels are tested—such as a 2×2 design examining two drug dosages and two therapy types to assess both individual and combined impacts on recovery rates. Fisher pioneered this method in the 1920s, arguing in his 1926 paper that complex designs like factorials were more informative and resource-efficient for discovering interactions, a concept formalized in his 1935 book The Design of Experiments. These designs are common in clinical and industrial settings, such as testing packaging materials and storage conditions in food science, where interactions might reveal synergistic effects not visible in simpler trials.^[50]^[51]^[52] Cluster randomization, or cluster randomized trials (CRTs), randomizes intact groups (clusters) such as schools, communities, or healthcare facilities to treatment arms rather than individuals, which is essential when interventions operate at the group level or to prevent contamination between participants. This design is prevalent in public health and epidemiology, where individual randomization might be impractical or unethical; for example, assigning entire villages to a vaccination program versus control to evaluate community-wide disease reduction. The approach accounts for intra-cluster correlation, which inflates variance and requires larger samples than individual randomization, but it aligns with real-world implementation of policies like sanitation improvements in low-resource settings. Modern CRTs trace roots to 19th-century agricultural and medical studies but gained prominence in the late 20th century for evaluating population-level interventions, as outlined in methodological guidelines from health authorities.^[53]^[54] Online or sequential designs adapt randomization dynamically as data accrue, often using multi-armed bandit algorithms to balance exploration of options with exploitation of promising ones, contrasting with fixed-allocation traditional experiments. In web-based A/B testing, for instance, traffic is sequentially assigned to variants (e.g., webpage layouts) based on interim performance, allowing early termination of inferior arms and faster optimization of metrics like user engagement. This approach minimizes opportunity costs in high-volume online services, where static randomization might delay benefits; a seminal framework from Google researchers describes bandits as sequential experiments that, using methods like Thompson sampling, can reduce lost conversions by adaptively reallocating users—achieving convergence in simulations far quicker than fixed A/B tests requiring millions of observations. Such designs are standard in digital platforms for continuous experimentation, like e-commerce recommendation systems.^[55]^[56]

Practical Steps

The planning phase of a randomized experiment begins with clearly defining the research hypothesis, which articulates the expected causal relationship between the intervention and outcome, ensuring the study addresses a specific, testable question.^[57] Researchers then select the sample population by establishing inclusion and exclusion criteria to ensure representativeness and feasibility, drawing from the target population while considering logistical constraints such as accessibility and recruitment timelines.^[58] Sample size determination follows a power analysis, aiming to achieve sufficient statistical power—typically 80% or higher—to detect a meaningful effect size with a low type II error rate, factoring in expected variability and dropout rates without delving into precise computations at this stage.^[59] Finally, the design type is chosen based on the hypothesis and context, such as parallel-group or cluster randomization, to balance internal validity with practical implementation.^[58] Executing randomization involves generating an allocation sequence to assign participants to treatment or control groups impartially, minimizing selection bias and ensuring comparable groups.^[57] This can be accomplished using statistical software like the R package randomizr, which automates the creation of random assignments for various designs, including simple, blocked, or stratified methods, and provides tools for verifying balance across covariates.^[60] Allocation concealment is critical to prevent foreknowledge of assignments from influencing enrollment or group composition, often achieved through centralized computer systems, sealed envelopes, or pharmacy-controlled dispensing to maintain blinding for investigators and participants where feasible.^[59] During intervention and data collection, treatments are administered according to the protocol, with the experimental group receiving the intervention and the control group a standard or placebo alternative to isolate effects.^[57] Outcomes are measured using reliable, validated instruments, ideally under blinded conditions to reduce observer bias, such as having independent assessors evaluate results without knowledge of group assignments.^[58] Compliance issues, including non-adherence or crossover, are monitored and documented through tracking mechanisms like pill counts or self-reports, with strategies such as reminders or incentives employed to maximize protocol adherence without compromising randomization integrity.^[59] Ethical considerations are paramount, particularly in experiments involving humans, requiring institutional review board (IRB) approval to evaluate risks, benefits, and scientific merit prior to initiation.^[61] Informed consent must be obtained from all participants, providing comprehensive details on study purpose, procedures, potential risks, benefits, and the right to withdraw at any time, in accordance with principles of respect for persons, beneficence, and justice outlined in foundational guidelines.^[62] Efforts to minimize harm include maintaining clinical equipoise—genuine uncertainty about treatment superiority—and ensuring vulnerable populations are protected through additional safeguards.^[58] Post-experiment activities commence with thorough data cleaning to address missing values, outliers, and inconsistencies while preserving the integrity of the original dataset.^[59] Adherence to a pre-specified analysis plan is essential to prevent p-hacking, where researchers might selectively report results favoring significance; this plan, registered prior to data collection (e.g., on platforms like ClinicalTrials.gov), details hypotheses, outcomes, and methods to ensure transparency and reproducibility.^[63]

Statistical Foundations

Role in Inference

Randomization enables valid statistical inference in experiments by making the treatment assignment independent of both observed and unobserved covariates, including potential outcomes, which ensures that any systematic differences between groups arise solely from the treatment rather than confounding factors. This independence allows researchers to draw probability-based conclusions about treatment effects without relying on strong parametric assumptions about the data-generating process. The randomization distribution encompasses all possible outcomes of a test statistic under the random assignment mechanism, assuming the null hypothesis of no treatment effect holds for every unit. This distribution provides a model-free foundation for exact hypothesis testing, as it depends only on the known assignment probabilities and the observed data, independent of any superpopulation model or sampling assumptions. For instance, in a completely randomized experiment, the randomization distribution can be enumerated exactly for small samples to compute p-values. Under randomization, estimators of the treatment effect, such as the difference in sample means between treatment and control groups, are unbiased for the average treatment effect even if underlying models for the outcomes are misspecified. This unbiasedness holds because the expected value of the estimator, taken over the randomization distribution, equals the true average effect in the finite sample of units, without requiring assumptions about homogeneity of effects or distributional forms. Neyman demonstrated that this property arises directly from the symmetry induced by random assignment, ensuring that each unit has an equal probability of receiving the treatment regardless of its potential responses. In the Neyman-Rubin framework, the potential outcomes model formalizes causal inference, where each unit has two potential responses: Y_i(1) under treatment and Y_i(0) under control. Randomization identifies the average treatment effect (ATE) as the expected difference in these outcomes across the population of units:

\tau = E[Y_i(1) - Y_i(0)]

Because randomization equates the distribution of potential outcomes in the treated and control groups, the ATE equals the difference in expected observed outcomes: E[Y \mid Z=1] - E[Y \mid Z=0], where Z denotes treatment assignment. This identification holds exactly in finite samples under the randomization distribution, providing a rigorous basis for causal claims. Rubin extended Neyman's approach by emphasizing the role of randomization in bounding the bias of estimators and enabling sensitivity analyses for nonrandomized settings. For finite samples, randomization supports exact inference through tests like Fisher's exact test, which computes the probability of the observed data (or more extreme) under the randomization distribution, contrasting with large-sample approximations that rely on asymptotic normality. This approach is particularly valuable in small experiments, where it avoids reliance on parametric models and delivers precise p-values without approximation error, as illustrated in Fisher's lady tasting tea experiment. Unlike superpopulation-based methods, randomization tests condition on the fixed set of observed units, treating inference uncertainty as stemming solely from the random assignment.

Analysis Techniques

In randomized experiments, analysis techniques focus on estimating treatment effects and assessing their statistical significance while accounting for the experimental design. Common methods include parametric tests for group comparisons, regression-based adjustments, and procedures to construct intervals and handle inferential challenges. These approaches leverage the randomization to ensure valid inferences about population parameters, such as the average treatment effect (ATE).^[14] For comparing means between groups, the Student's t-test is widely used in two-arm designs to evaluate differences in outcomes between treatment and control groups. It assumes normality of the outcome distribution within groups, homogeneity of variances, and independence of observations, with the test statistic computed as the difference in sample means divided by the standard error. In multi-arm designs, analysis of variance (ANOVA) extends this by partitioning total variance into between-group and within-group components, testing the null hypothesis of equal means across arms via an F-statistic; one-way ANOVA is suitable for completely randomized designs, assuming similar conditions as the t-test plus sphericity for repeated measures if applicable. Violations of normality can be addressed with transformations or non-parametric alternatives like the Wilcoxon rank-sum test, though parametric methods are preferred when assumptions hold due to their power.^[64] Regression models provide a flexible framework for analysis, particularly through linear regression for continuous outcomes, where the treatment indicator is included as a predictor to estimate the ATE as the coefficient on that indicator. Post-randomization covariate adjustment enhances precision by including baseline variables in the model, reducing residual variance without introducing bias under randomization; the adjusted estimator remains consistent for the ATE, and for generalized linear models like logistic regression in binary outcomes, coefficients represent log-odds ratios adjusted for covariates. Seminal work demonstrates that such adjustments, including interactions if needed, improve efficiency over unadjusted differences in means, especially with prognostic covariates strongly correlated with the outcome. Regulatory guidance endorses this approach for increasing power in clinical trials while maintaining validity.^[65]^[66] Confidence intervals (CIs) and p-values quantify uncertainty and significance for the ATE, typically constructed using the standard error from the difference-in-means or regression estimator under large-sample normality approximations; for a 95% CI, it spans the point estimate ± 1.96 times the standard error, providing a range of plausible ATE values. P-values assess evidence against the null of no treatment effect, derived from t- or z-distributions in t-tests, ANOVA, or regression. In experiments with multiple endpoints or comparisons, adjustments like the Bonferroni correction control the family-wise error rate by dividing the significance level α (e.g., 0.05) by the number of tests, yielding adjusted p-values or CIs to mitigate inflation of Type I error.^[14]^[67] Handling complexities in analysis requires careful guidelines. For subgroup analysis, interactions between treatment and subgroup indicators should be pre-specified and tested formally to detect heterogeneity, with exploratory post-hoc analyses interpreted cautiously to avoid false positives due to reduced power; regulatory standards recommend limiting confirmatory subgroups to those with strong prior evidence, reporting both overall and subgroup effects with interaction p-values. Missing data, common in experiments due to attrition, can bias estimates if not addressed; multiple imputation under the missing-at-random assumption creates m (typically 5–20) datasets by drawing from predictive distributions based on observed data and covariates, analyzes each, and pools results using Rubin's rules for variances and p-values, preserving power over complete-case analysis. Power analysis guides sample size planning, with the formula for equal-sized two-arm trials estimating the per-group size n as

n = \frac{(Z_{\alpha/2} + Z_{\beta})^2 \cdot 2\sigma^2}{\delta^2},

where Z_{\alpha/2} and Z_{\beta} are critical values for significance level α and power 1–β, σ is the outcome standard deviation, and δ is the minimal detectable effect; this basic formula ensures adequate power (e.g., 80%), with separate adjustment (e.g., inflating n by 1/(1 - dropout rate)) typically applied to account for dropout.^[68]

Benefits and Evidence

Empirical Validation

One of the earliest empirical demonstrations of the superiority of randomization in experimental design came from agricultural research at Rothamsted Experimental Station in the 1920s, led by Ronald A. Fisher. In comparisons between systematic plot arrangements and randomized designs, the latter produced more reliable yield estimates by mitigating soil heterogeneity and other environmental biases that systematically distorted results in non-randomized setups. For instance, analyses of crop yield data from these experiments showed that randomized blocks reduced error variance and improved the precision of treatment effect estimates, leading to better identification of effective fertilizers and practices.^[69] In medicine, reanalyses from the 1980s and onward highlighted how randomized controlled trials (RCTs) often overturned conclusions from studies using historical or observational controls, which tended to overestimate treatment benefits due to selection and temporal biases. A seminal review by Sacks et al. examined 50 RCTs and 56 studies with historical controls across six therapies, finding that 79% of historical control studies reported the therapy as superior, compared to only 20% of RCTs, demonstrating a substantial bias in non-randomized approaches.^[70] This pattern was starkly illustrated in the case of hormone replacement therapy (HRT), where numerous observational studies in the 1990s suggested cardiovascular benefits, but the Women's Health Initiative RCT in 2002 revealed increased risks of heart disease and stroke, correcting the earlier misleading findings.^[71] In social sciences, particularly education, meta-analyses from the 2000s and 2010s have quantified how randomized evaluations reduce biases inherent in quasi-experimental designs, often correcting effect estimates by 20-50%. For example, Glazerman et al.'s review of 12 replication studies comparing nonexperimental methods to RCTs in job training and welfare programs—a proxy for social interventions including educational components—found that nonexperimental approaches understated impacts in eight cases and overstated in four, with relative biases frequently reaching 20-50% of the true experimental effect due to unobserved confounders like participant motivation. Similar patterns emerged in education-specific meta-analyses, where quasi-experimental estimates of interventions like class size reductions were inflated by selection bias until validated by RCTs. In technology and online platforms, empirical work at Microsoft in the late 2000s underscored randomization's role in minimizing biases in user behavior testing. Kohavi et al. analyzed numerous A/B tests on web features, showing that without randomization, factors like user demographics or network effects could introduce substantial biases in engagement metrics, whereas randomized allocation ensured unbiased attribution of changes to interventions, as evidenced by consistent replication of results across thousands of daily users. This approach quantified bias reductions, enabling reliable decisions on product changes that improved user satisfaction metrics by up to 5-10% in validated experiments.

Comparative Advantages

Randomized experiments offer distinct advantages over observational studies by employing randomization to eliminate confounding factors, such as selection bias, which can distort associations in non-experimental data.^[9] This process ensures that treatment assignment is independent of potential confounders, both known and unknown, leading to unbiased estimates of causal effects.^[72] In contrast to quasi-experimental approaches like regression discontinuity designs, which rely on assumptions about continuity at a cutoff and may still be susceptible to local biases, randomized experiments provide stronger, more robust causal claims applicable across the entire sample.^[73] Randomized experiments excel in internal validity due to their rigorous controls, which minimize systematic errors and allow for confident attribution of outcomes to the intervention.^[74] However, they may face challenges in external validity, as the controlled settings and participant selection can limit generalizability to broader populations, whereas cohort studies, drawn from real-world settings, often capture a wider range of diversity and thus offer superior external applicability.^[75] Despite this trade-off, the high internal validity of randomized designs makes them preferable when establishing causality is paramount, even if additional steps are needed to enhance generalizability. In terms of statistical efficiency, randomized experiments reduce variance in treatment effect estimates by balancing covariates across groups, leading to improvements in precision compared to matched observational designs that require post-hoc adjustments. This efficiency stems from the absence of bias correction needs, allowing for more precise inference with the same sample size.^[76] Although randomized experiments entail higher upfront costs for design, implementation, and ethical oversight compared to observational studies, they yield lower long-term errors by providing reliable evidence that informs effective policy decisions.^[77] For instance, in policy evaluations such as welfare-to-work programs, randomized trials have delivered actionable insights that reduced ineffective spending and improved outcomes, outweighing initial investments through sustained societal benefits.^[78]

Advanced Perspectives

Causal Frameworks

Directed acyclic graphs (DAGs) provide a graphical framework for representing causal relationships in randomized experiments, where nodes denote variables such as the treatment T, the outcome Y, and potential confounders C, while directed arrows indicate the direction of causal influence from one variable to another.^[79] These graphs assume acyclicity, meaning no feedback loops, and encode assumptions about the causal structure underlying the data-generating process.^[80] In non-randomized studies, confounding occurs through backdoor paths in the DAG, such as T \leftarrow C \rightarrow Y, where a common cause C influences both the treatment assignment and the outcome, leading to biased estimates of the causal effect of T on Y.^[80] These paths allow non-causal associations to confound the direct effect, violating exchangeability between treated and untreated groups.^[81] Randomization addresses this by incorporating a random assignment node U in the DAG, with an arrow U \rightarrow T and no incoming arrows to U, ensuring U is independent of all confounders C.^[80] This structure blocks all backdoor paths from C to the T-Y association, as the independence of T from C (conditional on U) eliminates spurious correlations.^[81] Consequently, the conditional distribution P(Y \mid T) reflects the causal effect under the no unmeasured confounding assumption, which randomization enforces by design.^[79] The average causal effect is thus identifiable from observed data as the difference P(Y=1 \mid T=1) - P(Y=1 \mid T=0) (or marginal means for continuous outcomes), without needing adjustment for measured confounders, as backdoor paths are inherently closed.^[80] For extensions involving mediated effects, the front-door criterion in DAGs permits identification of the total effect even under unmeasured confounding, provided a set of mediator variables intercepts all directed paths from T to Y while satisfying conditions like no direct unblocked backdoor paths to the mediator.^[79]

Limitations and Critiques

Randomized experiments, while powerful for causal inference, face significant ethical constraints that limit their application. A primary concern is the inability to randomize participants to potentially harmful treatments, such as withholding proven therapies in favor of placebos, which violates ethical standards requiring the provision of established effective interventions unless no serious harm results. The doctrine of clinical equipoise further mandates that trials proceed only when there is genuine uncertainty in the expert community about the comparative merits of the interventions, ensuring no participant is knowingly disadvantaged. These principles, enshrined in documents like the Declaration of Helsinki, prevent experiments that could exploit vulnerable populations or expose them to undue risk. Practical limitations also undermine the feasibility and reliability of randomized experiments. Conducting them often incurs high costs, with median per-patient expenses ranging from $409 to over $6,000 depending on the trial's scale and setting, driven by recruitment, monitoring, and data collection needs that can total millions for large studies.^[82]^[83] Dropout bias poses another challenge, as differential attrition between treatment and control groups—common in up to 20-30% of trials—can distort results by making completers unrepresentative of the original sample, leading to biased estimates of treatment effects.^[84] Additionally, generalizability is compromised when experiments occur in controlled lab or clinical settings, where selected participants (often healthier or more compliant than real-world populations) yield findings that fail to translate to diverse, everyday contexts, such as community health or online user behaviors.^[85] Statistical critiques highlight vulnerabilities in the assumptions and procedures of randomized experiments, particularly at scale. In large-scale online experiments, like A/B tests run by tech companies, multiple testing across numerous variants inflates the false positive rate; without adjustments like Bonferroni correction or false discovery rate control, the family-wise error rate can exceed 50% even at a nominal 5% significance level, leading to spurious discoveries.^[86] The stable unit treatment value assumption (SUTVA), which posits no interference between experimental units, often fails in networked or social settings—such as peer effects in education trials or spillovers in marketing experiments—causing treatment effects to propagate indirectly and bias estimates of direct impacts. When randomized experiments are infeasible due to these constraints, alternatives like quasi-experiments and instrumental variables provide fallbacks, though they require stronger assumptions for causal identification. Quasi-experimental designs, such as regression discontinuity or difference-in-differences, leverage natural variation or policy changes to approximate randomization, as seen in evaluating minimum wage effects without direct manipulation.^[87] Instrumental variables use exogenous shocks uncorrelated with outcomes but predictive of treatment assignment, like lottery-based school assignments to estimate class size impacts.^[88] However, historical failures underscore risks; the 1956-1971 Willowbrook hepatitis experiments, which intentionally infected institutionalized children to test a vaccine, violated equipoise and consent principles, resulting in widespread ethical condemnation and contributing to stricter regulations like the 1974 National Research Act.^[89]

References

[1]
Randomized Experiment - an overview | ScienceDirect Topics
A randomized experiment refers to a scientific study design in which participants are randomly assigned to either an experimental group or a control group.
[2]
Why randomize? - Institution for Social and Policy Studies
Randomized field experiments allow researchers to scientifically measure the impact of an intervention on a particular outcome of interest.
[3]
R. A. Fisher and his advocacy of randomization - PubMed
The requirement of randomization in experimental design was first stated by RA Fisher, statistician and geneticist, in 1925 in his book Statistical Methods for ...
[4]
1.1 - A Quick History of the Design of Experiments (DOE) | STAT 503
... Ronald Fisher developed in the UK in the first half of the 20th century. He really laid the foundation for statistics and for design of experiments. He and ...
[5]
The history of randomized control trials: scurvy, poets and beer
Apr 18, 2018 · Over at the Rothamsted agricultural research station in the UK, Ronald Fisher is sorting through data and running randomized field experiments ...
[6]
Causal inference from experiment and observation - PMC - NIH
Results from well-conducted randomised controlled studies should ideally inform on the comparative merits of treatment choices for a health condition.Introduction · Making Causal Statements... · Use Of Inverse Probability...
[7]
The Importance of Being Causal - Harvard Data Science Review
Jul 30, 2020 · Causal inference is the study of how actions, interventions, or treatments affect outcomes of interest.
[8]
https://stats.libretexts.org/Bookshelves/Introductory_Statistics/OpenIntro_Statistics_(Diez_et_al.)./01%3A_Introduction_to_Data/1.06%3A_Experiments
[9]
An overview of randomization techniques - NIH
The basic benefits of randomization are as follows: it eliminates the selection bias, balances the groups with respect to many known and unknown confounding or ...Simple Randomization · Block Randomization · Covariate Adaptive...
[10]
Randomized Experiments - Wiley Online Library
Aug 20, 2021 · The purpose of a randomized experiment is to determine the possible causal relationship between an independent variable and a dependent variable ...
[11]
Study Design 101: Randomized Controlled Trial - Research Guides
Sep 25, 2023 · A study design that randomly assigns participants into an experimental group or a control group. As the study is conducted, the only expected difference
[12]
https://hbr.org/2017/06/a-refresher-on-ab-testing
[13]
Randomization | The Abdul Latif Jameel Poverty Action Lab
Conceptually, randomization simply means that every experimental unit has the same probability of being assigned to a given group.
[14]
[PDF] Causal Inference Chapter 2.1. Randomized Experiments: Fisher's ...
Under randomization, unconfoundedness holds by design. (without conditioning on covariates X. ▷ Causal effects are (nonparametrically) identified, ...Missing: benefits | Show results with:benefits<|control11|><|separator|>
[15]
Estimating causal effects from epidemiological data - PMC - NIH
In summary, randomisation produces exchangeability (design 1) or conditional exchangeability (design 2). In both cases, the causal effect can be calculated from ...
[16]
2 Exchangeability and experiments | Causal Inference Course
We further discuss why exchangeability is important: it allows us to link causal quantities to observable data. We discuss exchangeability in simple randomized ...
[17]
A roadmap to using randomization in clinical trials
Aug 16, 2021 · Randomization is the foundation of any clinical trial involving treatment comparison. It helps mitigate selection bias, promotes similarity ...<|separator|>
[18]
Intention-to-treat versus as-treated versus per-protocol approaches ...
Nov 14, 2023 · There are various group-defining strategies for analyzing RCT data, including the intention-to-treat (ITT), as-treated, and per-protocol (PP) approaches.
[19]
Intention to treat and per protocol analysis in clinical trials - PubMed
By using the ITT approach, investigators aim to assess the effect of assigning a drug whereas by adopting the PP analysis, researchers investigate the effect of ...
[20]
Per‐Protocol Versus Intention‐to‐Treat in Clinical Trials
May 16, 2022 · Per‐protocol analyzes data only from participants who follow the protocol, excluding the data after they become protocol deviant/nonadherent.
[21]
Francis Bacon - Stanford Encyclopedia of Philosophy
Dec 29, 2003 · ... inductive method, which implied the need for negative instances and refuting experiments. Bacon saw that confirming instances could not ...Scientific Method: The Project... · Scientific Method: Novum... · Bibliography
[22]
Ars Conjectandi | work by Bernoulli - Britannica
Jakob Bernoulli's pioneering work Ars Conjectandi (published posthumously, 1713; “The Art of Conjecturing”) contained many of his finest concepts.
[23]
[PDF] The Significance of Jacob Bernoulli's Ars Conjectandi - Glenn Shafer
More than 300 years ago, in a fertile period from 1684 to 1689, Jacob Bernoulli worked out a strategy for applying the mathematics of games of chance to the ...<|separator|>
[24]
https://www.britannica.com/biography/Karl-Pearson
[25]
History of the Statistical Design of Agricultural Experiments - jstor
R. A. Fisher popularized and made explicit the use of randomized designs in agricultural research, but there is evidence that randomized allocation was used in ...Missing: 19th | Show results with:19th
[26]
Randomisation, Causality and the Role of Reasoned Intuition
Oct 9, 2014 · Randomisation and probabilistic inference as a method for acquiring scientific knowledge goes back to the 19th century, at least. An ...
[27]
Regression to the mean (RTM) | Britannica
Sep 22, 2025 · Galton called this phenomenon regression toward mediocrity; it is now called RTM. This is a statistical, not a genetic, phenomenon. Equations ...
[28]
[PDF] Design of Experiments - Free
The design of experiments is, however, too large a subject, and of too great importance to the general body of scientific workers, for any incidental ...
[29]
Sir Ronald Fisher and the Design of Experiments - jstor
Introductory paper in the Experimentation section, September 12, 1963. SIR RONALD FISHER AND THE DESIGN OF EXPERIMENTS. F. YATES. Rothamsted Experimental ...
[30]
Medical Research Council (1948) - The James Lind Library
The UK Medical Research Council's 1948 report of a controlled trial of streptomycin for pulmonary tuberculosis was a methodological landmark.
[31]
The MRC randomized trial of streptomycin and its legacy - PMC - NIH
The initial trials involved patients with the most serious forms of the disease - miliary and meningitic (both previously almost uniformly fatal), and very ...
[32]
(PDF) The Salk Polio Vaccine Trial of 1954: risks, randomization and ...
Aug 10, 2025 · This article recounts the story of this important early clinical trial and how the social and political conditions at the time affected its planning and ...
[33]
Polio trial: an early efficient clinical trial - PubMed
The Salk Vaccine Field Trial was a randomized, placebo-controlled trial designed to test the efficacy of the Salk killed virus vaccine.Missing: 1950s | Show results with:1950s
[34]
[PDF] Online Experimentation at Microsoft - Stanford University
In the simplest controlled experiment, often referred to as an A/B test, users are randomly exposed to one of two variants: Control (A), or Treatment (B) as ...
[35]
ExP Platform – Accelerating Innovation through Trustworthy ...
Online controlled experiments, also called A/B testing, have been established as the mantra for data-driven decision making in many web-facing companies. In ...
[36]
The Surprising Power of Online Experiments
The Surprising Power of Online Experiments. Getting the most out of A/B and other controlled tests by Ron Kohavi and Stefan Thomke · From the ...
[37]
The Experimental Approach to Development Economics
Sep 1, 2009 · The Experimental Approach to Development Economics. Abhijit V. Banerjee1 and Esther Duflo1. View Affiliations Hide Affiliations. Department of ...
[38]
Handbook of Field Experiments
An Introduction to the "Handbook of Field Experiments" Abhijit Banerjee and Esther Duflo. Many (though by no means all) of the questions that economists and ...
[39]
[PDF] NBER WORKING PAPER SERIES THE NEGATIVE INCOME TAX ...
In the policy dimension, the negative income tax proposal inspired multi-million-dollar field experiments in the United States in the 1960s and 1970s to measure ...
[40]
[PDF] The negative income tax: would it discourage work?
The economists conducting the experiments expected that the results would show some negative effect on work effort; the important question was what the mag-.
[41]
Introduction to randomized evaluations | The Abdul Latif Jameel ...
Randomized evaluations can be used to measure impact in policy research: to date, J-PAL affiliated researchers have conducted more than 1,100 randomized ...Missing: century | Show results with:century
[42]
[PDF] UNDERSTANDING DEVELOPMENT AND POVERTY ALLEVIATION
Oct 14, 2019 · This section describes the cornerstones on which the modern approach to development economics is built. We start by discussing the three ...<|control11|><|separator|>
[43]
How randomised trials became big in development economics
Dec 9, 2019 · Since its creation in 2003, J-PAL has conducted 876 policy experiments in 80 countries. ... In this context, the advocates of randomised trials ...Missing: 21st | Show results with:21st
[44]
3.1 - Experiments with One Factor and Multiple Levels | STAT 503
The completely randomized design means there is no structure among the experimental units. There are 25 runs which differ only in the percent cotton, and these ...
[45]
A Discussion of Statistical Methods for Design and Analysis of ...
Blocking, the third of Fisher's fundamental design principles, is used when it is recognized before the beginning of an experiment that certain groups of ...
[46]
[PDF] DA Brief Introduction to Design of Experiments - Johns Hopkins APL
BRIEF HISTORY. Design of experiments was invented by Ronald A. Fisher in the 1920s and 1930s at Rothamsted Experi- mental Station, an agricultural research ...
[47]
The “completely randomised” and the ... - PubMed Central
Oct 16, 2020 · The “completely randomised” and the “randomised block” are the only experimental designs suitable for widespread use in pre-clinical research.
[48]
[PDF] Randomized Complete Block Design
Treatments and replications were assigned to experimental units through the process of randomization. The result of this effort is referred to as a Completely ...Missing: definition | Show results with:definition
[49]
Chapter 11: Randomized Complete Block Design
The CRD is an appropriate experimental design when all experimental units are assumed to be similar or homogeneous (as statisticians like to say).
[50]
[PDF] Factorial experiment
Ronald Fisher argued in 1926 that "complex" designs (such as factorial designs) were more efficient than studying one factor at a time.[2] Fisher wrote,. "No ...
[51]
[PDF] The design of experiments
By. Sir Ronald A. Fisher, Sc.D., F.R.S.. Honorary Research Fellow, Division of Mathematical Statistics,. C.S.I.R.O., University of Adelaide; Foreign ...
[52]
[PDF] The Design of Experiments By Sir Ronald A. Fisher.djvu
An Exceptional Design. 36. Practical Exercises . VI. THE FACTORIAL DESIGN IN. EXPERIMENTATION. 37. The Single Factor. " ·. 38. A Simple Factorial Scheme. 39.
[53]
[PDF] Cluster Randomized Trials - Effective Health Care Program
Background: Cluster randomized trials (CRTs) offer unique advan- tages over standard randomized controlled clinical trials (RCTs) and.
[54]
A brief history of the cluster randomised trial design - PMC - NIH
This has been defined as a comparative study in which the units randomised are pre-existing (natural or self-selected) groups whose members have an identifiable ...
[55]
[PDF] Multi-armed bandit experiments in the online service economy
Jun 10, 2014 · Multi-armed bandits are a type of sequential experiment that is naturally aligned with the economics of the service industry. This article is a ...
[56]
Multi-armed Bandit Experiments - Google Analytics Blog
Jan 23, 2013 · The name "multi-armed bandit" describes a hypothetical experiment where you face several slot machines ("one-armed bandits") with potentially ...Background · Examples · A Simple A/b Test
[57]
Guide to Experimental Design | Overview, 5 steps & Examples
Dec 3, 2019 · Step 1: Define your variables · Step 2: Write your hypothesis · Step 3: Design your experimental treatments · Step 4: Assign your subjects to ...
[58]
How to Conduct a Randomized Controlled Trial - PMC
In this design, each study participant is randomly assigned to receive both study interventions in a predetermined sequence over a specified period. The time in ...
[59]
How to design a randomized clinical trial: tips and tricks for conduct ...
In particular have a realistic timeline, define a clear objective and precise endpoints, balance the study with a correct randomization and focus on the right ...
[60]
Design and Analysis of Experiments with randomizr - CRAN
randomizr is a small package for r that simplifies the design and analysis of randomized experiments. In particular, it makes the random assignment procedure ...
[61]
Informed Consent FAQs - HHS.gov
This requirement is founded on the principle of respect for persons, one of the three ethical principles governing human subjects research described in the ...
[62]
Read the Belmont Report | HHS.gov
Jul 15, 2025 · The Belmont Report outlines ethical principles for research involving human subjects, summarizing basic principles and guidelines to resolve ...Missing: randomized | Show results with:randomized
[63]
How to design a pre-specified statistical analysis approach to limit p ...
In this article, we describe a five-point framework (the Pre-SPEC framework) for designing a pre-specified analysis approach that does not allow p-hacking.
[64]
Application of Student's t-test, Analysis of Variance, and Covariance
The Student's t test is used to compare the means between two groups, whereas ANOVA is used to compare the means among three or more groups.
[65]
[PDF] Adjusting for Covariates in Randomized Clinical Trials for Drugs and ...
When adjusting for covariates based on fitting nonlinear regression models, such as logistic regression models in studies with binary outcomes, there are ...
[66]
[PDF] Agnostic notes on regression adjustments to experimental data - arXiv
Freedman argued regression adjustment can worsen precision, but this paper shows that in large samples, these problems are minor or easily fixed.
[67]
Methods to adjust for multiple comparisons in the analysis and ...
Jun 21, 2019 · If the Bonferroni method was used, the p-values could have been adjusted to 0.020, 0.004 and compared to the significance level α of 0.05.
[68]
Statistical Considerations for Subgroup Analyses - PMC - NIH
This article reviews key statistical concepts associated with planning, conducting, and interpreting subgroup analyses in RCTs.Heterogeneity Of Treatment... · Figure 1 · Confirmatory Versus...Missing: imputation formula<|control11|><|separator|>
[69]
Multiple imputation for missing data in epidemiological and clinical ...
Jun 29, 2009 · In this article, we review the reasons why missing data may lead to bias and loss of information in epidemiological and clinical research.
[70]
[PDF] E 9 Statistical Principles for Clinical Trials Step 5
a loss of power in the analysis; this should be accounted for in the sample size calculation. 2.3 Design Techniques to Avoid Bias. The most important design ...
[71]
https://jamanetwork.com/journals/jama/fullarticle/195120
[72]
8.1 - Randomization | STAT 509
Randomization is effective in reducing bias because it guarantees that treatment assignment will not be based on the patient's prognostic factors.
[73]
On making causal claims: A review and recommendations
The gold standard: the randomized field experiment. This design ensures that the correlation between an outcome and a treatment is causal; more specifically, ...Missing: stronger | Show results with:stronger
[74]
Randomized controlled trials – a matter of design - PMC
The internal validity of a clinical trial is directly related to appropriate design, conduction, and reporting of the study. The two main threats to internal ...
[75]
Rethinking the pros and cons of randomized controlled trials and ...
Jan 18, 2024 · In these contexts, observational studies may provide better external validity than RCTs, which typically occur under well-controlled and, by ...
[76]
[PDF] Batch Adaptive Designs to Improve Efficiency in Social Science ...
Dec 7, 2023 · Finally, through simulations and a literature review, we show that researchers in the political science could gain up to 15–30% improve- ments ...
[77]
Matching methods for causal inference: A review and a look forward
Analytic expressions for the bias and variance reduction possible for these situations are given in Rubin and Thomas (1992b). Specifically, Rubin and Thomas ...
[78]
A Comparison of Observational Studies and Randomized ...
Jun 22, 2000 · Observational studies have several advantages over randomized, controlled trials, including lower cost, greater timeliness, and a broader range of patients.
[79]
[PDF] CAUSAL DIAGRAMS FOR EMPIRICAL RESEARCH
In this section, we brie y review the properties of DAGs as carriers of conditional independence information [Pearl 1988]. Readers familiar with this aspect of ...
[80]
[PDF] Causal Inference: What If - HSPH Content
Causal Inference is an admittedly pretentious title for a book. A complex scientific task, causal inference relies on triangulating evidence ...
[81]
[PDF] PSC - Observational Studies and Confounding - Matthew Blackwell
Remember in the DAGs, randomization implies no arrows pointing into the treatment or we know exactly which arrows because we have done a block-randomized ...
[82]
Systematic review on costs and resource use of randomized clinical ...
The median costs per recruited patient were USD 409 (range: USD 41–6,990). Overall costs of an RCT, as provided in 16 articles, ranged from USD 43–103,254 per ...Review · Abstract · Introduction<|control11|><|separator|>
[83]
The Costs of Conducting Clinical Research - ASCO Publications
On average, excluding overhead expenses, it cost slightly more than $6,094 (range, $2,098 to $19,285) per enrolled subject for an industry-sponsored trial, ...
[84]
Generalizability of Findings from Randomized Controlled Trials - NIH
There is growing concern that the results from randomized controlled trials (RCTs) may not generalize to real world settings (1–6). Perhaps due to this, many ...
[85]
Best (but oft-forgotten) practices: the multiple problems of multiplicity ...
This article first discusses how multiple tests lead to an inflation ... Testing for baseline differences in a randomized controlled trial (RCT).2The ...
[86]
Chapter 26 Quasi-Experimental Methods | A Guide on Data Analysis
Unlike randomized experiments, quasi-experiments lack formal statistical proof of causality. Instead, researchers must build a plausible argument supported by ...
[87]
A Gentle Introduction to Instrumental Variables - ScienceDirect
IV methods in nonexperimental settings mimic a randomized experiment by using a source of “as good as” random variation in treatment instead. The main challenge ...
[88]
Hepatitis Studies at the Willowbrook State School for Children
Dec 8, 2020 · From 1956 through 1971, residents at the Willowbrook State School for Children with Mental Retardation were infected with live hepatitis in order to develop a ...Missing: randomized | Show results with:randomized