Fact-checked by Grok 2 weeks ago

Mutual information

Mutual information is a fundamental concept in information theory that quantifies the amount of information one random variable contains about another, serving as a measure of their statistical dependence.^[1] Formally defined for two random variables X and Y as I(X; Y) = H(X) - H(X \mid Y), where H(X) denotes the entropy of X and H(X \mid Y) the conditional entropy of X given Y, it represents the reduction in uncertainty about X upon knowing Y.^[1] Introduced by Claude E. Shannon in his seminal 1948 paper "A Mathematical Theory of Communication," mutual information provides a rigorous foundation for analyzing communication channels and data transmission efficiency.^[1] This measure is symmetric, such that I(X; Y) = I(Y; X) = H(X) + H(Y) - H(X, Y), where H(X, Y) is the joint entropy, and it is always non-negative, achieving zero if and only if X and Y are statistically independent.^[2] Mutual information generalizes notions of correlation beyond linear relationships, making it particularly valuable in scenarios involving nonlinear or complex dependencies.^[3] Expressed in units of bits (or nats in natural logarithm base), it enables precise quantification of shared information content between variables.^[4] Beyond its origins in communication theory, mutual information finds broad applications across disciplines. In statistics, it detects and evaluates dependencies between variables, offering a nonparametric alternative to traditional correlation metrics.^[2] In machine learning, it supports feature selection by identifying relevant predictors that maximize information about the target variable while minimizing redundancy.^[3] Fields like neuroscience employ it to assess neural coding efficiency and information flow in brain networks, while in genetics, it helps uncover associations in high-dimensional genomic data.^[4] Despite its power, estimating mutual information from finite samples poses challenges due to its sensitivity to data distribution and the curse of dimensionality, spurring ongoing research into scalable approximation methods.^[3]

Definition

Discrete random variables

Mutual information between two discrete random variables X and Y is a measure of the amount of information that one variable contains about the other, originally introduced by Shannon in the context of communication over noisy channels.^[5] Formally, it is defined as the expected value under the joint distribution P_{XY} of the pointwise mutual information, which is the logarithm of the ratio of the joint probability to the product of the marginal probabilities:

I(X;Y) = \mathbb{E}_{P_{XY}} \left[ \log \frac{dP_{XY}}{dP_X dP_Y} \right].

For discrete random variables with probability mass functions p(x,y), p(x), and p(y), this expectation becomes a double summation over their supports:

I(X;Y) = \sum_{x,y} p(x,y) \log \frac{p(x,y)}{p(x) p(y)}.

^[6]^[7] This summation formula follows directly from the definition of entropy and conditional entropy in information theory. Specifically, mutual information equals the marginal entropy minus the conditional entropy: I(X;Y) = H(X) - H(X|Y), where H(X) = -\sum_x p(x) \log p(x) is the entropy of X and H(X|Y) = \sum_y p(y) H(X|Y=y) = -\sum_{x,y} p(x,y) \log p(x|y) is the conditional entropy.^[5] Substituting and simplifying yields the summation form, as the terms involving only marginals cancel out, leaving the log ratio weighted by the joint probabilities.^[8] To illustrate computation, consider two binary random variables X and Y related through a binary symmetric channel with crossover probability \epsilon = 0.1, where X \sim \text{Bernoulli}(0.5) and Y = X \oplus Z with Z \sim \text{Bernoulli}(0.1) independent of X. The joint probabilities are p(0,0) = p(1,1) = 0.45 and p(0,1) = p(1,0) = 0.05, with marginals p(x) = p(y) = 0.5. The mutual information simplifies to I(X;Y) = H(Y) - H(Y|X) = 1 - h_2(0.1), where h_2(p) = -p \log_2 p - (1-p) \log_2 (1-p) is the binary entropy function, yielding I(X;Y) \approx 0.531 bits.^[9] The logarithm in the definition is typically taken base 2, measuring mutual information in bits, or natural logarithm for nats; the base scales the numerical value but preserves key properties like symmetry.^[7] This connection to entropy highlights mutual information as the reduction in uncertainty about X provided by observing Y.^[6]

Continuous random variables

For continuous random variables X and Y defined on a common probability space with joint probability density function f_{X,Y}(x,y) and marginal density functions f_X(x) and f_Y(y) with respect to Lebesgue measure, mutual information extends the discrete case by replacing sums with integrals over the densities.^[8] This formulation assumes the joint distribution is absolutely continuous with respect to the product of the marginals, allowing the use of Radon-Nikodym derivatives to define the densities rigorously.^[10] The mutual information I(X;Y) is then given by

I(X;Y) = \iint f_{X,Y}(x,y) \log \frac{f_{X,Y}(x,y)}{f_X(x) f_Y(y)} \, dx \, dy,

where the logarithm is typically base 2 for bits or natural for nats, and the integral is taken over the support of the densities.^[8] Unlike the discrete case, this expression can yield infinite values if the densities lead to divergences, such as when X = Y almost surely, reflecting perfect dependence in continuous spaces where differential entropy is unbounded below.^[11] The definition relies on differential entropy as a prerequisite, which generalizes discrete entropy to continuous variables but differs fundamentally due to the infinite divisibility of the real line.^[8] For a continuous random variable X with density f_X(x), the differential entropy is

H(X) = -\int f_X(x) \log f_X(x) \, dx.

This quantity can be negative, unlike Shannon entropy, because it measures uncertainty relative to Lebesgue measure rather than a finite partition.^[8] Mutual information for continuous variables can equivalently be expressed as I(X;Y) = H(X) + H(Y) - H(X,Y), where H(X,Y) is the joint differential entropy, preserving non-negativity despite the potential negativity of individual terms.^[8] When the joint distribution is singular with respect to the product of the marginals—meaning it concentrates on a lower-dimensional manifold without a density in the full space—the standard integral form does not apply directly, and mutual information may be infinite to capture complete dependence.^[12] In such cases, the rigorous definition invokes the Radon-Nikodym derivative of the joint measure with respect to the product measure, ensuring the expression I(X;Y) = \int \log \frac{dP_{X,Y}}{d(P_X \times P_Y)} \, dP_{X,Y} holds where the derivative exists, with infinity otherwise.^[10] This handles singular continuous distributions, like those uniform on a curve, by embedding them in the broader measure-theoretic framework without assuming full-dimensional densities.^[12] A representative example is the bivariate Gaussian distribution, where X and Y have zero mean, unit variance, and correlation coefficient \rho \in (-1,1). The mutual information admits a closed-form expression I(X;Y) = -\frac{1}{2} \log(1 - \rho^2), measured in nats, which increases monotonically from 0 (at \rho = 0, independence) toward infinity as |\rho| approaches 1 (perfect linear dependence).^[13] This formula arises from computing the differential entropies: H(X) = H(Y) = \frac{1}{2} \log(2\pi e) and H(X,Y) = \log(2\pi e) + \frac{1}{2} \log(1 - \rho^2), highlighting how correlation reduces joint uncertainty beyond marginals.^[13]

General measure-theoretic formulation

In the measure-theoretic framework, mutual information between two random variables X and Y is defined with respect to the \sigma-algebras \mathcal{G} and \mathcal{H} they generate on a probability space (\Omega, \mathcal{F}, P). It quantifies the shared information as I(X; Y) = \int_{\Omega} \log \left( \frac{dP_{XY}}{d(P_X \times P_Y)} \right) dP_{XY}, where P_{XY} is the joint probability measure induced by X and Y, P_X \times P_Y is the product measure of the marginals, and the logarithm argument is the Radon-Nikodym derivative assuming P_{XY} \ll P_X \times P_Y. This integral expression is equivalent to the Kullback-Leibler divergence I(X; Y) = D_{\mathrm{KL}}(P_{XY} \| P_X \times P_Y), which measures how the joint distribution deviates from independence under the product measure.^[8] The formulation extends to arbitrary probability spaces beyond standard Euclidean settings, encompassing atomic measures (discrete components with point masses) and diffuse measures (continuous components without atoms), as long as the absolute continuity condition holds to ensure the Radon-Nikodym derivative exists. The concept originated in Claude Shannon's 1948 paper, where mutual information was introduced as a foundational quantity in information theory for analyzing discrete communication channels.^[5]

Interpretation

Relation to entropy

Mutual information between two random variables X and Y, denoted I(X; Y), measures the amount of information one variable contains about the other and is defined in terms of entropy as

I(X; Y) = H(X) - H(X \mid Y) = H(Y) - H(Y \mid X) = H(X) + H(Y) - H(X, Y),

where H(X) is the entropy of X, H(X \mid Y) is the conditional entropy of X given Y, and H(X, Y) is the joint entropy of X and Y.^[5] This formulation arises directly from the foundational concepts in information theory, where entropy quantifies uncertainty.^[5] The term H(X) - H(X \mid Y) specifically represents the reduction in uncertainty about X when Y is known. The entropy H(X) measures the average information needed to specify X, while the conditional entropy H(X \mid Y) captures the remaining uncertainty in X after observing Y. Thus, their difference I(X; Y) quantifies the information that Y provides about X, symmetrically expressed as H(Y) - H(Y \mid X). The joint form H(X) + H(Y) - H(X, Y) follows from the chain rule for entropy, H(X, Y) = H(X) + H(Y \mid X), highlighting how mutual information accounts for the overlap in the uncertainties of X and Y.^[5] A simple example illustrates this relation using two fair coin flips, X and Y, each with outcomes heads or tails equally likely, so H(X) = H(Y) = 1 bit. If X and Y are independent, then H(X \mid Y) = H(X) = 1 bit, yielding I(X; Y) = 0 bits and indicating no reduction in uncertainty. In contrast, if Y = X (perfect dependence), then H(X \mid Y) = 0 bits, so I(X; Y) = 1 bit, showing that observing Y eliminates all uncertainty about X. This entropy difference demonstrates mutual information's role in quantifying shared uncertainty.^[14] Mutual information serves as a measure of dependence or "correlation" between variables in information units (such as bits), but it differs from statistical correlation like the Pearson coefficient by capturing any form of statistical dependence, including nonlinear ones, without assuming linearity.^[15]

Information-theoretic motivation

Mutual information emerged as a cornerstone of information theory through Claude Shannon's foundational work in his 1948 paper "A Mathematical Theory of Communication," where it was defined to capture the essence of how much uncertainty about one event is resolved by observing another. Shannon developed this concept to address the challenges of reliable communication in the presence of noise, framing mutual information I(X;Y) as the precise measure of shared information between a source variable X and a received variable Y. This quantification allowed for the first time a mathematical treatment of information as a tradable commodity, independent of semantics or physical representation.^[5] In the context of noisy communication channels, mutual information provides the theoretical foundation for understanding the limits of information transmission. It represents the average amount of information about the input that is conveyed through the channel's output, serving as the key metric for determining how much reliable communication is possible without error. For instance, in a channel where noise corrupts the signal, I([X;Y](/page/X;Y)) quantifies the reduction in the sender's message uncertainty that the receiver can achieve, motivating its role as the building block for concepts like channel capacity. This perspective underscores mutual information's origin in solving practical engineering problems of the era, such as telegraphy and early telephony.^[5] To illustrate, consider a thought experiment with two fair six-sided dice representing random variables X and Y. If the dice are rolled independently, the outcome of one die offers no predictive power about the other; knowing X = 3 does not alter the probabilities for Y, resulting in I(X;Y) = 0. This zero mutual information directly corresponds to statistical independence between the variables, a property that holds symmetrically: if X and Y are independent, then I(X;Y) = 0.^[16]^[6] This equivalence between zero mutual information and independence is a hallmark of the concept, providing a rigorous test for dependence in probabilistic systems. However, in continuous random variables, while the implication remains valid, the computation involves probability densities and can encounter subtleties such as potential divergences if the joint distribution lacks absolute continuity with respect to the product measure. Nonetheless, under standard assumptions, mutual information faithfully detects the absence of informational linkage in both discrete and continuous settings.^[6]

Geometric interpretation

Mutual information admits a geometric interpretation as the Kullback-Leibler (KL) divergence between the joint probability distribution P_{X,Y} and the product of the marginal distributions P_X P_Y, quantifying the deviation of the joint distribution from the independence assumption.^[17] This formulation positions mutual information as a measure of "distance" in the space of probability distributions, where zero mutual information corresponds to the joint aligning perfectly with the independence surface.^[17] Geometrically, this KL divergence can be visualized as the area under the curve of the log-ratio \log \frac{P_{X,Y}(x,y)}{P_X(x) P_Y(y)}, weighted by the joint density P_{X,Y}(x,y), highlighting regions where dependence inflates or deflates probabilities relative to independence.^[18] In the probability simplex or density space, mutual information thus traces the excess "volume" or separation from the independence manifold. Mutual information emerges as a special case of f-divergences, a broader class of divergences defined by a convex function f, with the KL divergence (and hence mutual information) corresponding to f(u) = u \log u.^[20] This connection underscores mutual information's role within the family of information measures that asymmetrically compare distributions, emphasizing its geometric asymmetry in capturing directional dependence.^[20] Visualizations often employ contour plots of the joint density P_{X,Y}(x,y) overlaid against the independence surface P_X(x) P_Y(y); for independent variables, the contours align, but dependence introduces distortions, with the extent of mismatch reflecting higher mutual information.^[21] Such plots reveal how correlation skews probability mass away from rectangular independence contours toward diagonal or clustered patterns in bivariate space.^[21] In bivariate examples, mutual information increases with the strength of dependence, as seen in scatter plots where linear correlation tightens points along a line (elevating MI from near zero for scattered data to higher values for perfect alignment), while nonlinear dependencies like quadratic relations similarly boost MI beyond what linear measures capture.^[22] For Gaussian variables, this scaling is explicit, with MI growing logarithmically with the absolute correlation coefficient.^[23]

Basic Properties

Non-negativity and symmetry

Mutual information is always non-negative for any pair of random variables X and Y, that is, I(X;Y) \geq 0.^[24] This property follows from the fact that mutual information can be expressed as the Kullback-Leibler divergence between the joint distribution P_{XY} and the product of the marginals P_X \times P_Y, i.e., I(X;Y) = D(P_{XY} \| P_X \times P_Y), and the Kullback-Leibler divergence is non-negative.^[24] To prove the non-negativity of the Kullback-Leibler divergence using Jensen's inequality, consider the discrete case where X and Y take values in finite sets. The divergence is given by

D(P_{XY} \| P_X \times P_Y) = \sum_{x,y} P_{XY}(x,y) \log \frac{P_{XY}(x,y)}{P_X(x) P_Y(y)}.

This can be rewritten as

D(P_{XY} \| P_X \times P_Y) = -\sum_{x,y} P_{XY}(x,y) \log \frac{P_X(x) P_Y(y)}{P_{XY}(x,y)} = E_{P_{XY}} \left[ -\log \frac{P_X(X) P_Y(Y)}{P_{XY}(X,Y)} \right].

The function f(t) = -\log t is convex for t > 0. By Jensen's inequality applied to the expectation under P_{XY},

E_{P_{XY}} \left[ f\left( \frac{P_X(X) P_Y(Y)}{P_{XY}(X,Y)} \right) \right] \geq f\left( E_{P_{XY}} \left[ \frac{P_X(X) P_Y(Y)}{P_{XY}(X,Y)} \right] \right).

The expectation on the right simplifies to \sum_{x,y} P_X(x) P_Y(y) = 1, so f(1) = -\log 1 = 0. Thus, D(P_{XY} \| P_X \times P_Y) \geq 0.^[25] For continuous random variables, the proof is analogous, replacing sums with integrals and relying on the same convexity of -\log.^[26] Mutual information is also symmetric, meaning I(X;Y) = I(Y;X).^[24] This follows directly from the definition, as the joint probability P_{XY}(x,y) = P_{YX}(y,x) and the expression \sum_{x,y} P_{XY}(x,y) \log \frac{P_{XY}(x,y)}{P_X(x) P_Y(y)} remains unchanged when X and Y are swapped.^[5] The continuous case holds similarly. Equality in the non-negativity holds if and only if X and Y are independent, i.e., I(X;Y) = 0 precisely when P_{XY}(x,y) = P_X(x) P_Y(y) for all x,y (discrete case) or almost everywhere (continuous case).^[24] This is because the Kullback-Leibler divergence equals zero if and only if the two distributions are identical, and Jensen's inequality achieves equality when the argument \frac{P_X(x) P_Y(y)}{P_{XY}(x,y)} is constant almost surely, which occurs under independence.^[27] To illustrate non-negativity, consider two binary random variables each uniform on \{[0](/page/0),[1](/page/1)\}. If X and Y are independent, then I(X;Y) = [0](/page/0). If instead Y = X (perfect dependence), the joint distribution has P_{XY}([0](/page/0),[0](/page/0)) = P_{XY}([1](/page/1),[1](/page/1)) = 1/2 and P_{XY}([0](/page/0),[1](/page/1)) = P_{XY}([1](/page/1),[0](/page/0)) = [0](/page/0), yielding I(X;Y) = 1 bit, which is positive.^[28]

Additivity under independence

One key property of mutual information is its additivity for independent components. Specifically, if the joint distribution factors as P_{X,Y,W,Z}(x,y,w,z) = P_{X,Y}(x,y) P_{W,Z}(w,z) (i.e., the pairs (X,Y) and (W,Z) are independent), then I(X, W; Y, Z) = I(X; Y) + I(W; Z).^[6] This additivity reflects the fact that mutual information between independent systems adds up without cross terms. It leverages the non-negativity of mutual information as a foundational bound where I(X; Y) \geq [0](/page/0) and equality holds if and only if X and Y are independent. A related property is the data processing inequality, which states that mutual information cannot increase when one of the variables is further processed through a function. Formally, for any function f, I(X; Y) \geq I(X; f(Y)), with equality if f is invertible.^[29] This inequality implies that in a Markov chain X \to Y \to Z, the mutual information decreases or stays the same along the chain: I(X; Y) \geq I(X; Z). For example, consider a simple binary Markov chain where X is a fair coin flip, Y = X with probability 0.9 and flipped with probability 0.1 (noisy channel), and Z is Y passed through another identical noisy channel; here, I(X; Y) \approx 0.531 bits while I(X; Z) \approx 0.321 bits, illustrating the non-increase. This additivity extends naturally to multiple independent pairs. If several pairs of variables are mutually independent in the same sense, the total mutual information is the sum over the individual pairs.^[6]

Chain rule

The chain rule for mutual information expresses the total mutual information between a joint random variable consisting of a sequence X_1, \dots, X_n and another random variable Y as a sum of conditional mutual informations. Formally,

I(X_1, \dots, X_n; Y) = \sum_{i=1}^n I(X_i; Y \mid X_1, \dots, X_{i-1}),

where the conditioning set is empty for i=1. This identity derives from the definition of mutual information in terms of entropy and the chain rule for entropy. Mutual information satisfies I(X_1, \dots, X_n; Y) = H(X_1, \dots, X_n) - H(X_1, \dots, X_n \mid Y). Applying the chain rule for entropy to the first term gives

H(X_1, \dots, X_n) = \sum_{i=1}^n H(X_i \mid X_1, \dots, X_{i-1}),

and similarly for the conditional entropy,

H(X_1, \dots, X_n \mid Y) = \sum_{i=1}^n H(X_i \mid X_1, \dots, X_{i-1}, Y).

Subtracting these expansions yields

I(X_1, \dots, X_n; Y) = \sum_{i=1}^n \bigl[ H(X_i \mid X_1, \dots, X_{i-1}) - H(X_i \mid X_1, \dots, X_{i-1}, Y) \bigr] = \sum_{i=1}^n I(X_i; Y \mid X_1, \dots, X_{i-1}),

since the conditional mutual information is the difference between these conditional entropies. In applications to sequential prediction, the chain rule decomposes the total information as the incremental contribution of each successive variable: the i-th term measures how much additional uncertainty about Y is reduced by observing X_i after the previous observations X_1, \dots, X_{i-1}. This perspective highlights the marginal benefit of incorporating variables one at a time, which is particularly useful in feature selection or predictive modeling where variables arrive in sequence.^[14] For an illustration with three variables X, Y, Z, the rule specializes to I(X, Y; Z) = I(X; Z) + I(Y; Z \mid X), where the first term captures the direct dependence between X and Z, and the second quantifies the remaining dependence of Y on Z after accounting for X.

Advanced Properties

Relation to Kullback-Leibler divergence

Mutual information I(X; Y) between two random variables X and Y is equivalently defined as the Kullback-Leibler (KL) divergence between their joint probability distribution P_{XY} and the product of their marginal distributions P_X \times P_Y:

I(X; Y) = D_{\mathrm{KL}}(P_{XY} \| P_X \times P_Y).

This representation underscores mutual information as a measure of the deviation from statistical independence, where D_{\mathrm{KL}}(P_{XY} \| P_X \times P_Y) = 0 if and only if X and Y are independent. The identity holds in both discrete and continuous settings, with the KL divergence computed as a summation over joint support or an integral over the joint density, respectively. To derive this equivalence, substitute the definition of the KL divergence into the expression. For discrete random variables,

D_{\mathrm{KL}}(P_{XY} \| P_X \times P_Y) = \sum_{x, y} p_{XY}(x, y) \log \frac{p_{XY}(x, y)}{p_X(x) p_Y(y)},

where the logarithm argument simplifies to \frac{p_{Y|X}(y|x)}{p_Y(y)}. Expanding yields

\sum_x p_X(x) \sum_y p_{Y|X}(y|x) \log \frac{p_{Y|X}(y|x)}{p_Y(y)} = \mathbb{E}_{X} \left[ D_{\mathrm{KL}}(P_{Y|X} \| P_Y) \right] = H(Y) - H(Y|X),

matching the entropic definition of mutual information; the continuous analog uses integrals and differential entropies. This proof demonstrates that mutual information is a special case of the KL divergence tailored to independence testing. Asymptotically, mutual information connects to hypothesis testing via Stein's lemma, which characterizes the optimal error exponent in distinguishing distributions. Specifically, for testing the null hypothesis of independence (H_0: P_{XY} = P_X \times P_Y) against dependence (H_1: P_{XY}), the best type II error probability decays exponentially with rate I(X; Y) = D_{\mathrm{KL}}(P_{XY} \| P_X \times P_Y) as sample size increases, for fixed type I error. This log-likelihood ratio interpretation positions mutual information as the expected excess information needed to discriminate joint dependence from marginal independence in large samples.^[30] For categorical data, this KL form facilitates direct computation. Consider binary variables X, Y \in \{0,1\} with P(X=0)=0.5, P(Y=0|X=0)=0.9, P(Y=0|X=1)=0.1, yielding marginal P(Y=0)=0.5 and joint probabilities p_{00}=0.45, p_{01}=0.05, p_{10}=0.05, p_{11}=0.45. Then,

I(X; Y) = \sum_{x,y \in \{0,1\}} p_{xy} \log \frac{p_{xy}}{0.5 \cdot p_y} \approx 0.531 \text{ bits},

obtained by evaluating each term (e.g., $0.45 \log(0.45 / 0.25) \approx 0.382, and symmetrically for others); this equals H(Y) - H(Y|X) \approx 1 - 0.469. Such examples illustrate how KL computation quantifies dependence in finite discrete spaces.

Supermodularity

Mutual information exhibits submodularity as a set function when considering set-valued random variables. Specifically, for random variables X, Y, and Z where Y and Z are sets, the mutual information satisfies the inequality

I(X; Y \cup Z) + I(X; Y \cap Z) \leq I(X; Y) + I(X; Z).

This property holds because mutual information can be expressed in terms of entropy: I(X; W) = H(X) - H(X \mid W), where H(X) is constant with respect to W. The inequality is thus equivalent to the supermodularity of the conditional entropy function H(X \mid \cdot).^[31] The proof proceeds via inclusion-exclusion principles on entropies, leveraging the non-negativity of mutual information. The submodularity of mutual information follows from the supermodularity of conditional entropy, with equality holding when X is conditionally independent of the symmetric differences given the intersection. This derivation relies on the chain rule for entropy applied to the disjoint components Y \setminus Z, Z \setminus Y, and Y \cap Z.^[31] In feature selection, the submodularity of mutual information implies diminishing returns when incrementally adding variables to a feature set, meaning the marginal information gain from a new variable decreases as the set grows. This property enables efficient greedy algorithms for selecting informative features, providing constant-factor approximation guarantees relative to the optimal subset in high-dimensional settings. For instance, in supervised learning tasks, maximizing I(X; S) over feature sets S benefits from this structure, avoiding exhaustive search while bounding suboptimality.^[32] An application arises in graphical models, particularly Bayesian networks, where mutual information quantifies conditional dependencies between nodes. The submodular property supports optimization in structure learning by facilitating greedy edge additions that respect diminishing marginal contributions, leading to scalable inference with theoretical performance assurances in discovering network topologies from data.^[33]

Bayesian estimation

Bayesian estimation of mutual information addresses the challenge of inferring the dependency between random variables from finite data samples by incorporating prior distributions on the underlying probability densities, thereby quantifying uncertainty in the estimate. Unlike frequentist approaches, Bayesian methods treat the mutual information I(X;Y) as a random variable derived from the posterior distribution over densities, enabling robust inference even with limited data. This is particularly useful in scenarios where parametric assumptions fail, such as in high-dimensional or non-standard distributions.^[34] A prominent non-parametric approach employs Dirichlet process priors to model the joint and marginal densities without specifying a parametric form, allowing flexible estimation of I(X;Y) through posterior sampling. In this framework, the Dirichlet process mixture model generates densities by placing a prior on partitions of the sample space, with the base measure and concentration parameter tuned to the data; the mutual information is then computed as the expected value under the posterior, providing a full distribution rather than a point estimate. This method has demonstrated lower mean squared error compared to some frequentist nonparametric estimators in simulation studies, making it suitable for applications like feature selection in machine learning. Recent advances include flow-based variational Bayesian estimators for improved scalability in high dimensions (as of 2025).^[34]^[35]^[36] Plug-in estimators within a Bayesian context approximate I(X;Y) by first estimating the densities via kernel density estimation (KDE) and then applying Bayesian bias correction to account for finite-sample effects. KDE constructs smooth estimates of the joint density p(x,y) and marginals p(x), p(y) using a kernel function (e.g., Gaussian) and bandwidth parameter, after which the plug-in formula I(X;Y) \approx \sum p(x,y) \log \frac{p(x,y)}{p(x)p(y)} (discretized for computation) yields the estimate; priors on the bandwidth or kernel parameters can be incorporated to regularize the posterior. Bias correction, often via analytical adjustments like those derived from asymptotic expansions, mitigates underestimation in low-sample regimes, with Bayesian variants using Dirichlet priors on binned approximations of KDE outputs for discrete-like handling of continuous data. This approach excels for bivariate cases where direct computation is feasible.^[37]^[38] In model selection tasks, Bayesian mutual information facilitates choosing among competing models by maximizing the expected information gain between parameters and observed data, often integrated into variational inference frameworks. For instance, variational methods approximate intractable posteriors by optimizing a lower bound that incorporates mutual information terms, such as in estimating dependencies for latent variable models; here, self-consistent Bayesian updates enhance data efficiency by iteratively refining the variational distribution to align with the true MI. This is applied in scenarios like structure learning in Bayesian networks, where MI scores with uniform Dirichlet priors guide edge selection, balancing fit and complexity. Recent advancements leverage this for experimental design, prioritizing queries that maximize MI for parameter identifiability.^[39]^[38]^[40] A key challenge in Bayesian estimation of mutual information is the curse of dimensionality, where the required sample size grows exponentially with the number of variables, leading to unreliable density estimates and posterior collapse in high dimensions. For bivariate data (d=2), accurate estimation is achievable with thousands of samples using DP or KDE methods, but in dimensions d > 10, even millions of samples yield biased or high-variance results due to sparse effective support in the density. Mitigation strategies include dimensionality reduction or restricted priors, but these trade off flexibility for tractability.^[41]

Variations

Conditional mutual information

Conditional mutual information is a measure in information theory that quantifies the mutual dependence between two random variables X and Y when a third random variable Z is known, extending the unconditional mutual information to scenarios with conditioning. It represents the expected reduction in uncertainty about X provided by Y, after accounting for the information already available from Z. Formally, for discrete random variables, the conditional mutual information I(X; Y \mid Z) is defined as

I(X; Y \mid Z) = H(X \mid Z) - H(X \mid Y, Z),

where H(\cdot \mid \cdot) denotes conditional entropy. Equivalently, it can be expressed as the expectation over Z:

I(X; Y \mid Z) = \sum_z p(z) \, I(X; Y \mid Z = z),

with a similar integral form \int p(z) \, I(X; Y \mid Z = z) \, dz for continuous variables. This definition arises naturally from the properties of conditional entropy introduced in foundational work on information theory.^[5] Key properties of conditional mutual information include non-negativity, I(X; Y \mid Z) \geq 0, with equality if and only if X and Y are conditionally independent given Z, i.e., p(x, y \mid z) = p(x \mid z) p(y \mid z) for all x, y, z with p(z) > 0. It is also symmetric in X and Y, so I(X; Y \mid Z) = I(Y; X \mid Z). Additionally, it satisfies a chain rule analogous to that for unconditional mutual information:

I(X; YZ) = I(X; Y) + I(X; Z \mid Y),

which decomposes the mutual information between X and the joint variable (Y, Z) into the direct dependence on Y plus the additional dependence on Z given Y. These properties hold for both discrete and continuous cases and facilitate derivations in multi-variable settings. The interpretation of conditional mutual information is the amount of shared information between X and Y that remains after conditioning on Z, capturing dependencies not explained by Z alone. For instance, in a Markov chain where X \to Z \to Y, the conditional mutual information I(X; Y \mid Z) = 0, indicating that Z fully mediates the dependence between X and Y, leaving no residual direct information flow. This property is central to applications in causal modeling and graphical models.

Normalized mutual information

Normalized mutual information (NMI) provides a bounded measure of dependence between two random variables by normalizing the mutual information to the range [0, 1], enabling straightforward interpretation and comparison across datasets with varying scales or cardinalities. A widely used symmetric variant is given by

\text{NMI}(X; Y) = \frac{I(X; Y)}{\sqrt{H(X) H(Y)}},

where I(X; Y) denotes the mutual information between X and Y, and H(X) and H(Y) are their respective entropies; this formulation ensures the measure is invariant to monotonic transformations of the variables and achieves a maximum value of 1 when X and Y have equal entropy and are fully dependent (e.g., one is a bijection of the other). This version was originally proposed for evaluating multimodality image alignment, where it demonstrated robustness to changes in image overlap. Alternative normalizations include dividing by the minimum marginal entropy, \text{NMI}(X; Y) = I(X; Y) / \min(H(X), H(Y)), which also bounds the value at most 1 since I(X; Y) \leq \min(H(X), H(Y)), or the uncertainty coefficient, an asymmetric form U(X \mid Y) = I(X; Y) / H(X), which quantifies the fraction of uncertainty in X resolved by knowing Y.^[42] The uncertainty coefficient originates from early applications in statistical computing for assessing variable associations.^[42] For discrete random variables, NMI is computed using estimated probabilities from a contingency table, where joint probabilities p(x, y) are the normalized counts of co-occurrences, marginal probabilities p(x) and p(y) are row and column sums divided by the total count, and entropies and mutual information follow the standard sums: H(X) = -\sum_x p(x) \log p(x), I(X; Y) = \sum_{x,y} p(x, y) \log \frac{p(x, y)}{p(x) p(y)}.^[43] This approach is particularly effective for categorical data in practice, such as cluster labels. The primary advantages of NMI lie in its scale invariance and bounded range, making it suitable for tasks like clustering evaluation, where it compares predicted partitions to ground-truth labels regardless of the number of clusters in each, with values near 1 indicating strong agreement and 0 indicating independence.^[43] However, NMI is not a true metric, as it fails to satisfy the triangle inequality, and remains sensitive to imbalances in marginal entropies, which can diminish its value for strong dependencies involving high-entropy variables.^[42]

Directed and transfer entropy variants

Directed information extends the concept of mutual information to time series data by incorporating causality and feedback, quantifying the information flow from one process to another in a directed manner. Formally, for two stochastic processes X and Y, the directed information from X to Y is defined as

I(X \to Y) = \sum_{t=1}^n I(X_t ; Y_t \mid Y_{1:t-1}),

where I(\cdot ; \cdot \mid \cdot) denotes conditional mutual information, capturing how past values of Y condition the dependence between current X_t and Y_t. This measure was introduced to address limitations of standard mutual information in channels with feedback, providing a more appropriate framework for causal inference in sequential data.^[44] Transfer entropy, a related asymmetric variant, specifically measures the directed information transfer from the past of one process to the future of another, conditional on the receiver's own past. It is given by

TE_{X \to Y} = I(X_{1:t-1} ; Y_t \mid Y_{1:t-1}),

averaged over time t, and serves as a model-free tool to detect effective connectivity in complex systems without assuming underlying dynamics. Proposed as a practical implementation for empirical data analysis, transfer entropy distinguishes driving influences from mere correlations by isolating predictive information beyond the target process's autocorrelation.^[45] Both directed information and transfer entropy find applications in inferring causality, particularly through links to Granger causality, where non-zero values indicate that one time series contains information that improves prediction of the other beyond its own history. For instance, in econometric and neural data, directed information has been shown to align with Granger-noncausality conditions under Gaussian assumptions, enabling the construction of causality graphs for multivariate processes. Transfer entropy extends this to nonlinear settings, offering robustness in detecting asymmetric interactions in fields like climate modeling and finance.^[46]^[47] A representative example is unidirectional coupling in stochastic processes, such as a system where process X drives Y but not vice versa, modeled by Y_t = f(Y_{t-1}, X_{t-\tau}) + \epsilon_t with noise \epsilon_t. Here, transfer entropy TE_{X \to Y} yields a positive value reflecting the information flow, while TE_{Y \to X} approaches zero, demonstrating the measure's ability to uncover directional dependencies in simulated bidirectional versus unidirectional scenarios.^[45]

Applications

In statistics and machine learning

In statistics, mutual information serves as a key measure for feature selection by quantifying the dependency between features and the target variable while accounting for redundancies among features. The minimum redundancy maximum relevance (mRMR) algorithm, for instance, selects features that maximize mutual information with the target (relevance) while minimizing mutual information among the selected features themselves (redundancy), enabling efficient handling of high-dimensional datasets in classification tasks.^[48] This approach has been shown to outperform traditional correlation-based methods in gene expression analysis and text categorization by preserving predictive power with fewer features.^[49] In machine learning, mutual information is integral to clustering evaluation, where adjusted mutual information (AMI) provides a normalized metric to compare predicted clusterings against ground truth partitions, correcting for chance agreements. AMI is computed as the mutual information between cluster assignments minus its expected value under a random hypergeometric model, divided by the square root of the product of entropies adjusted for chance, yielding values between -1 and 1 where 1 indicates perfect agreement and 0 indicates random labeling.^[50] This measure is particularly useful in hierarchical clustering algorithms, such as those applied to bioinformatics datasets, as it remains robust to varying cluster sizes and numbers, unlike unadjusted variants.^[51] For dimensionality reduction, the information bottleneck (IB) method employs mutual information to compress input data into a lower-dimensional representation that retains maximal relevant information about a target variable. The IB Lagrangian balances the compression term, which minimizes mutual information between input X and representation Z (i.e., I(X; Z)), against the preservation term, which maximizes mutual information between Z and target Y (i.e., I(Z; Y)), optimized via:

\min_{p(z|x)} I(X; Z) - \beta I(Z; Y),

where \beta > 0 trades off compression and relevance; solutions are found iteratively using generalized Blahut-Arimoto algorithms.^[52] This framework has influenced manifold learning and has been applied in speech recognition to extract succinct features that improve downstream prediction accuracy.^[53] Recent advances since 2020 have integrated mutual information into deep generative models, particularly variational autoencoders (VAEs), to enhance latent space disentanglement and generation quality. The variational mutual information maximizing (VMI) framework for VAEs maximizes mutual information between latent variables and data while constraining posterior collapse, leading to more informative encodings in image synthesis tasks.^[54] Similarly, the VOLTA autoencoder integrates variational mutual information maximization within a Transformer-VAE structure to improve generative diversity in natural language generation tasks, enhancing metrics such as Self-BLEU and Distinct-n compared to standard VAEs.^[55] These methods leverage MI estimation techniques, such as variational bounds, to address limitations in traditional VAEs' posterior approximations.

In communication theory

In communication theory, mutual information plays a central role in quantifying the reliable transmission of information over noisy channels, as established by Claude Shannon's foundational work. Shannon's noisy-channel coding theorem, published in 1948, demonstrates that reliable communication is possible at rates below the channel capacity, defined as the maximum mutual information between the input and output of the channel.^[5] Specifically, for a discrete memoryless channel, the capacity C is given by

C = \max_{p(x)} I(X; Y),

where the maximum is taken over all possible input distributions p(x), and I(X; Y) measures the reduction in uncertainty about the input X provided by the output Y. This theorem shows that error probability can be made arbitrarily small for rates R < C, but impossible for R > C, marking a profound shift from prior beliefs that noise fundamentally limited communication efficiency.^[5] Mutual information also bounds the trade-off between data compression and fidelity in rate-distortion theory, another cornerstone of Shannon's contributions. In this framework, the rate-distortion function R(D) represents the minimum rate required to encode a source at distortion level D, expressed as the infimum of mutual information I(X; \hat{X}) over all conditional distributions p(\hat{x}|x) satisfying the expected distortion constraint E[d(X, \hat{X})] \leq D.^[56] Shannon proved that this function is achievable, providing the theoretical limit for lossy compression schemes, where mutual information captures the essential information preserved in the reconstruction \hat{X}. For example, in encoding a discrete source with quadratic distortion, R(D) decreases as allowable distortion D increases, reflecting the diminishing returns of additional bits for higher fidelity.^[56] In multi-user communication scenarios, mutual information extends to conditional forms to characterize capacities of broadcast and multiple-access channels. For a broadcast channel, where a single transmitter sends to multiple receivers over correlated channels, the capacity region involves maximizing rates using expressions like I(X; Y_1 | U) and I(X; Y_2 | U) for auxiliary random variable U, enabling degraded message sets or superposition coding strategies.^[57] Similarly, in a multiple-access channel with multiple transmitters sharing a common receiver, the capacity region is bounded by individual rates R_1 \leq I(X_1; Y | X_2), R_2 \leq I(X_2; Y | X_1), and sum rate R_1 + R_2 \leq I(X_1, X_2; Y), where conditional mutual information accounts for interference between users. These formulations, developed in the 1970s building on Shannon's foundations, guide practical systems like cellular networks by optimizing resource allocation under multi-user constraints.^[58]

In neuroscience and other fields

In neuroscience, mutual information serves as a robust measure for quantifying functional connectivity between neural signals, capturing nonlinear dependencies that linear correlation methods often miss. For instance, in electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) analyses, mutual information has been applied to construct brain networks that reveal physiologically relevant architectures, such as altered connectivity in conditions like post-stroke depression.^[59] This approach highlights shared information between brain regions during cognitive tasks, providing insights into network dynamics beyond pairwise correlations. Directed variants of mutual information, such as transfer entropy, extend this to infer causal influences in neural circuits.^[60] In genetics, normalized mutual information quantifies linkage disequilibrium (LD), the non-random association of alleles at different loci, offering a multivariate extension to traditional pairwise measures like D'. By extending mutual information theory, researchers have developed multilocus LD metrics that assess statistical dependencies across multiple single nucleotide polymorphisms (SNPs), aiding in tagging SNP selection for genome-wide association studies.^[61] This normalization ensures comparability across datasets, revealing epistatic interactions that influence disease susceptibility. In physics, mutual information acts as the classical analog in quantum information theory, measuring correlations between subsystems of quantum states while underpinning thermodynamic interpretations of information processing. It quantifies entanglement growth in quantum systems and bounds the second law for open quantum systems coupled to reservoirs, linking information flows to entropy production. For example, in thermodynamic contexts, mutual information describes how correlations evolve under quantum scrambling, providing a bridge between classical information theory and quantum irreversibility. Emerging applications in 2025 leverage mutual information to model interactions among climate variables, disentangling internal variability in global circulation models through network-based analyses. By computing mutual information between time series of meteorological and soil variables, researchers identify nonlinear dependencies that enhance drought prediction and bias correction in Earth system models, preserving inter-variable structures under climate change scenarios.^[62]

References

[1]
[PDF] The Bell System Technical Journal
JIlly, 1948. No.3. A Mathematical Theory of Communication. By c. E. SHANNON. IXTRODUCTION. THE recent development of various methods of modulation such as reM.
[2]
Detecting and Evaluating Dependencies between Variables
Aug 7, 2025 · Mutual information quantifies the statistical dependency between two random variables by measuring how much knowing one reduces uncertainty ...
[3]
Sliced Mutual Information: A Scalable Measure of Statistical ...
Oct 11, 2021 · Mutual information (MI) is a fundamental measure of statistical dependence, with a myriad of applications to information theory, statistics, and ...
[4]
Equitability, mutual information, and the maximal information ... - PNAS
Mutual information rigorously quantifies, in units known as “bits,” how much information the value of one variable reveals about the value of another. This has ...<|control11|><|separator|>
[5]
[PDF] A Mathematical Theory of Communication
Reprinted with corrections from The Bell System Technical Journal,. Vol. 27, pp. 379–423, 623–656, July, October, 1948. A Mathematical Theory of Communication.
[6]
Mutual information - Scholarpedia
Oct 11, 2018 · Mutual information is one of many quantities that measures how much one random variables tells us about another.
[7]
[PDF] Entropy and Information Theory - Stanford Electrical Engineering
This book is devoted to the theory of probabilistic information measures and their application to coding theorems for information sources and noisy channels ...
[8]
[PDF] Entropy, Relative Entropy and Mutual Information - Columbia CS
It is the reduction in the uncertainty of one random variable due to the knowledge of the other. Definition: Consider two random variables X and Y with a joint.
[9]
[PDF] Lecture 10: Mutual Information
Mar 5, 2020 · (i) Mutual information is a fundamental measure of dependence between random variables: it is invari- ant to invertible transformations of the ...
[10]
[PDF] CS258: Information Theory
This is the master definition of mutual information that always applies, even to joint distributions with atoms, densities, and singular parts. Page 15 ...
[11]
Mutual information of continuous variables - Math Stack Exchange
Jun 6, 2018 · For continuous variables, if X=Y, the mutual information is infinite, as H(Y|X) is -infinity in differential entropy.How is a singular continuous measure defined?Algorithm for calculating the mutual information between continuous ...More results from math.stackexchange.comMissing: singular | Show results with:singular
[12]
https://ieeexplore.ieee.org/document/6875281
[13]
[PDF] ECE 587 / STA 563: Lecture 7 – Differential Entropy - Galen Reeves
Aug 24, 2023 · 1. 2 log(1 − ρ2) = 1. 2 log. 1. 1 − ρ2. ◦ Note that if ρ = ±1 then X = Y and the mutual information is positive infinity! • Example: ( ...
[14]
[PDF] Lecture 2 — January 12 2.1 Outline 2.2 Entropy 2.3 The Chain Rule ...
Chain rules like this are important because we often encounter long chains of random variables, not just one or two! 2.4 Mutual Information. Given two random ...Missing: sum | Show results with:sum
[15]
[PDF] On Complexity and Efficiency of Mutual Information Estimation on ...
Mar 26, 2018 · Mutual Information (MI) is an established measure for the de- pendence of two variables and is often used as a generalization of correlation ...
[16]
[PDF] Entropy and Mutual Information
For example, suppose X represents the roll of a fair 6-sided die, and Y represents whether the roll is even (0 if even, 1 if odd). Clearly, the value of Y ...Missing: dice | Show results with:dice
[17]
[PDF] On Study of Mutual Information and Its Estimation Methods - arXiv
Jun 21, 2021 · The preliminaries section helps the reader to understand the basic concepts of information theory. The MI: definitions and properties section is ...Missing: scholarly | Show results with:scholarly
[18]
How the Choice of Distance Measure Influences the Detection of ...
The KL-divergence expresses the loss of information that occurs when we rely on ... KL-divergence, which is equal to the highlighted area under the curve.
[19]
[PDF] f-divergences and their applications in lossy compression ... - arXiv
Jan 26, 2023 · F-divergence, or generalized relative entropy, is a measure of dissimilarity between two distributions on the same sample space.
[20]
[PDF] An application of mutual information in ... - Korea Science
... joint distribution f(x, y) be defined as H(X, Y ) ... Figure 2.2 Surface and contour plot of joint pdf in example 2.2 with α = 1 ... Estimating mutual information.
[21]
[PDF] MEASURING DEPENDENCE VIA MUTUAL INFORMATION
Mutual information has properties that are desirable for a dependence measure. For example, (1) I(X;Y ) ≥ 0; (2) I(X;Y ) = 0 if and only if X and Y are inde- ...<|control11|><|separator|>
[22]
Proof: Mutual information of the bivariate normal distribution
Nov 1, 2024 · Proof: Mutual information can be written in terms of marginal and joint differential entropy: I(X,Y)=h(X)+h(Y)−h(X,Y). (3)Missing: closed form
[23]
[PDF] elements of information theory - IIS Windows Server
Page 1. Page 2. ELEMENTS OF. INFORMATION THEORY. Second Edition. THOMAS M. COVER. JOY A. THOMAS. A JOHN WILEY & SONS, INC., PUBLICATION. Page 3. ELEMENTS OF.
[24]
[PDF] Lecture 3 — Jan 17 3.1 Outline 3.2 Recap 3.3 Relative Entropy
For random variables X and Y ,. H(X|Y ) ≤ H(X). Interpretation in words - The nonnegativity of mutual information implies that “on average” the entropy of X ...
[25]
[PDF] KL-divergence and connections 1 Recap 2 More mutual information
Jan 22, 2013 · The proof again uses concavity and Jensen's Inequality. We will show that −D(p||q) ≤ 0. −D(p||q) = ∑ x.
[26]
[PDF] Lecture 3: Chain Rules and Inequalities
Since I(X;Y ) = D(p(x, y)||p(x)p(y)). • Conditional relative entropy and mutual information are also nonnegative. Dr. Yao Xie, ECE587, Information Theory, Duke ...
[27]
[PDF] Lecture 1: Entropy and mutual information
So we see that conditioning of the mutual information can both increase or decrease it depending on the situation.
[28]
[PDF] Lecture 4: October 9, 2017 1 More on mutual information - TTIC
Oct 9, 2017 · Even though the KL-divergence is not symmetric, it is often used as a measure of “dissimilarity” between two distribution. Towards this, we.<|control11|><|separator|>
[29]
[PDF] INFORMATION THEORY AND STATISTICS - FR
The best achievable error exponent in the probability of error for this problem is given by the. Chernoff–Stein lemma. We first prove the Neyman–Pearson lemma, ...
[30]
[PDF] Submodular functions - Columbia University
May 9, 2021 · Abstract. These notes contain examples of submodular functions and describe certain algorithms for optimizing them.
[31]
[PDF] Near-Optimal Sensor Placements in Gaussian Processes
We first prove that maximizing mutual information is an NP-complete problem. Then, by exploiting the fact that mutual information is a submodular function (cf.<|control11|><|separator|>
[32]
[PDF] A submodular-supermodular procedure with applications to ... - arXiv
In this paper, we present an algorithm for minimizing the difference between two sub- modular functions using a variational frame-.
[33]
A Bayesian Nonparametric Estimation of Mutual Information - arXiv
Aug 9, 2021 · Mutual information is a widely-used information theoretic measure to quantify the amount of association between variables. It is used ...
[34]
A test for independence via Bayesian nonparametric estimation of ...
Aug 23, 2021 · In this article, a Bayesian nonparametric estimator of mutual information is established by means of the Dirichlet process and the k-nearest neighbour distance.
[35]
Estimation of mutual information using kernel density estimators
Sep 1, 1995 · It is shown here that kernel density estimation of the probability density functions needed in estimating the average mutual information across two coordinates ...
[36]
Bayesian and Quasi-Bayesian Estimators for Mutual Information ...
Mutual information (MI) quantifies the statistical dependency between a pair of random variables, and plays a central role in the analysis of engineering ...
[37]
https://link.aps.org/doi/10.1103/PhysRevE.52.2318
[38]
[PDF] Bayesian Experimental Design for Implicit Models by Mutual ...
The field of Bayesian experimental design advocates that, ideally, we should choose designs that maximise the mutual information (MI) between the data and the ...<|control11|><|separator|>
[39]
Accurate estimation of the normalized mutual information of ...
Aug 2, 2024 · In the continuous case, on the other hand, the probability density is normalized in Eq. (10) such that the area under its curve is one ...
[40]
http://proceedings.mlr.press/v119/kleinegesse20a/kleinegesse20a.pdf
[41]
Information Theoretic Measures for Clusterings Comparison
In this paper, we perform an organized study of information theoretic measures for clustering comparison, including several existing popular measures in the ...
[42]
[PDF] Causality, feedback and directed information.
It is shown that, when feedback is present, directed information is a more useful quantity than the traditional mutual information. INTRODUCTION. Information ...Missing: seminal | Show results with:seminal
[43]
Measuring Information Transfer | Phys. Rev. Lett.
Jul 10, 2000 · ... transfer entropy is able to distinguish effectively driving and responding elements and to detect asymmetry in the interaction of subsystems.Missing: seminal | Show results with:seminal
[44]
The relation between Granger causality and directed information ...
Nov 14, 2012 · Abstract:This report reviews the conceptual and theoretical links between Granger causality and directed information theory.
[45]
Estimating the directed information to infer causal relationships ... - NIH
This paper connects a newly defined information-theoretic concept of “directed information” to the Granger's philosophical relationship between causality and ...Missing: seminal | Show results with:seminal
[46]
Feature selection based on mutual information criteria of max ...
We study how to select good features according to the maximal statistical dependency criterion based on mutual information.
[47]
[PDF] Criteria of Max-Dependency, Max-Relevance, and Min-Redundancy
In this paper, we focus on the discussion of mutual-information-based feature selection. Given two random variables x and y, their mutual information is defined ...Missing: seminal | Show results with:seminal<|separator|>
[48]
[PDF] Information Theoretic Measures for Clusterings Comparison
Abstract. Information theoretic measures form a fundamental class of measures for comparing clusterings, and have recently received increasing interest.
[49]
[PDF] Information Theoretic Measures for Clusterings Comparison
In this context, it is preferable to employ a normalized measure such as the Normalized Mutual Information (NMI), with fixed bounds 0 and 1. The NMI however ...
[50]
[physics/0004057] The information bottleneck method - arXiv
Apr 24, 2000 · The information bottleneck method finds a short code for signal x that preserves maximum information about signal y, squeezing information ...
[51]
[PDF] The information bottleneck method - Princeton University
We define the relevant information in a signal x ∈ X as being the in- formation that this signal provides about another signal y ∈ Y . Examples.
[52]
[2005.13953] VMI-VAE: Variational Mutual Information Maximization ...
May 28, 2020 · We propose a Variational Mutual Information Maximization Framework for VAE to address this issue. It provides an objective that maximizes the mutual ...
[53]
[PDF] Coding Theorems for a Discrete Source With a Fidelity Criterion
With a Fidelity Criterion-. Claude E. Shannon**. Abstract. Consider a discrete source producing a sequence of message letters from a finite alphabet. A single ...
[54]
[PDF] Broadcast Channels - Information Systems Laboratory
Broadcast channels model a single source communicating with multiple receivers, like a broadcaster or lecturer, using different channels with a common input ...
[55]
[PDF] Elements of Information Theory
Page 1. Page 2. ELEMENTS OF. INFORMATION THEORY. Second Edition. THOMAS M. COVER ... First, certain quantities like entropy and mutual information arise as the ...