Independence

Independence is the one concept in probability with no counterpart in general measure theory — and even it is only a disguise, since it says exactly that a joint law is a product measure (product-measures.md). What makes it central is that essentially every classical theorem (the laws of large numbers, the central limit theorem, the zero–one laws) is a statement about independent sequences, and that independence is what makes infinite-dimensional constructions tractable.

References: Durrett, Probability, §2.1; Billingsley, Probability and Measure, §4; Kallenberg, Foundations, ch. 3.

1. Definitions

Definition (events). Events $A_1,\dots ,A_n$ are independent if for every subset $S\subseteq \{1,\dots ,n\}$,

$$P\left(\bigcap _{i\in S}{A}_i\right)=\prod _{i\in S}P(A_i).$$

An infinite family is independent if every finite subfamily is.

Definition ($\sigma$-algebras). Sub-$\sigma$-algebras ${\mathcal{G}}_1,\dots ,>{\mathcal{G}}_n\subseteq \mathcal{F}$ are independent if $P(G_1\cap \cdots \cap G_n)={\prod }_{i}P(G_i)$ for all $G_i\in {\mathcal{G}}_i$.

Definition (random variables). $X_1,\dots ,X_n$ are independent if the $\sigma$-algebras $\sigma (X_1),\dots ,\sigma (X_n)$ are — equivalently, if the joint law factors:

$$P_{(X_1,\dots ,X_n)}=P_{X_1}\otimes \cdots \otimes P_{X_n}.$$

The third definition is the fundamental one; the others are special cases ($A$ independent of $B$ iff $1_A$ and $1_B$ are). Note that all subsets must factor in the first definition — pairwise independence is strictly weaker.

Pairwise $\ne$ mutual. Toss two fair coins; let $A$ = first is heads, $B$ = second is heads, $C$ = the results agree. Any two of $A,B,C$ are independent, but $P(A\cap B\cap C)=1/4\ne 1/8$. Any theorem quoted for "independent" variables should be checked against which version it needs — the strong law needs only pairwise independence (Etemadi), while most others need mutual.

2. Checking Independence on a $\pi$-System

Verifying the factorization for all events in a $\sigma$-algebra is impossible directly; as always, one checks it on generators.

Theorem. Let ${\mathcal{P}}_i$ be $\pi$-systems with $\sigma ({\mathcal{P}}_i)=>{\mathcal{G}}_i$. If $P({\bigcap }_i{A}_i)={\prod }_{i}P(A_i)$ for all $A_i\in >{\mathcal{P}}_i$, then ${\mathcal{G}}_1,\dots ,{\mathcal{G}}_n$ are independent.

Proof. Fix $A_2,\dots ,A_n$ and apply the $\pi$–$\lambda$ theorem (measure-theory.md §4) to $\{A_1:P(\bigcap A_i)=\prod P(A_i)\}$, which is a $\lambda$-system containing ${\mathcal{P}}_1$; iterate coordinate by coordinate. $\square$

Corollary (the practical criterion). $X_1,\dots ,X_n$ are independent iff

$$F_{(X_1,\dots ,X_n)}(a_1,\dots ,a_n)=\prod _{i=1}^n{F}_{X_i}(a_i)\text{for all }{a}_i,$$

since the rays $\{X_i\le a\}$ generate $\sigma (X_i)$ and form a $\pi$-system. Equivalently, when densities exist, $p_{(X_1,\dots ,X_n)}={\prod }_i{p}_{X_i}$ a.e.

Independence is preserved by functions: if $X_1,\dots ,X_n$ are independent and $g_i$ Borel, so are $g_1(X_1),\dots ,g_n(X_n)$ (immediate, since $\sigma (g_i(X_i))\subseteq \sigma (X_i)$). More useful is the grouping lemma: disjoint blocks of an independent family generate independent $\sigma$-algebras, so $(X_1+X_2)$ and $(X_3{X}_4)$ are independent when $X_1,\dots ,X_4$ are.

3. Existence of Independent Sequences

"Let $X_1,X_2,\dots$ be i.i.d. with law $\nu$" presupposes a probability space carrying them. It exists:

Theorem. For any sequence of Borel probability measures ${\nu }_k$ on $\mathbb{R}$ there is a probability space carrying independent $X_k$ with $P_{X_k}={\nu }_k$ — namely $({\mathbb{R}}^{\mathbb{N}},>{⨂}_k\mathcal{B},{⨂}_k{\nu }_k)$ with $X_k$ the coordinate maps.

This is the infinite product measure of product-measures.md §5. Alternatively, and more economically, the binary digits of a uniform $U\in (0,1)$ are i.i.d. fair bits; regrouping them into countably many subsequences and applying the quantile transform (random-variables.md §5) manufactures any i.i.d. sequence on the unit interval. Either way the existence question is settled once and never revisited.

4. Consequences for Expectation

Theorem (product rule). If $X,Y$ are independent and either both are non-negative or both integrable, then $XY$ is integrable and $E[XY]=E[X]E[Y]$.

Proof. Fubini/Tonelli applied to $\int xyd(P_X\otimes P_Y)$ (product-measures.md §3). $\square$

Consequences: $\mathrm{Var}(X+Y)=\mathrm{Var}(X)+\mathrm{Var}(Y)$ for independent (indeed merely uncorrelated) variables, and ${\phi }_{X+Y}={\phi }_X{\phi }_Y$ for characteristic functions (expectation.md §5) — the identity on which the central limit theorem is built. The law of $X+Y$ is the convolution $P_X\ast P_Y$.

Uncorrelated $\ne$ independent. $E[XY]=E[X]E[Y]$ is strictly weaker: with $X$ standard normal and $Y=X^2$, $E[XY]=E[X^3]=0=E[X]E[Y]$, yet $Y$ is a function of $X$. Independence says all bounded functions decorrelate, not just the identity. The exception is the jointly Gaussian case, where uncorrelated does imply independent — a special property of that family, not a general fact.

5. Borel–Cantelli

For events $A_1,A_2,\dots$, write

$$\{A_k\text{ i.o.}\}=\underset{k}{\mathrm{limsup}}{A}_k=\bigcap _{n=1}^{\infty }\bigcup _{k\ge n}{A}_k$$

for the event that infinitely many $A_k$ occur ("infinitely often").

First Borel–Cantelli lemma. If ${\sum }_{k}P(A_k)<\infty$ then $P(A_k\text{ i.o.})=0$. No independence required.

Proof. $P({\bigcup }_{k\ge n}{A}_k)\le {\sum }_{k\ge n}P(A_k)\to 0$ as the tail of a convergent series; conclude by continuity from above. $\square$

Second Borel–Cantelli lemma. If the $A_k$ are independent and ${\sum }_{k}P(A_k)=\infty$, then $P(A_k\text{ i.o.})=1$.

Proof. $P\left({\bigcap }_{k=n}^m{A}_k^c\right)={\prod }_{k=n}^m(1-P(A_k))\le \exp (-{\sum }_{k=n}^{m}P(A_k))\to 0$ as $m\to \infty$, using $1-x\le e^{-x}$. So $P({\bigcup }_{k\ge n}{A}_k)=1$ for every $n$. $\square$

Together they give a zero–one dichotomy for independent events: $P(A_k\text{ i.o.})$ is $0$ or $1$ according as ${\sum }_{k}P(A_k)$ converges or diverges. This is the prototype of zero–one laws, and the standard tool for converting a summability computation into an almost-sure statement — the route by which the strong law of large numbers is proved (laws-of-large-numbers.md).

Worked instance (monkeys and typewriters). In an i.i.d. sequence of uniform letters, let $A_k$ be the event that positions $kn+1,\dots ,kn+n$ spell a fixed word of length $n$. The $A_k$ are independent with $P(A_k)=26^{-n}>0$, so ${\sum }_{k}P(A_k)=\infty$ and the word appears infinitely often almost surely. Nothing about "large numbers" is used — just divergence of a constant series.

6. Independence as a Modelling Assumption

Independence is a property of $P$, not of the physical setup: it is stipulated when a model is written down, exactly as §2(c) of kolmogorov-axioms.md says the measure must be. The theorems that follow are conditional on that stipulation. Two structural weakenings matter downstream:

• Exchangeability — invariance of the joint law under finite permutations — is strictly weaker, and de Finetti's theorem shows exchangeable sequences are precisely mixtures of i.i.d. ones (exchangeability-de-finetti.md).
• Martingale differences — conditional independence of increments given the past — is weaker still, and suffices for versions of the limit theorems (martingales.md).

7. Where This Is Used

• Laws of large numbers, the CLT — the classical i.i.d. theorems.
• Zero–one laws — Borel–Cantelli generalized.
• Exchangeability and de Finetti — what replaces independence when only symmetry is assumed.
• Product measures — the measure-theoretic content of the whole page.