Modes of Convergence

"$X_k\to X$" is four different statements in probability, and the limit theorems of this part differ mainly in which one they assert. The weak law of large numbers and the strong law have the same hypotheses and the same limit; they differ only in the mode of convergence, and that difference is the whole reason both are stated.

This page fixes the definitions, settles the implications between them, and introduces the two technical devices — uniform integrability and tightness — that convert weaker convergence into stronger.

References: Billingsley, Convergence of Probability Measures; Durrett, Probability, §2.3, 3.2; Williams, Probability with Martingales, ch. 13.

1. The Four Modes

Let $X,X_1,X_2,\dots$ be random variables.

Almost sure. $X_k\stackrel{\text{a.s.}}{\to }X$ if $P(\{\omega :X_k(\omega )\to X(\omega )\})=1$.

In probability. $X_k\stackrel{P}{\to }X$ if $P(|X_k-X|>\epsilon )\to 0$ for every $\epsilon >0$.

In $L^p$ ($p\ge 1$). $X_k\stackrel{L^p}{\to }X$ if $E|X_k-X{|}^p\to 0$.

In distribution. $X_k\Rightarrow X$ if $F_{X_k}(a)\to F_X(a)$ at every continuity point $a$ of $F_X$.

The first three compare functions on a common $\Omega$; the fourth compares only laws, and the variables need not live on the same space. That is why $\Rightarrow$ behaves differently from the others throughout.

Why "continuity points" in the definition of $\Rightarrow$. Take $X_k\equiv >1/k$ and $X\equiv 0$. Then $F_{X_k}(0)=0$ for all $k$ but $F_X(0)=1$, so requiring convergence everywhere would deny that $1/k\to 0$ in distribution. The restriction is the minimal repair, and it is what makes $\Rightarrow$ a statement about measures rather than about the arbitrary $\le$ convention in the CDF.

2. The Implication Lattice

$$\begin{array}{ccc}{X}_k\stackrel{\text{a.s.}}{\to }X& & \\ & \searrow & \\ & & X_k\stackrel{P}{\to }X⟹X_k\Rightarrow X\\ & \nearrow & \\ X_k\stackrel{L^p}{\to }X& & \end{array}$$

No other arrow holds in general, and the following facts complete the picture.

• a.s. $\Rightarrow$ in probability. By dominated convergence applied to $1_{\{|X_k-X|>\epsilon \}}$.
• $L^p\Rightarrow$ in probability. Markov: $P(|X_k-X|>\epsilon )\le E|X_k-X{|}^p/{\epsilon }^p$.
• In probability $\Rightarrow$ in distribution. A short $\epsilon$-argument on CDFs at continuity points.
• In probability $\nRightarrow$ a.s. The typewriter sequence: indicators of $[j/2^n,(j+1)/2^n]$ on $([0,1],{\lambda }^1)$, listed in order. Each has $L^1$ norm $2^{-n}\to 0$, so convergence holds in probability and in $L^1$; but every $\omega$ is covered infinitely often, so $X_k(\omega )\nrightarrow 0$ for any $\omega$.
• a.s. $\nRightarrow L^p$. $X_k=k{1}_{(0,1/k)}$ tends to $0$ pointwise but $E[X_k]=1$ — escaping mass, exactly the failure mode Fatou tolerates (lebesgue-integral.md §4).
• In distribution $\nRightarrow$ in probability. With $X$ symmetric and $X_k=-X$ for all $k$: $X_k\Rightarrow X$ trivially, yet $|X_k-X|=2|X|$.

Two partial converses do most of the practical work:

Subsequence principle. $X_k\stackrel{P}{\to }X$ $⟺$ every subsequence has a further subsequence converging a.s. In particular convergence in probability implies a.s. convergence along a subsequence — the same extraction used to prove $L^p$ complete.

Convergence to a constant. $X_k\Rightarrow c$ (a constant) $⟹X_k>\stackrel{P}{\to }c$. This is why the weak law can be stated either way.

Slutsky's theorem. If $X_k\Rightarrow X$ and $Y_k\Rightarrow c$ then $X_k+Y_k\Rightarrow X+c$ and $X_k{Y}_k\Rightarrow cX$. Note the asymmetry: $\Rightarrow$ is not preserved by sums in general, because it says nothing about joint laws (random-variables.md §5).

Also constantly used: the continuous mapping theorem — if $g$ is continuous $P_X$-a.e., then $X_k\Rightarrow X⟹g(X_k)\Rightarrow g(X)$, and likewise for the other three modes.

3. Uniform Integrability

The gap between a.s. (or in-probability) convergence and $L^1$ convergence is exactly escaping mass. Uniform integrability closes it.

Definition. A family $\{X_i{\}}_{i\in I}$ is uniformly integrable (UI) if

$$\underset{M\to \infty }{\lim }\underset{i\in I}{\sup }E[|X_i|1_{\{|X_i|>M\}}]=0.$$

Sufficient conditions: domination by a single $Y\in L^1$; or boundedness in $L^p$ for some $p>1$ (de la Vallée Poussin, in its simplest form). So UI is strictly weaker than the hypothesis of dominated convergence.

Theorem (Vitali). $X_k\stackrel{L^1}{\to }X$ $⟺$ $X_k\stackrel{P}{\to }X$ and $\{X_k\}$ is UI.

This is the sharp form of the dominated convergence theorem, and it is the natural hypothesis in martingale theory: a martingale converges in $L^1$ — and is closed by its limit — precisely when it is UI (martingales.md).

4. Weak Convergence and Tightness

Convergence in distribution is best seen as convergence of measures, and in that form it generalizes beyond $\mathbb{R}$.

Portmanteau theorem. For probability measures ${\mu }_k,\mu$ on a metric space, the following are equivalent:

1. $\int fd{\mu }_k\to \int fd\mu$ for all bounded continuous $f$;
2. ${\mathrm{limsup}}_k{\mu }_k(F)\le \mu (F)$ for closed $F$;
3. ${\mathrm{liminf}}_k{\mu }_k(G)\ge \mu (G)$ for open $G$;
4. ${\mu }_k(A)\to \mu (A)$ for every $A$ with $\mu (\partial A)=0$.

On $\mathbb{R}$ these are equivalent to convergence of CDFs at continuity points.

Item (1) is the definition of weak convergence ${\mu }_k\Rightarrow \mu$, and exhibits it as weak-* convergence in the dual of $C_b$ — the functional-analytic reading (normed-banach.md §Duals). Item (4) explains the continuity-point caveat of §1: boundaries are where mass can hide.

Compactness in this topology is governed by:

Definition. $\{{\mu }_i\}$ is tight if for every $\epsilon >0$ there is a compact $K$ with ${\mu }_i(K)>1-\epsilon$ for all $i$.

Prokhorov's theorem. On a Polish space, a family of probability measures is tight $⟺$ it is relatively compact for weak convergence. In particular, every tight sequence has a weakly convergent subsequence.

Tightness rules out mass escaping to infinity (${\mu }_k={\delta }_k$ on $\mathbb{R}$ is not tight, and ${\delta }_k$ has no weakly convergent subsequence). It is the standard route to existence proofs: produce a tight family, extract a limit, identify it — the strategy used for Brownian motion.

Finally, weak convergence can always be realized as a.s. convergence after a change of space:

Skorokhod representation. If ${\mu }_k\Rightarrow \mu$ on a Polish space, there exist random elements $Y_k\sim {\mu }_k$, $Y\sim \mu$ on a common probability space with $Y_k\to Y$ a.s.

For $\mathbb{R}$ this is the quantile construction of random-variables.md §5 applied to all $F_k$ at once, using a single uniform $U$. It converts distributional statements into pointwise ones and is the reason many proofs may assume a.s. convergence without loss of generality.

5. Where This Is Used

• Laws of large numbers — weak law = in probability, strong law = almost surely.
• The central limit theorem — a statement about $\Rightarrow$, proved via characteristic functions and Lévy continuity.
• Martingales — a.s. and $L^1$ convergence, and UI.
• The Lebesgue integral §8 — the same counterexamples in their measure-theoretic form.