1.5 Central Limit Theorem
Definition 1.17 (Convergence in distribution) Let \(\{X_i\}_{i \in \mathbb{N}}\) be a sequence of RVs with CDF \(F_i\). Let \(X\) be a RV with CDF \(F\). We say that \(X_n\) convergence to \(X\) in distribution if \[ \lim_{n\to \infty} F_n (x) = F(x) \] at every point at which \(F\) is continuous.
Theorem 1.15 (Continuity theorems) Let \(X_i\), \(i \in \mathbb{N}\) RVs with CDF \(F_i\) and \(X\) a RV with CDF \(\bar F\). Suppose that either
\(\varphi_{X_n}(s)\) converges to \(\varphi_X(s)\) for all \(s\) in some open interval around \(0\).
\(\lim_{n\to\infty} \phi_{X_n}(s) = \phi_X(s)\) for every \(s \in \mathbb{R}\).
Then \(X_n \to X\) in distribution.
Theorem 1.16 (Central limit theorem) Let \(\{ X_i \}_{i\in \mathbb{N}}\) be a sequence of IID RVs with mean \(0\) and variance \(\sigma^2 < \infty\). Define \[ Z_n = \frac{\sum_{i=1}^n X_i}{\sigma \sqrt{n}}. \] Then \(Z_n\) converges to \(Z \sim N(0,1)\) in distribution.
There are a few ways to go about proving this theorem. Two most common ways employ the MGF and the characteristic function. Both methods rely on the one crucial idea of using the Taylor expansion, which we will see shortly. We present the proof using MGF (adopted from Rice’s book) and leave it to the reader the proof using characteristic function.
Proof. For each \(n \in \mathbb{N}\), we have that \[ \varphi_{Z_n} (t) = \left( \varphi_{X_1} \left( \frac{t}{\sigma \sqrt{n}} \right) \right)^n. \]
Observe first that \(\varphi_{X_1}'(0) = \mathbb{E}X_1 = 0\) and \(\varphi_{X_1}''(0) = \mathbb{E}X_1^2 = \sigma^2\). So, performing Taylor expansion for \(\varphi_{X_1}\), we get \[ \begin{aligned} \varphi_{X_1} \left( \frac{t}{\sigma \sqrt{n}} \right) &= 1 + \varphi_{X_1}'(0) \left( \frac{t}{\sigma \sqrt{n}} \right) + \frac{1}{2} \varphi_{X_1}''(0) \left( \frac{t}{\sigma \sqrt{n}} \right)^2 + \epsilon_n \\ & = 1 + \frac{1}{2} \sigma^2 \left( \frac{t}{\sigma \sqrt{n}} \right)^2 + \epsilon_n \\ & = 1 + \frac{1}{2} \left( \frac{t^2}{n} \right) + \epsilon_n \,. \end{aligned}\] where \(\epsilon_n / (t^2 / (n\sigma^2)) \to 0\) as \(n \to \infty\). It can then be shown that \[ \lim_{n\to \infty} \varphi_{Z_n}(t) = \lim_{n\to \infty} \left( 1 + \frac{1}{2} \left( \frac{t^2}{n} \right) + \epsilon_n \right) = e^{t^2/2} = \varphi_Z .\] Combine this with Theorem 1.15, we arrive at our result.
Exercise 1.23 Prove the central limit theorem using the characteristic function.