Fundamental Statistics Learning Note (13)

Confidential Interval

  • $theta$ parameter, $x_1,x_2,\dots,x_n$ are data
  • An interval $(a,b)$ is a $1-\alpha$ confidence interval (CI) for $\theta$
    • if $p(a\leq \theta \leq b)\geq 1-\alpha$, $a$ and $b$ are function of data.

Two ways to construct CI:

  1. Approximate CIs based on MLE (large $n$)
    • Suppose $\hat \theta$ is the MLE, then
      • $\hat \theta \overset{\text{approx}}\sim N(\theta, \frac{1}{I_n(\theta)})$
      • $g(\hat \theta) \overset{\text{approx}}\sim N(g(\theta), \frac{[g’(\theta)]^2}{I_n(\theta)})$

Hence, $\frac{\hat \theta - \theta}{\sqrt{1/I_n(\theta)}} \overset{\text{approx}}\sim N(0,1)$, $P(-1.96 \leq \frac{\hat \theta - \theta}{\sqrt{1/I_n(\theta)}} \leq 1.96) = 0.95$
Equivalently, $\frac{\hat \theta - \theta}{\sqrt{1/I_n(\theta)}} \geq -1.96 \Rightarrow \hat \theta - \theta \geq -1.96\sqrt{1/I_n (\theta)} \Rightarrow \theta \leq \hat \theta + 1.96\sqrt{1/I_n(\theta)}$
$\Rightarrow p(\hat \theta - 1.96\sqrt{1/I_n(\theta)} \leq \theta \leq \hat \theta + 1.96\sqrt{1/I_n(\theta)})$ is an approximate $95\%$ CI for $\hat \theta(MLE)$ when $n$ is large, this can also be written as $\hat \theta \pm 1.96\sqrt{-\frac{[g(\hat \theta)]^2}{l’’(\hat \theta)}}$.

For binomial odds, $\frac{p}{1-p}$, $\hat Var(\frac{\hat p}{1-\hat p})=\frac{\hat p}{n(1-\hat p)^3}$, so approximate $95\%$ CI of the odds is $\frac{\hat p}{1-\hat p}\pm 1.96\sqrt{\frac{\hat p}{n(1-\hat p)^3}}$.

  1. Exact CIs based on pivots
    Definition: A pivot is a function, $Q(\theta, x_1,x_2,\dots,x_n)$ whose distribution does not depend on $\theta$.

Ex. $x\sim N(\mu, \sigma^)$ with unknown $\mu$ and $\sigma^2$, given $\frac{x-\mu}{\sigma}\sim N(0,1))$, then $Q(\mu, x_1,x_2,\dots,x_n)=\frac{x-\mu}{\sigma}$ is a pivot.

Ex. $x_1,x_2,\dots,x_n \overset{\text{iid}}\sim N(\mu,1)$ with unknown $\mu$, given $\bar x \sim N(\mu, \frac{1}{N})$, then $Q(\mu, x_1,x_2,\dots,x_n)=\frac{\bar x-\mu}{\sqrt{1/N}}$ is a pivot.

Ex. $x_1,x_2,\dots,x_n \overset{\text{iid}}\sim N(0,\sigma^2)$, then $Q(\mu, x_1,x_2,\dots,x_n)=\frac{(n-1)S^2}{\sigma^2}\sim \chi_{n-1}^2$ is a pivot.

How to make CIs from pivots?

  1. Find/show $Q(\mu, x_1,x_2,\dots,x_n)$ is a pivot
  2. Find $a_1$, $a_2$ to satisfy $p(a_1\leq Q(\mu, x_1,x_2,\dots,x_n) \leq a_2) = 1-\alpha \leftarrow$ Confidence level
  3. Rearrange to form $a\leq \theta \leq b$

Ex $x_1 \sim Expo(\beta)$, a) shown that $Q = \frac{x}{\beta}$ is a pivot, b) use the 0.025 and 0.975 quantiles of $Q$ to construct an exact $95\%$ CI for $\beta$.

a) $p(Q\leq q)=p(\frac{x_1}{\beta}\leq q)=p(x_1\leq q\beta)=\int_0^{q\beta}\frac{1}{\beta}e^{-x/\beta}dx = 1-e^{-\frac{q\beta}{\beta}}=1-e^{-q}$
so $Q$ is a pivot as the CDF doesn’t depend on $\beta$.
$Q$ has PDF, $f(q)=\frac{d}{dq}(1-e^{-q})=e^{-q}$ for $q$, so $ Q\sim Expo(1)$.

b) $p(a_1\leq Q \leq a_2)=0.95$
Here, the 0.025 quantiles satisfies $0.025 = \int_0^{a_1}e^{-q}dq = 1-e^{-a_1}\Rightarrow a_1 = -log(0.975) = 0.0253$
he 0.975 quantiles satisfies $0.975 = \int_0^{a_2}e^{-q}dq = 1-e^{-a_2}\Rightarrow a_2 = -log(0.025) = 3.69$
so $p(0.0253\leq \frac{x_1}{\beta} \leq 3.69)=0.95$
$\Rightarrow p(\frac{x_1}{3.69}\leq \beta \frac{x_1}{0.0253})=0.95$
$\Rightarrow CI = (\frac{x_1}{3.69}, \frac{x_1}{0.0253})$

Ex. $x_1,x_2,\dots,x_n \overset{\text{iid}}\sim Expo(\beta)$, $\frac{x_i}{\beta}\sim Expo(1)$
a) show that $Q(\beta,x_1,x_2,\dots,x_n)=\frac{\sum x_i}{\beta}$ is a pivot.
We know $\frac{x_1}{\beta}, \frac{x_2}{\beta},\dots, \frac{x_n}{\beta} \overset{\text{iid}}\sim Expo(1)$ and so $\frac{\sum x_i}{\beta}\sim Gamma(n,1)$ (Hint: MGF), which is a pivot since it doesn’t depend on $\beta$.

b) construct a pivot CI for $\beta$ when $c_1$ and $c_2$ satisfy $p(c_1 \leq \frac{\sum x_1}{\beta} \leq c_2)$
$$p(c_1 \leq \frac{\sum x_1}{\beta} \leq c_2) = p(\frac{\sum x_i}{c_2}\leq \beta \leq \frac{\sum x_i}{c_1})$$
so $(\frac{\sum x_i}{c_2}, \frac{\sum x_i}{c_1})$ is an exact 95% CI for $\beta$.

Two famours pivots, where $x_1,x_2,\dots,x_n \overset{\text{iid}}\sim N(\mu,\sigma^2)$ with unkonwn $\mu, \sigma^2$.
1) $\frac{\bar x - \mu}{s/\sqrt{n}} \sim t_{n-1}$ is a pivot
so if we choose $c_1$ and $c_2$ such that $p(c_1 \leq \frac{\bar x - \mu}{s/\sqrt{n}} \leq c_2)=1-\alpha$
then we get a $1-\alpha$ exact CI for $\mu$
$$p(c_1 \leq \frac{\bar x - \mu}{s/\sqrt{n}} \leq c_2)=1-\alpha \Rightarrow p(-\frac{c_2S}{\sqrt{n}} \leq \mu \leq -\frac{c_1S}{\sqrt{n}}) = 1-\alpha$$
Usually, we choose $c_1 = \frac{\alpha}{2}$ quantile of $t_{n-1}$ and $c_2 = 1- \frac{\alpha}{2}$ quantile of $t_{n-1}$
Denote $c_2 = t_{n-1,1-\frac{\alpha}{2}}$ and notice that $c_1 = -c_2$, since $t$ is symmetric
So exact $1-\alpha$ CI for $\mu$ is $(\bar x - t_{n-1,1-\frac{\alpha}{2}}\cdot \frac{S}{\sqrt{n}}, \bar x + t_{n-1,1-\frac{\alpha}{2}}\cdot \frac{S}{\sqrt{n}})$ or $\bar x \pm t_{n-1,1-\frac{\alpha}{2}}\cdot \frac{S}{\sqrt{n}}$, whic is the t-based CI for $\mu$.

2) $\frac{(n-1)S^2}{\sigma^2}\sim\chi_{n-1}^2$ is a pivot, so if we choose $c_1$, $c_2$ such that $p(c_1 \leq \frac{(n-1)S^2}{\sigma^2} \leq c_2) = 1-\alpha$, then we get the $1-\alpha$ exact CI for $\sigma^2$
$p(\frac{(n-1)S^2}{c_1}\leq \sigma^2 \leq \frac{(n-1)S^2}{c_2}) = 1-\alpha$

If you like my article, please feel free to donate!