Fundamental Statistics Learning Note (15)

Hypothesis testing

Definition: A hypothesis is a statement about a parameter
Goal: Decide which of 2 complementatry hypothesis is true based on data.

  • $H_O$: null hypothesis
  • $H_A$: alternate hypothesis
    This setting originates from the experimental design.

Definition: A hypothesis test is a rule that specifies, (often) based on the value of a test statistics $W(x_1,x_n)$.
a. when to accept $H_O$ as true
b. when to reject $H_O$ and accpect $H_A$

Ex. $x_1,\dots,x_n \overset{\text{iid}}\sim N(\mu,\sigma^2)$ where $\mu$ is unknown
a. $H_O:\mu = 0$ vs $H_A:\mu \neq 0$ is an example of 2 complementary hypothesis.

Here $\bar x$ gives information whihc hypothesis is true, so $\bar x$ would be a good test statistics. Our hypothesis test is to reject $H_O$ if $\bar x > c$ or $\bar x < -c$, for some constant $c >0 $.

b. $H_O: \mu \geq 0\ vs\ H_A: \mu <0$.

If we use $\bar x $ as test statistics, we would reject $H_O\ if\ \bar x<-c\ for\ some\ c>0$.

Definition: The likelihood ratio (LR) test statistics for testing
$H_O: \theta \in \Omega_0$ vs $ H_A: \theta \in \Omega_1$, where $\Omega=\Omega_0 \cup \Omega_1$ is the parameter space of $\theta$ and $\Omega_0 \cap \Phi_1 = \Phi$ $$\lambda(x_1,\dots,x_n)=\frac{\max \limits_{\theta \in \Omega_0}L(\theta|x_1,\dots,x_n)}{\max \limits_{\theta \in \Omega}L(\theta|x_1,\dots,x_n)}$$
Note: $0\leq\lambda\leq 1$, $\lambda \approx 1$ if true value of $\theta \in \Omega_0$(ie: $H_0$ is true); $\lambda \approx 0$ if $H_0$ is false (since values $\theta \in \Omega_0$ don’t fit the data well). So accept $H_0$ if $\lambda$ is close to 1, accept $H_A$ if $\lambda$ if $\lambda$ close to $0$.

Definition: Let $w$ be a test statitiscs, suppose the hypothesis test is the rule “reject $H_o$ if $w\in \text{R}$”, then $\text{R}$ is called the rejection region.

Ex. The rejection region for a LR test is $\lbrace \lambda \leq c \rbrace$ from some $0<c<1$.
Ex. $x_1,\dots,x_n \overset{\text{iid}}\sim N(\mu,\sigma^2)$ with $\sigma^2$ known, $\mu$ is unknown parameter.
$$H_O: \mu=\mu_0, H_A: \mu\neq \mu_0$$
Derive LR test statistics $\lambda$.
L(\mu|x_1,\dots,x_n)&=\prod \limits_{i=1}^n \frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x_i-\mu)^2}{2\sigma^2}} \\
\lambda(x_1,\dots,x_n)&=\frac{\max \limits_{\theta \in \Omega_0}L(\theta|x_1,\dots,x_n)}{\max \limits_{\theta \in \Omega}L(\theta|x_1,\dots,x_n)}\\
&=\frac{\max \limits_{\mu=\mu_0}L(\theta|x_1,\dots,x_n)}{\max \limits_{\mu\in \text{R}}L(\theta|x_1,\dots,x_n)} \\
& = \frac{\frac{1}{(2\pi)^{n/2}\sigma^{n}}e^{-\frac{\sum(x_i-\mu_0)^2}{2\sigma^2}}}{\frac{1}{(2\pi)^{n/2}\sigma^{n}}e^{-\frac{\sum(x_i-\bar x)^2}{2\sigma^2}}} \\
& = \exp(-\frac{\sum(x_i-\mu_i)^2-\sum(x_i-\bar x)^2}{2\sigma^2}) \\
& = \exp(-\frac{n}{2\sigma^2}(\bar x-\mu_0)^2)
It should notice that $$\begin{equation}\begin{split}\sum(x_i - \mu_0)^2 &= \sum(x_i - \bar x+\bar x-\mu_0)^2 \\
& = \sum(x_i-\bar x)^2 + 2\sum (x_i-\bar x)(\bar x - \mu_0)+\sum(\bar x-\mu_0)^2 \\
&= \sum(x_i - \bar x)^2 + n(\bar x - \mu_0)^2

So the LR test rejects $H_O$ if $$\begin{equation}\begin{split} &e^{-\frac{n}{2\sigma^2}(\bar x-\mu_0)^2} \leq C \\
&\Rightarrow -\frac{n}{2\sigma^2}(\bar x-\mu_0)^2 \leq logC \\
&\Rightarrow (\bar x - \mu_0)^2 \geq -\frac{2\sigma^2}{n}logC \\
&\Rightarrow \bar x -\mu_0 \geq \sqrt{-\frac{2\sigma^2}{n}logC} \\
& or \ \ \bar x -\mu_0 \leq \sqrt{-\frac{2\sigma^2}{n}logC} \\
&\Rightarrow \bar x \geq \mu_0+\sqrt{-\frac{2\sigma^2}{n}logC} \\
& or \ \ \bar x \leq \mu_0-\sqrt{-\frac{2\sigma^2}{n}logC}

Based on LR test, we reject $H_O: \mu=\mu_0$ if $\bar x$ is quite different than $\mu_0$ (either larger or smaller)

Definition: $w$ is test statistics, $H_O: \theta \in \Omega_0$. A hypothesis test has level $\alpha$ if $\max \limits_{\theta\in\Omega_0} p(w\in \text{R}|\theta)=\alpha$, where $\text{R} $ the rejection region.

i.e. $\alpha=probability$ of rejecting $H_O$ when $H_O$ is true. $\alpha$ is also called type-I error rate.

Continue our example above:
Let $\mathbf{C}=-\sqrt{-\frac{2\sigma^2}{n}logC}$. What value of $\mathbf{C}$ achieves a level $\alpha$ test?
Ans: reject $H_O$ if $\bar x\leq \mu_0+\mathbf{C}$ or $\bar x \leq \mu_0-\mathbf{C}$
If $H_O$ is true, the $\mu=\mu_0$ and so $\bar x\sim N(\mu, \frac{\sigma^2}{n})\Rightarrow \frac{\bar x-\mu_0}{\sigma/\sqrt{n}}\sim N(0,1)$ under $H_O$
\alpha &=\max \limits_{\mu = \mu_0}p(\bar x\geq \mu_0+\mathbf{C}\ or\ \bar x\leq \mu_0-\mathbf{C}) \\
&=p(\frac{\bar x-\mu_0}{\sigma/\sqrt{n}} \geq \frac{\mathbf{C}}{\sigma/\sqrt{n}})+p(\frac{\bar x-\mu_0}{\sigma/\sqrt{n}} \leq \frac{-\mathbf{C}}{\sigma/\sqrt{n}})
\end{split}\end{equation}$$where $\bar x \sim N(\mu,\frac{\sigma^2}{n})$
$\Rightarrow \frac{\mathbf{C}}{\sigma/\sqrt{n}}$ is the $1-\frac{\alpha}{2}$ quantile pf $N(0,1)$.
So the final rule to achieve $\alpha$ level test is reject $H_O$ if $\frac{\bar x-\mu_0}{\sigma/\sqrt{n}}\geq Z_{1-\frac{\alpha}{2}}$ or $\frac{\bar x-\mu_0}{\sigma/\sqrt{n}}\leq -Z_{1-\frac{\alpha}{2}}$
$\Rightarrow |\frac{\bar x-\mu_0}{\sigma/\sqrt{n}}|\geq Z_{1-\frac{\alpha}{2}}$,this is the one-sample z-test for a mean.

If $x_1,\dots,x_n \overset{\text{iid}}\sim N(\mu,\sigma^2)$ with both $\mu, \sigma^2$ unknown, the LR test for $H_O:\mu=\mu_0$ vs $H_A:\mu\neq \mu_0$ yields one-sample test. HINT: in numerator of LR, we need $\max \limits_{\mu=\mu_0,\sigma^2>0}L(\mu,\sigma^2|x_1,\dots,x_n)$

i.e. substitute $\mu=\mu_0$, maximize w.r.t $\sigma^2$ alone.

Often, we would not know the exact distribution of LR test statistics $\lambda_0$. The, we can use the large sample approximation.

Theory: $H_O: \theta \in \Omega_0, H_A:\theta \in \Omega_1, \Omega = \Omega_0 \cup \Omega_1$. $\lambda=$ LR test statistics based on $x_1,\dots,x_n$. When n is large: if $H_O$ is true, then $$-2log\lambda\overset{\text{approx}}\sim \chi_p^2$$ for any $\theta \in \Omega_0$.
$p = dim(\Omega)-dim(\Omega_0)$, i.e. difference in # free parameters.

LR test rejects $H_O$ if $\lambda \leq C$, then $-2log\lambda \geq \underbrace{-2logC}_{\mathbf{C}}$
and $\alpha = p(-2log\lambda \geq \mathbf{C})$ gives the approx level $\alpha$ test.
So the approx level $\alpha$ of LR test is reject $H_O$ if $-2log\lambda\geq (1-\alpha)$ quantile of $\chi_p^2$.

If you like my article, please feel free to donate!