Fundamental Statistics Learning Note (12)


& \hat\theta_{MLE}\overset{\text{approx}}\sim N(\theta, \frac{1}{I_n(\theta)}) \\
& i.e. Var(\hat \theta_{MLE})\approx \frac{1}{I_n(\theta)} \\
& \hat Var(\hat \theta_{MLE}) = \frac{-1}{l’’(\hat \theta_{MLE})}

Suppose $g(\theta)$ is a function of the parameter, by MLE invaiance property, $g(\theta)$ has MLE, $g(\hat \theta_{MLE})$.
& g(\hat\theta_{MLE})\overset{\text{approx}}\sim N(g(\theta), \frac{[g’(\theta)]^2}{I_n(\theta)}) \\
& Var(g(\hat\theta_{MLE}))\approx \frac{[g’(\theta)]^2}{I_n(\theta)} \\
& \hat Var(g(\hat\theta_{MLE}))= -\frac{[g’(\hat \theta_{MLE})]^2}{l’’(\hat \theta_{MLE})}

Ex. $x_1,x_2,\dots,x_n \overset{\text{iid}}\sim Bern(p)$, recall MLE is $\hat p = \bar x$, what is MVUE of $p$?
Ans: It is $\bar x$ since $E(\bar x)=p$ and minimum sufficient statistics $\sum_{i=1}^n x_i$.
In this case, we know exactly $$Var(\hat p)= Var(\bar x)=\frac{p(1-p)}{n}$$
estimate by $$\hat Var(\hat p)= \frac{\hat p(1-\hat p)}{n}$$
Odds is $\frac{p}{1-p}$, the MLE of the odds is $\frac{\hat p}{1-\hat p} (invariance, g(p)=\frac{p}{1-p})$, however, $$ Var(\frac{\hat p}{1-\hat p}) = Var(\frac{\bar x}{1-\bar x})$$
is not known.

When n is large, $$\hat {Var}(\frac{\hat p}{1-\hat p})=-\frac{[g’(\hat p)]^2}{l’’(\hat p)} $$
L(p|x_1,x_2,\dots,x_n)&=\prod_{i=1}^n p^{x_i}(1-p)^{1-x_i} \\
&=p^{\sum x_i}(1-p)^{n-\sum x_i}
$$l(p) = \sum x_i logp+(n-\sum x_i)log(1-p) $$
$$l’(p) = \frac{\sum x_i}{p}-\frac{n-\sum x_i}{1-p} $$
$$l’’(p)=-\frac{\sum x_i}{p^2} - \frac{n-\sum x_i}{(1-p)^2} = -\frac{n\hat p}{p^2}-\frac{n-n\hat p}{(1-p)^2} $$

l’’(\hat p) &= -\frac{n\hat p}{\hat p^2}-\frac{n(1-\hat p)}{(1- \hat p)^2} \\
&= -\frac{n}{\hat p}-\frac{n}{1-\hat p} \\
&= -\frac{-n}{\hat p(1-\hat p)}

$$\Rightarrow \hat {Var}(\frac{\hat p}{1-\hat p})=\frac{[\frac{1}{(1-\hat p)^2}]^2}{\frac{n}{\hat p(1-\hat p)}}=\frac{\hat p}{n(1-\hat p)^3}$$
This is approximate Var of the odds MLE when n is large.

If you like my article, please feel free to donate!