Fundamental Statistics Learning Note(19)

Intepretation of p-value

p-vale represent $p(reject\ H_O|H_O\ is\ true)$, calculated on your data. It is Not $p(H_O\ is\ true)$.

E.X. There are 1000 new cancer drugs developed. To test the effectiveness of each, suppose we have a hypothesis test that rejects $H_O$: there is no treatment effect at level $\alpha=0.5$. Suppose we can achieve $90%$ power for their test if $H_O$ is false.
Case I: none of drugs actually work

  1. For how many drugs do we expect $H_O$ to be rejected?
    Ans: according to $\alpha = 0.5$, though none of drugs acutually work, there is $5\%$ probability we could incorrectly reject $H_O$. Thus $50$ drugs are expected to be rejected.
  2. What is $p(H_O\ is\ true)$ among drugs where $H_O$ is rejected?
    Ans: It is $\frac{50}{50}=1$ because the rejected drugs are incorrectly rejected, they don’t have effects.

Case II: 100 of them actually work

  1. For how many drugs do we expect $H_O$ to be rejected?
    Ans: Here we use the fact that we can achive $90\%$ power for the test if $H_O$ is false, which means we could correctly reject $H_O$ by $90\%$ when $H_O$ is false.
    $$\begin{matrix}
    \text{} & \text{reject } H_O & \text{accept } H_O & \text{Total} \
    \text{work} & 90 & 10 & 100 \
    \text{doesn’t wrok} & 45 & 855 & 900 \
    \text{} & 135 & 865 & \text{}
    \end{matrix}$$
  2. What is $p(H_O\ is\ true)$ among drugs where $H_O$ is rejected?
    Ans: It is $\frac{45}{135}=\frac{1}{3}$
If you like my article, please feel free to donate!