Intepretation of p-value

p-vale represent $p(reject\ H_O|H_O\ is\ true)$, calculated on your data. It is **Not** $p(H_O\ is\ true)$.

E.X. There are 1000 new cancer drugs developed. To test the effectiveness of each, suppose we have a hypothesis test that rejects $H_O$: there is no treatment effect at level $\alpha=0.5$. Suppose we can achieve $90%$ power for their test if $H_O$ is false.

Case I: none of drugs actually work

- For how many drugs do we expect $H_O$ to be rejected?

Ans: according to $\alpha = 0.5$, though none of drugs acutually work, there is $5\%$ probability we could incorrectly reject $H_O$. Thus $50$ drugs are expected to be rejected. - What is $p(H_O\ is\ true)$ among drugs where $H_O$ is rejected?

Ans: It is $\frac{50}{50}=1$ because the rejected drugs are incorrectly rejected, they don’t have effects.

Case II: 100 of them actually work

- For how many drugs do we expect $H_O$ to be rejected?

Ans: Here we use the fact that we can achive $90\%$ power for the test if $H_O$ is false, which means we could correctly reject $H_O$ by $90\%$ when $H_O$ is false.

$$\begin{matrix}

\text{} & \text{reject } H_O & \text{accept } H_O & \text{Total} \

\text{work} & 90 & 10 & 100 \

\text{doesn’t wrok} & 45 & 855 & 900 \

\text{} & 135 & 865 & \text{}

\end{matrix}$$ - What is $p(H_O\ is\ true)$ among drugs where $H_O$ is rejected?

Ans: It is $\frac{45}{135}=\frac{1}{3}$