Linear Regression

This chapter will talk about using MLE to estimate the parameters in linear regression. Though the most common way nowdays is to use least square. I think it can bridge the gap between probability and regression theoretically.

Suppose we want to construct a regression model to simulate the relationship between the weight of moose and the size of its anteler.
Model: $y_i\sim N(\beta_0+\beta_1x_i,\sigma^2)$, where $y_i$ is the weight of moose and $x_i$ is the size of the anteler. Because $y_i$ is dependent on $x_i$, so $y_i$ is indepedent.
First we get likelihood function:
$$\begin{equation}\begin{split} L(\beta_0,\beta_1,\sigma^2|y_1,\dots,y_n) & = \prod\limits_{i=1}^n \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{y_i-(\beta_0+\beta_1x_i)^2}{2\sigma^2}} \\ & = \frac{1}{\sigma^n (2\pi)^{\frac{n}{2}}}e^{-\frac{\sum(y_i-\beta_0-\beta_1x_i)^2}{2\sigma^2}} \end{split}\end{equation}$$
Then get log-likelihood function
$$\begin{equation}\begin{split} l(\beta_0,\beta_1,\sigma^2) & = -\frac{n}{2}log(\sigma^2)-\frac{n}{2}log(2\pi)-\frac{\sum(y_i-\beta_0-\beta_1x_i)^2}{2\sigma^2} \end{split}\end{equation}$$
Take partial derivative and set partitals to 0
$$\begin{equation}\begin{split} \frac{\partial l}{\partial \beta_0} & = \sum 2(y_i-\beta_0-\beta_1x_i)(-1)\overset{\text{set}}=0 \\ & \Rightarrow \sum(y_i-\beta_0-\beta_1x_i)=0 \\ & \Rightarrow \sum y_i-n\beta_0-\beta_1\sum x_i = 0 \\ & \Rightarrow n\beta_0 = \sum y_i - \beta_1\sum x_i \\ & \Rightarrow \beta_0 = \bar y - \beta_1 \bar x \end{split}\end{equation}$$
which is intercept.
$$\begin{equation}\begin{split} \frac{\partial l}{\partial \beta_1} & = \sum 2(y_i-\bar y + \beta_1\bar x+\beta_1x_i)(-1) \\ & = -\frac{1}{2\sigma^2}[\sum2\lbrace (y_1-\bar y)-\beta_1(x_i-\bar x)\rbrace(x_i-\bar x)(-1)] \overset{\text{set}}=0 \\ & \Rightarrow \sum[(y_i-\bar y)(x_i-\bar x)-\beta_1\sum(x_i-\bar x)^2=0 \\ & \Rightarrow \hat \beta_1 = \frac{\sum(y_i-\bar y)(x_i-\bar x)}{\sum(x_i-\bar x)^2} \end{split}\end{equation}$$
which is slope.

If you like my article, please feel free to donate!