Ordinary Least Squares Univariate Proof

The model is constructed like:

where:

• y and ε are vectors of size n

• β is a vector of size p

• X is a matrix of shape n by p

The model is fit by a minimization problem:

This is estimated as:

This line must pass through the mean and the slope of the line must be the marginal change in Y given a unit change in X. In other words, the line must pass through two points:

where:

• X‾ is the sample mean of X (estimating μ_X)

• Y‾ is the sample mean of Y (estimating μ_Y)

• s_X is the sample standard deviation of X (estimating σ_X)

• s_Y is the sample standard deviation of Y (estimating σ_Y)

• and r_XY is the sample correlation coefficient between X and Y (estimating ρ_XY)

Insert the first point into the estimation. This is quickly solved for α.

Insert the second point and the solution for α into the estimation.

This reduced form can be quickly solved for β.

Because the correlation coefficient can be expressed in terms of covariance and standard deviations...

...the solution for β can be further reduced.

Therefore, the regression line is estimated to be:

CategoryRicottone