Degrees of Freedom
Degrees of freedom is a measure of how much an estimate is able to vary.
Contents
Variance
Variance is a function of independent observations and the independent mean.
Given a random sample of n observations (xi) from a larger unknown population (X), the true population's mean (μ) can be estimated using the sample mean.
With the mean estimated from the sample, the sample has lost a degree of freedom. This can be imagined as the mean losing independence from the sample. More accurately, the sample mean has fixed one of the observations and caused it to lose independence. The first n - 1 observations are allowed to vary, but the nth observation is fixed at whatever value enables the sample mean to remain true.
The consequence is that statistics making use of the sample mean must deduct 1 from the sample size. The use of n - 1 in estimation of variance is known as Bessel's correction.
Regression
A regression is a (frequently but not necessarily linear) model in terms of variables that minimizes an error term. Consider OLS:
This model describes (1) the mean observation and (2) the marginal changes to a dependent variable per unit changes in independent variables, given a standard error term on each variable.
Intuitively consider:
Given a model of 2 variables (an independent x and a dependent y) and 2 observations, a line can be drawn in terms of those 2 variables but there can be no error; the line will directly connect those 2 observations.
- Given a model of 3 variables and 3 observations, a plane can be drawn but there can be no error; the plane will directly connect those 3 observations.
For the independent variables to be able to independently vary, there must be more degrees of freedom.
The sample mean deducts 1 degree of freedom, and every independent variable's standard error also deducts 1 degree of freedom. The degrees of freedom on a regression are n - k - 1.