Loss Function
A loss function is a mapping that assigns a real valued 'loss' or 'cost' to a system of equations.
Contents
Description
Given a prediction ŷ, the observable case-level difference between that and the true value is called a residual: y - ŷ. A function can be designed to take an entire system of equations, i.e. a set of observations and a model, and produce a score that represents 'loss' or 'cost' that should be minimized. Such a function is known either as a loss function or a cost function.
A loss function is conventionally notated as 𝓁 but for simplicity L will be used on this page.
Examples
The most obvious forms of loss functions are absolute loss (i.e., L(p,q) = |p-q|) and squared loss (i.e., L(p,q) = (p-q)2 = (p-q)T(p-q); sometimes called squared error loss or SEL). When applied to probabilities (i.e., predictions range from 0 to 1), squared loss is also called a Brier score. Squared loss divided by the number of observations is mean squared error.
Log loss is appropriate for classifications: L(p,q) = -p log(q) - (1-p) log(1-q). Note that it is equivalent to cross entropy.