Bagging
Bagging (bootstrap aggregating) is a non-parametric ensemble model using decision trees.
Contents
Description
Bagging is an iterative ensemble model. For each iteration, a SRS with replacement of observations are selected. A decision tree model is trained on this data. (This is the bootstrapping step.)
The final prediction of the bagging model is the average prediction across all trees. (This is the aggregating step.)
This model is designed to mitigate overfitting.
Out-of-bag Error
For any single tree, the cases not selected for a single tree's estimation are called out-of-bag. These can be used as a test set to evaluate the trained model.
For each single tree, the corresponding out-of-bag set is used to calculate an error rate. The weighted average (weighted by size of the out-of-bag set) of these error rates is the out-of-bag error.