Lecture 20 – Decision Trees
by Josh Hug (Fall 2019)
Important: This lecture is taken from the Fall 2019 semester.
- The reference to Lecture 22 in 20.1 should be a reference to Lecture 19.
- The references to lec25-decision-trees.ipynb in 20.1 should be references to lec20-decision-trees.ipynb.
- The slides in 20.4 should say: “Bagging often isn’t enough to reduce model variance!” Without selecting a random subset of features at each split, the decision trees fitted on bagged data often look very similar to one another; this means that they make similar predictions. As a result, the ensemble of decision trees would still have low bias and high model variance. Selecting a random subset of features at each split helps reduce model variance by making the decision trees in the ensemble different from one another.
Decision tree basics. Decision trees in scikit-learn.
Overfitting and decision trees.
Decision tree generation. Finding the best split. Entropy and weighted entropy.
Restricting decision tree complexity. Preventing growth and pruning. Random forests and bagging.
Regression trees. Summary of decision trees, classification, and regression.