Decision Trees & Random Forests

Advanced 10 hrs 2 Concepts

Your Learning Map

📌 You already know

You know how to train and cross-validate a model with caret.

🎯 You'll learn here

Decision trees (rpart) and random forests — intuitive, powerful classifiers.

🌍 Where it's used

Credit scoring, churn and medical triage often run on tree-based models for accuracy and interpretability.

🔗 Unlocks next

Contrasts with the label-free methods in Unsupervised Learning.

Decision Trees

Concept 1

rpart — Decision Tree

rpart() builds classification and regression trees. Control complexity with cp (complexity parameter) — higher cp = simpler tree.

library(rpart); library(rpart.plot)
tree <- rpart(Species~., data=iris, method='class', control=rpart.control(cp=0.01))
rpart.plot(tree, type=4, extra=101)   # visualise
best_cp <- tree$cptable[which.min(tree$cptable[,'xerror']),'CP']
pruned <- prune(tree, cp=best_cp)     # prune to optimal size

R — Class averages a tree splits on LIVE READY

aggregate(. ~ Species, data = iris, FUN = mean)

Output below is verified. Click to run real R in your browser (first run loads ~20 MB once).

Output (verified)

     Species Sepal.Length Sepal.Width Petal.Length Petal.Width
1     setosa        5.006       3.428        1.462       0.246
2 versicolor        5.936       2.770        4.260       1.326
3  virginica        6.588       2.974        5.552       2.026

Solved Examples

Example 1 Apply the concept of rpart — Decision Tree to a sample dataset. Show at least two approaches.

# See the code example above and adapt it to your data. # Always check your output with str() and head().

Self-Assessment (2 questions)

Q1. A decision tree splits the data in order to:

Each split picks the feature/threshold that best separates the target classes.

Q2. Which R package builds a single (CART) decision tree?

rpart fits recursive-partitioning (CART) decision trees.

Concept 2

Random Forest

randomForest() builds an ensemble of trees. Key hyperparameter: mtry (number of features to try at each split). Default: sqrt(p) for classification.

library(randomForest)
rf <- randomForest(Species~., data=iris, ntree=500, importance=TRUE)
print(rf)            # OOB error estimate
varImpPlot(rf)       # which features matter most?
partialPlot(rf, iris, 'Petal.Length')  # partial dependence plot

Solved Examples

Example 1 Apply the concept of Random Forest to a sample dataset. Show at least two approaches.

# See the code example above and adapt it to your data. # Always check your output with str() and head().

Self-Assessment (2 questions)

Q1. A random forest improves on a single tree by:

A forest aggregates many de-correlated trees (bagging + random features), reducing overfitting.

Q2. Compared with a single decision tree, a random forest is usually:

Forests trade one tree's easy interpretability for higher accuracy and stability.

Machine Learning with caret Unsupervised Learning & PCA