Ch. 5: Resampling methods

Estimation of prediction error

• Validation set method

• Cross-validation

Estimation of standard error

• Bootstrap

Validation set

Schematic of validation set approach.

LOOCV

Schematic of leave-one-out cross-validation (LOOCV) set approach.

K-fold CV

Schematic of \(K\)-fold CV fold approach.

Ch. 6 Model selection and regularization

Selection

• Model score: AIC, \(C_p\), \(R^2\)

• Best subset

• Forward / backward selection

Shrinkage

• Ridge regression

• LASSO (also does selection!)

• Principal Components Regression

Visualizing Ridge and the LASSO with 2 predictors

Ridge regression of balance in the Credit dataset

A lot of pesky small coefficients throughout the regularization path

Lasso regression of balance in the Credit dataset

• Those coefficients are shrunk to zero

Ch. 7: Moving Beyond Linearity

• Basis expansion: polynomial regression

• Splines

• Local linear regression

• Generalized Additive Model (GAM): backfitting

Piecewise polynomial functions

Local linear regression

Example: Regression for Wage

• year: smoothing spline with df=4.

• age: smoothing spline with df=5.

• education: step function

Ch. 8: Tree-based methods

• Regression and classification trees: training, pruning

• Bagging and random forests

• Boosting trees

• BART (Bayesian Additive Regression Trees)

Trees and partitions

Fitted tree

Bagging

Ch. 12: Unsupervised methods

PCA (Principal Components Analysis)

• SVD of centered and scaled data matrix

• Biplot

• Screeplot to choose number of components

Clustering

• \(K\)-means

• Hierarchical clustering: different linkages, dendrograms

Biplot

Screeplot

K-means

K-means algorithm

Hierarchical clustering

Dendrograms

Complete linkage

Single linkage

Average linkage

Ch. 10: Deep learning

• Neural network to represent nonlinear functions

• Multiple hidden layers

• Convolutional Neural Networks

• Stochastic Gradient Descent / mini batches

Multi-layer network

Typical activation functions

Convolution neural networks

Convolution neural networks

Gradient descent