Chapter 3 key points

The main ideas from every episode.

3.1 Decision trees: learning a set of questions

• A tree partitions feature space with successive questions and predicts a constant within each leaf.
• Depth, leaf size and pruning control complexity; select them without consulting the final test set.
• A readable decision rule describes a fitted model, not a causal law.
• Trees need no standardisation but still require honest splitting and sensible features.

3.2 Ensembles: forests and gradient boosting

• Random forests average diverse trees; boosting adds trees sequentially to reduce a loss.
• Compare models on identical folds, features and metrics before attributing a gain to the algorithm.
• More forest trees reduce simulation noise; they do not repair leakage or an unsuitable split.
• Learning rate and boosting iterations work together and must be chosen within development data.

3.3 Support vector machines and the kernel idea

• A kernel expresses similarity; the RBF kernel makes nearby feature vectors influence each other more strongly.
• Scale features inside every training fold before fitting a distance-based kernel model.
• C controls the penalty for errors; gamma controls the locality of the RBF similarity.
• An SVM decision score is not automatically a calibrated probability.

3.4 PCA: seeing variation in many measurements

• PCA finds orthogonal directions of high input variance without using the target.
• Scale deliberately: PCA centres features but does not automatically standardise their units.
• Explained variance measures retained input variation, not predictive accuracy.
• When PCA is part of prediction, fit both scaling and PCA inside each training fold.

3.5 Clustering: useful groups without supplied labels

• A cluster depends on the representation, scaling, distance and algorithm.
• K-means minimises within-cluster squared distance and prefers compact groups.
• A high silhouette is evidence of separation under the chosen distance, not proof of a natural category.
• Cluster identifiers are arbitrary; check stability, metadata and scientific meaning before naming groups.

3.6 Interpreting models without inventing mechanisms

• Check predictive quality before interpreting a fitted model’s behaviour.
• Permutation importance measures a score drop for a particular model, dataset and metric.
• Correlated features can share importance and make hypothetical feature changes unrealistic.
• Partial dependence averages model responses; ICE reveals variation across individual rows.
• Explanations describe model behaviour and do not establish an intervention’s causal effect.