{ "cells": [ { "cell_type": "raw", "id": "dc6b35c9", "metadata": {}, "source": [ "---\n", "title: \"Lab 1 solutions: Guided EDA of the breast cancer dataset\"\n", "chapter: 1\n", "lab-solutions: true\n", "---" ] }, { "cell_type": "markdown", "id": "01321914", "metadata": {}, "source": [ "# Lab 1: Guided EDA of the breast cancer Wisconsin dataset\n", "\n", "In this lab you run the exploratory-data-analysis checklist from episode 1.5 on a new dataset, without a tutor\n", "typing ahead of you. The data are 569 breast tumours imaged by fine-needle aspiration; for each tumour, 30\n", "numbers describe the cell nuclei in the image, and a pathologist's diagnosis (malignant or benign) is the label.\n", "Chapter 2 uses this same table to train a classifier, so the picture you build here is one you will need again.\n", "\n", "**Goals**\n", "\n", "- Inspect a new table systematically: shape, types, missing values, class balance and summary statistics.\n", "- Compare the distributions of several features between two classes with histograms, KDEs and box plots.\n", "- Read a correlation heatmap and a pairplot and say which features are redundant and which separate the classes.\n", "- Rank features by how well they separate the classes, and write a short, honest summary of what you found.\n", "\n", "**How to work**\n", "\n", "- Run the cells from top to bottom. Each task builds on the previous one.\n", "- Cells that start with `# TODO` are yours to write. The comment line states the task.\n", "- Cells marked ✅ are checks: they tell you whether you are on track. If one fails, read the message and fix the cell above it.\n", "- You will not be asked to change anything that is not marked TODO." ] }, { "cell_type": "code", "execution_count": null, "id": "68761ef3", "metadata": { "tags": [ "setup" ] }, "outputs": [], "source": [ "# Setup: works locally and on Google Colab. Installs only what is missing.\n", "import importlib.util\n", "\n", "try:\n", " import google.colab # noqa: F401\n", "\n", " IN_COLAB = True\n", "except ImportError:\n", " IN_COLAB = False\n", "\n", "REQUIRED = {\"numpy\": \"numpy\", \"pandas\": \"pandas\", \"matplotlib\": \"matplotlib\", \"seaborn\": \"seaborn\", \"sklearn\": \"scikit-learn\"}\n", "missing = [pkg for mod, pkg in REQUIRED.items() if importlib.util.find_spec(mod) is None]\n", "if IN_COLAB and missing:\n", " %pip install -q {\" \".join(missing)}\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "\n", "# Use the same class order and colours in every figure.\n", "diagnosis_order = [\"malignant\", \"benign\"]\n", "diagnosis_palette = dict(zip(diagnosis_order, sns.color_palette(\"colorblind\")[:2]))\n", "\n", "print(\"running on Colab\" if IN_COLAB else \"running locally\", \"| missing packages:\", missing or \"none\")" ] }, { "cell_type": "markdown", "id": "16b74067", "metadata": {}, "source": [ "## The data\n", "\n", "This dataset ships with scikit-learn, so there is nothing to download: `load_breast_cancer(as_frame=True)` returns\n", "a pandas DataFrame with the 30 features and an integer `target` column (0 = malignant, 1 = benign). We add a\n", "readable `diagnosis` column so that plots label themselves." ] }, { "cell_type": "code", "execution_count": null, "id": "5d3a54bd", "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import load_breast_cancer\n", "\n", "bc = load_breast_cancer(as_frame=True)\n", "df = bc.frame.assign(diagnosis=bc.frame[\"target\"].map({0: \"malignant\", 1: \"benign\"}))\n", "print(bc.target_names) # index 0 -> malignant, index 1 -> benign\n", "df.sample(5, random_state=42)" ] }, { "cell_type": "markdown", "id": "fcd0c8fd", "metadata": {}, "source": [ "**Loading the course CSVs elsewhere.** Other labs read files from the course's `data/` folder (for example the\n", "penguins). On your own machine those files sit next to the notebook or at the repository root; on Colab they do\n", "not exist, so the helper below tries the local paths first and then falls back to the raw file on GitHub. You do\n", "not need it for this lab, but run it once so that you recognise the pattern when it appears in Lab 2." ] }, { "cell_type": "code", "execution_count": null, "id": "75a7a34c", "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "\n", "def load_course_csv(name: str) -> pd.DataFrame:\n", " \"\"\"Read a course CSV from the first location that exists: next to the notebook, at the repo root, or GitHub.\"\"\"\n", " candidates = [\n", " Path(\"data\") / name,\n", " Path(\"../../../data\") / name,\n", " f\"https://raw.githubusercontent.com/Awguhst/MolModels/main/data/{name}\",\n", " ]\n", " for candidate in candidates:\n", " if isinstance(candidate, str) or candidate.exists():\n", " return pd.read_csv(candidate)\n", " raise FileNotFoundError(name)\n", "\n", "\n", "# Example (not needed below): penguins = load_course_csv(\"penguins.csv\")\n", "print(\"helper defined; first candidate path:\", Path(\"data\") / \"penguins.csv\")" ] }, { "cell_type": "markdown", "id": "cb0a4791", "metadata": {}, "source": [ "## Task 1: shape, types and missing values\n", "\n", "Start exactly as in episode 1.5: how big is the table, what type is each column, and is anything missing?\n", "Store the number of missing values (over the whole table) in a variable called `n_missing`." ] }, { "cell_type": "code", "execution_count": null, "id": "e6d6907c", "metadata": { "tags": [ "solution" ] }, "outputs": [], "source": [ "# Task 1: print df.shape, show df.dtypes, and compute n_missing = total number of missing values in df\n", "# TODO: your code here" ] }, { "cell_type": "code", "execution_count": null, "id": "0b5e5280", "metadata": { "tags": [ "check" ] }, "outputs": [], "source": [ "assert df.shape == (569, 32), \"expected 569 rows and 32 columns (30 features + target + diagnosis)\"\n", "assert n_missing == 0, \"this dataset has no missing values; check df.isna().sum().sum()\"\n", "print(\"✅ Task 1: 569 rows, 32 columns, no missing values\")" ] }, { "cell_type": "markdown", "id": "f39888f8", "metadata": {}, "source": [ "## Task 2: class balance\n", "\n", "How many tumours of each class are there? Count them with `value_counts()` and store the result in `counts`,\n", "then draw a bar chart of the counts with the object-oriented matplotlib API (`fig, ax = plt.subplots()`;\n", "`ax.bar(...)`). Label the y axis.\n", "Use `diagnosis_palette[label]` to give each diagnosis the same colour as in the later plots." ] }, { "cell_type": "code", "execution_count": null, "id": "4efd0041", "metadata": { "tags": [ "solution" ] }, "outputs": [], "source": [ "# Task 2: counts = df[\"diagnosis\"].value_counts(); then a bar chart of counts with labelled axes\n", "# TODO: your code here" ] }, { "cell_type": "code", "execution_count": null, "id": "ead0148c", "metadata": { "tags": [ "check" ] }, "outputs": [], "source": [ "assert counts[\"benign\"] == 357 and counts[\"malignant\"] == 212, \"expected 357 benign and 212 malignant\"\n", "print(f\"✅ Task 2: {counts['malignant']} malignant, {counts['benign']} benign ({counts['benign'] / len(df):.0%} benign)\")" ] }, { "cell_type": "markdown", "id": "95fe1f1b", "metadata": {}, "source": [ "About 63% of the tumours are benign. Remember this number: in Chapter 2 a classifier that always says \"benign\"\n", "will score 63% accuracy while missing every malignant case, which is why accuracy alone is never enough." ] }, { "cell_type": "markdown", "id": "9d4faaf8", "metadata": {}, "source": [ "## Task 3: summary statistics\n", "\n", "The 30 features come in three groups of ten: the `mean ...` of each nucleus measurement over the image, the\n", "`... error` (standard error) and the `worst ...` (mean of the three largest values). Build a list `mean_features`\n", "of the ten column names that start with `\"mean \"`, then show `describe()` for those columns, transposed so\n", "that one row is one feature." ] }, { "cell_type": "code", "execution_count": null, "id": "daef0b82", "metadata": { "tags": [ "solution" ] }, "outputs": [], "source": [ "# Task 3: mean_features = the ten column names starting with \"mean \"; show df[mean_features].describe().T\n", "# TODO: your code here" ] }, { "cell_type": "code", "execution_count": null, "id": "da43f0e6", "metadata": { "tags": [ "check" ] }, "outputs": [], "source": [ "assert len(mean_features) == 10, f\"expected 10 'mean' features, got {len(mean_features)}\"\n", "assert all(c.startswith(\"mean \") for c in mean_features), \"every name should start with 'mean '\"\n", "print(\"✅ Task 3:\", \", \".join(c.replace(\"mean \", \"\") for c in mean_features))" ] }, { "cell_type": "markdown", "id": "48cb40bc", "metadata": {}, "source": [ "Look at the `max` column against the `mean` column. `mean area` runs from 144 to 2501, while `mean smoothness`\n", "runs from 0.05 to 0.16: the features live on wildly different scales. That will matter for every model in\n", "Chapter 2 that measures distances, and it is why we do not put raw features of different units on one axis." ] }, { "cell_type": "markdown", "id": "5be29cec", "metadata": {}, "source": [ "## Task 4: distributions by diagnosis\n", "\n", "Draw a 2 × 2 grid of histograms with KDE curves for `mean radius`, `mean texture`, `mean smoothness` and\n", "`mean concave points`, each split by `diagnosis` (`hue`). Use `sns.histplot(..., kde=True, ax=ax)` and loop over\n", "`axes.flat`. Which of the four features separates the classes most clearly by eye?\n", "Use `hue_order=diagnosis_order` and `palette=diagnosis_palette` for consistent class colours." ] }, { "cell_type": "code", "execution_count": null, "id": "007edc3e", "metadata": { "tags": [ "solution" ] }, "outputs": [], "source": [ "# Task 4: 2x2 grid of sns.histplot(..., hue=\"diagnosis\", kde=True) for the four features below\n", "# TODO: your code here" ] }, { "cell_type": "code", "execution_count": null, "id": "7d282a3d", "metadata": { "tags": [ "check" ] }, "outputs": [], "source": [ "assert set(four) <= set(df.columns), \"one of the four feature names is misspelt\"\n", "print(\"✅ Task 4: four histograms drawn\")" ] }, { "cell_type": "markdown", "id": "f8126324", "metadata": {}, "source": [ "`mean concave points` and `mean radius` separate the classes well: the malignant distribution sits clearly to the\n", "right, with limited overlap. `mean smoothness` and especially `mean texture` overlap heavily. Keep this\n", "impression; Task 8 puts a number on it." ] }, { "cell_type": "markdown", "id": "2e7f3f3d", "metadata": {}, "source": [ "## Task 5: box plots\n", "\n", "Box plots show the same comparison more compactly. Draw a 1 × 3 row of box plots of `mean radius`,\n", "`mean area` and `mean concave points` by diagnosis (`sns.boxplot(data=df, x=\"diagnosis\", y=..., hue=\"diagnosis\",\n", "legend=False, ax=ax)`). Then compute `medians`: the median of `mean radius` per diagnosis, via `groupby`.\n", "Keep `order=diagnosis_order`, `hue_order=diagnosis_order` and `palette=diagnosis_palette`." ] }, { "cell_type": "code", "execution_count": null, "id": "f506b6bf", "metadata": { "tags": [ "solution" ] }, "outputs": [], "source": [ "# Task 5: three box plots by diagnosis, then medians = df.groupby(\"diagnosis\")[\"mean radius\"].median()\n", "# TODO: your code here" ] }, { "cell_type": "code", "execution_count": null, "id": "e07cb755", "metadata": { "tags": [ "check" ] }, "outputs": [], "source": [ "assert medians[\"malignant\"] > medians[\"benign\"], \"malignant tumours should have the larger median radius\"\n", "assert abs(medians[\"malignant\"] - 17.3) < 0.2 and abs(medians[\"benign\"] - 12.2) < 0.2, \"unexpected medians; check the groupby\"\n", "print(f\"✅ Task 5: median mean radius is {medians['malignant']:.1f} (malignant) vs {medians['benign']:.1f} (benign)\")" ] }, { "cell_type": "markdown", "id": "6bc36c5e", "metadata": {}, "source": [ "Malignant nuclei are larger (median radius about 17 versus 12) and far more irregular (concave points). Note the\n", "outliers above the benign boxes: a few benign tumours have nuclei as large as a typical malignant one, which is\n", "where any classifier will make mistakes." ] }, { "cell_type": "markdown", "id": "113cbb30", "metadata": {}, "source": [ "## Task 6: correlation heatmap\n", "\n", "Compute the Pearson correlation matrix of the ten `mean_features` and store it in `corr`. Draw it with\n", "`sns.heatmap(corr, annot=True, fmt=\".2f\", vmin=-1, vmax=1, cmap=\"vlag\")`. Shorten the tick labels by stripping\n", "the `\"mean \"` prefix (`corr.rename(columns=..., index=...)` or set the tick labels on the axes).\n", "Which pairs of features are nearly interchangeable?" ] }, { "cell_type": "code", "execution_count": null, "id": "ebd564b9", "metadata": { "tags": [ "solution" ] }, "outputs": [], "source": [ "# Task 6: corr = correlation matrix of df[mean_features]; heatmap with annot=True, fmt=\".2f\", vmin=-1, vmax=1\n", "# TODO: your code here" ] }, { "cell_type": "code", "execution_count": null, "id": "d7df5660", "metadata": { "tags": [ "check" ] }, "outputs": [], "source": [ "assert corr.shape == (10, 10), \"corr should be a 10 x 10 matrix of the 'mean' features\"\n", "assert abs(corr.loc[\"mean radius\", \"mean perimeter\"] - 1.0) < 0.01, \"radius and perimeter should correlate almost perfectly\"\n", "print(f\"✅ Task 6: radius–perimeter r = {corr.loc['mean radius', 'mean perimeter']:.3f}, radius–area r = {corr.loc['mean radius', 'mean area']:.3f}\")" ] }, { "cell_type": "markdown", "id": "010be1c4", "metadata": {}, "source": [ "Radius, perimeter and area are three measurements of one thing, size, and correlate at approximately 0.99 or above. Compactness,\n", "concavity and concave points form a second block (0.8 to 0.9). These strong correlations show substantial redundancy, although correlation alone does not\n", "determine how many features a predictive model needs." ] }, { "cell_type": "markdown", "id": "c73019e6", "metadata": {}, "source": [ "## Task 7: pairplot\n", "\n", "Draw a pairplot of `mean radius`, `mean texture`, `mean smoothness` and `mean concave points`, coloured by\n", "`diagnosis`, with `corner=True`. (`sns.pairplot` is a figure-level function: do not call `plt.subplots()` first.)\n", "Which *pair* of features separates the classes best?\n", "Reuse `hue_order=diagnosis_order` and `palette=diagnosis_palette` here too." ] }, { "cell_type": "code", "execution_count": null, "id": "ecd3159c", "metadata": { "tags": [ "solution" ] }, "outputs": [], "source": [ "# Task 7: sns.pairplot of df[four + [\"diagnosis\"]], hue=\"diagnosis\", corner=True\n", "# TODO: your code here" ] }, { "cell_type": "code", "execution_count": null, "id": "92d1c16e", "metadata": { "tags": [ "check" ] }, "outputs": [], "source": [ "print(\"✅ Task 7: pairplot drawn (if the previous cell shows a triangular grid of 10 panels)\")" ] }, { "cell_type": "markdown", "id": "8284df93", "metadata": {}, "source": [ "`mean radius` against `mean concave points` gives the cleanest split: the benign tumours bunch in the bottom-left\n", "corner (small, smooth nuclei) and the malignant ones spread up and to the right. Combining two features separates\n", "more than either alone, which is the whole idea behind the classifiers in Chapter 2." ] }, { "cell_type": "markdown", "id": "502e5c95", "metadata": {}, "source": [ "## Task 8: which features separate the classes best?\n", "\n", "Eyeballing histograms does not scale to 30 features. A simple score for how well a feature separates two groups is\n", "the **standardised mean difference**: the difference between the group means divided by the pooled standard\n", "deviation,\n", "\n", "$$d = \\frac{|\\bar{x}_{\\text{malignant}} - \\bar{x}_{\\text{benign}}|}{\\sqrt{(s^2_{\\text{malignant}} + s^2_{\\text{benign}}) / 2}} .$$\n", "\n", "A $d$ of 0 means the groups have the same mean; a $d$ above 2 means their means are more than two\n", "within-group standard deviations apart on this scale. It does not by itself guarantee little overlap.\n", "Compute $d$ for all\n", "30 numeric features (every column except `target` and `diagnosis`), store the result as a Series `separation`\n", "sorted from largest to smallest, and print the top five." ] }, { "cell_type": "code", "execution_count": null, "id": "1aa84e3b", "metadata": { "tags": [ "solution" ] }, "outputs": [], "source": [ "# Task 8: separation = standardised mean difference of every feature between the two classes, sorted descending\n", "# TODO: your code here" ] }, { "cell_type": "code", "execution_count": null, "id": "0533d8ff", "metadata": { "tags": [ "check" ] }, "outputs": [], "source": [ "assert len(separation) == 30, f\"expected one score per feature (30), got {len(separation)}\"\n", "assert separation.index[0] in (\"worst concave points\", \"worst perimeter\", \"mean concave points\", \"worst radius\", \"worst area\"), \"unexpected top feature; check the formula\"\n", "assert separation.is_monotonic_decreasing, \"sort the scores from largest to smallest\"\n", "print(\"✅ Task 8: top three features:\", \", \".join(separation.index[:3]))" ] }, { "cell_type": "markdown", "id": "bcf3f8cf", "metadata": {}, "source": [ "The top of the list is dominated by the `worst ...` versions of concave points, perimeter, radius and area, and by\n", "`mean concave points`: the most irregular and largest nuclei in the image are the most telling. At the bottom sit\n", "the `... error` features and `mean fractal dimension`, which separate the classes hardly at all.\n", "\n", "A quick cross-check with a different score: the point-biserial correlation between each feature and the 0/1 target\n", "is just `df[features].corrwith(df[\"target\"])`. Its ranking should agree with yours at the top." ] }, { "cell_type": "code", "execution_count": null, "id": "a85e3f16", "metadata": {}, "outputs": [], "source": [ "# Cross-check (nothing to write): correlation of each feature with the 0/1 target, largest magnitude first\n", "df[features].corrwith(df[\"target\"]).abs().sort_values(ascending=False).head(5).round(3)" ] }, { "cell_type": "markdown", "id": "22c04a60", "metadata": {}, "source": [ "## Task 9: a short written summary\n", "\n", "Edit the markdown cell below (double-click it) and replace each bullet with one or two sentences of your own.\n", "A good EDA summary is short, concrete and names the features it talks about.\n", "\n", "- **Size and balance:** how many samples, how many features, how balanced are the classes, anything missing?\n", "- **Scales:** are the features comparable in magnitude, or do some dwarf others?\n", "- **Redundancy:** which features say the same thing?\n", "- **Signal:** which three features separate malignant from benign best, and how cleanly?\n", "- **Caution:** where would you expect a classifier to make mistakes?" ] }, { "cell_type": "markdown", "id": "2afb6fc4", "metadata": {}, "source": [ "**Example summary (one of many good answers)**\n", "\n", "- *Size and balance:* 569 tumours and 30 numeric features, no missing values; 357 benign (63%) and 212 malignant,\n", " so a mild imbalance that any accuracy figure has to be read against.\n", "- *Scales:* areas run into the thousands while smoothness and fractal dimension are below 0.2; features must be\n", " scaled before any distance-based model.\n", "- *Redundancy:* radius, perimeter and area are one measurement in three guises (r ≈ 0.99 or higher); compactness, concavity\n", " and concave points form a second block.\n", "- *Signal:* `worst concave points`, `worst perimeter` and `mean concave points` separate the classes with $d$ above 2;\n", " the irregularity of the nucleus outline is the strongest single signal.\n", "- *Caution:* a few benign tumours have nuclei as large as malignant ones, and texture and symmetry overlap almost\n", " completely; errors will cluster among large-but-smooth benign tumours and small-but-irregular malignant ones." ] }, { "cell_type": "markdown", "id": "05d8f2fe", "metadata": {}, "source": [ "## Reflection questions\n", "\n", "1. The dataset is 63% benign. If a model predicted \"benign\" for every tumour, what would its accuracy be, and what\n", " would be wrong with reporting that number? Which single number from this lab would you quote instead?\n", "2. Radius, perimeter and area correlate at approximately 0.99 or above. Would you remove two of the three before training a\n", " model? What might you lose, and what might you gain?\n", "3. The standardised mean difference in Task 8 only sees the *means* of each class. Describe a feature that could\n", " separate the classes well yet score a low $d$, and which plot from this lab would reveal it." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.14" } }, "nbformat": 4, "nbformat_minor": 5 }