{
 "cells": [
  {
   "cell_type": "raw",
   "id": "f297df1b",
   "metadata": {},
   "source": [
    "---\n",
    "title: \"Lab 2 solutions: predicting aqueous solubility end to end\"\n",
    "chapter: 2\n",
    "lab-solutions: true\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ac26840",
   "metadata": {},
   "source": [
    "# Lab 2: predicting aqueous solubility end to end\n",
    "\n",
    "**Goal.** Take the ESOL dataset from a raw CSV to a tuned, honestly evaluated model, using every\n",
    "habit from Chapter 2: a reproducible split, a baseline, cross-validated hyperparameter search inside\n",
    "a pipeline, a final held-out evaluation after model selection, a parity plot and an error analysis.\n",
    "\n",
    "By the end you will have:\n",
    "\n",
    "- a tidy table comparing a baseline, linear regression, ridge and k-nearest neighbours on the same test molecules;\n",
    "- a parity plot of your best model;\n",
    "- a list of the ten worst-predicted molecules and a hypothesis about *why* they are hard;\n",
    "- a short written conclusion.\n",
    "\n",
    "**How to work.** Run the notebook top to bottom. Cells marked `# TODO` are yours to complete.\n",
    "The ✅ check cells after each task confirm that you are on track; they only use what the task asked for,\n",
    "so if one fails, re-read the task. Nothing here needs a GPU or an account: the whole notebook runs on a\n",
    "laptop CPU in well under a minute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9d9a7f3",
   "metadata": {
    "tags": [
     "setup"
    ]
   },
   "outputs": [],
   "source": [
    "# Setup: works locally (course environment) and on Google Colab.\n",
    "import importlib.util\n",
    "import sys\n",
    "\n",
    "try:\n",
    "    import google.colab  # noqa: F401\n",
    "    IN_COLAB = True\n",
    "except ImportError:\n",
    "    IN_COLAB = False\n",
    "\n",
    "missing = [p for p in (\"numpy\", \"pandas\", \"matplotlib\", \"sklearn\") if importlib.util.find_spec(p) is None]\n",
    "if missing:\n",
    "    pip_names = {\"sklearn\": \"scikit-learn\"}\n",
    "    %pip install -q {\" \".join(pip_names.get(p, p) for p in missing)}\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "print(f\"Python {sys.version.split()[0]} | Colab: {IN_COLAB} | numpy {np.__version__} | pandas {pd.__version__}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "525b32b0",
   "metadata": {},
   "source": [
    "## Data\n",
    "\n",
    "ESOL (Delaney, 2004) lists 1,128 small organic molecules with their measured aqueous solubility,\n",
    "`logS` in log10(mol/L), and five whole-molecule descriptors. The file is not bundled with the course\n",
    "(its redistribution terms are unclear), so we try the local copy first and fall back to the canonical\n",
    "download. See `data/README.md` for the citation and checksum."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0074ad2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load ESOL: first candidate that works wins.\n",
    "# The raw.githubusercontent.com URL will 404 on purpose: esol.csv is git-ignored and not bundled,\n",
    "# so the loop falls through to the canonical DeepChem copy.\n",
    "CANDIDATES = [\n",
    "    \"data/esol.csv\",\n",
    "    \"../../../data/esol.csv\",\n",
    "    \"https://raw.githubusercontent.com/Awguhst/MolModels/main/data/esol.csv\",\n",
    "    \"https://deepchemdata.s3-us-west-1.amazonaws.com/datasets/delaney-processed.csv\",\n",
    "]\n",
    "\n",
    "esol = None\n",
    "for candidate in CANDIDATES:\n",
    "    try:\n",
    "        esol = pd.read_csv(candidate)\n",
    "        print(f\"loaded {len(esol)} molecules from {candidate}\")\n",
    "        break\n",
    "    except (FileNotFoundError, OSError, ValueError):\n",
    "        continue\n",
    "if esol is None:\n",
    "    raise RuntimeError(\"Could not load ESOL from any candidate location.\")\n",
    "\n",
    "esol.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b13f75c",
   "metadata": {
    "tags": [
     "check"
    ]
   },
   "outputs": [],
   "source": [
    "assert esol.shape == (1128, 10), f\"expected 1128 rows and 10 columns, got {esol.shape}\"\n",
    "assert \"measured log solubility in mols per litre\" in esol.columns\n",
    "print(\"✅ ESOL loaded: 1128 molecules, 10 columns\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7100860c",
   "metadata": {},
   "source": [
    "## Task 1: features and target\n",
    "\n",
    "Define `X` as the five descriptors used in the chapter (`Molecular Weight`, `Number of H-Bond Donors`,\n",
    "`Number of Rings`, `Number of Rotatable Bonds`, `Polar Surface Area`) and `y` as the measured logS.\n",
    "Optionally add `Minimum Degree` as a sixth feature (it is 1 for almost every molecule, so expect little\n",
    "gain). Keep the column names in a list called `features` so that later cells can reuse it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "562478a9",
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Task 1: build X (descriptors) and y (measured logS)\n",
    "# TODO: your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55196645",
   "metadata": {
    "tags": [
     "check"
    ]
   },
   "outputs": [],
   "source": [
    "assert X.shape[0] == 1128 and X.shape[1] in (5, 6), \"X should have 1128 rows and 5 (or 6) descriptor columns\"\n",
    "assert y.shape == (1128,), \"y should be a Series with one logS value per molecule\"\n",
    "assert abs(y.mean() - (-3.05)) < 0.05, \"y does not look like measured logS (mean should be about -3.05)\"\n",
    "print(\"✅ Task 1: X and y defined\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd9b3250",
   "metadata": {},
   "source": [
    "## Task 2: split once, reproducibly\n",
    "\n",
    "Split 80/20 with `train_test_split(..., test_size=0.2, random_state=42)`. Keep the test set untouched\n",
    "until the final evaluation in Task 7. Every model choice comes from the training set; the later plots\n",
    "diagnose the selected, frozen model without retuning it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c5799f8",
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Task 2: 80/20 split with random_state=42\n",
    "# TODO: your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89259507",
   "metadata": {
    "tags": [
     "check"
    ]
   },
   "outputs": [],
   "source": [
    "assert len(X_train) == 902 and len(X_test) == 226, \"expected 902 training and 226 test molecules\"\n",
    "assert X_train.shape[0] == y_train.shape[0] and X_test.shape[0] == y_test.shape[0], \"X and y sizes disagree: check the unpacking order\"\n",
    "assert X_test.index[0] == 1091, \"the split does not match random_state=42 (did you shuffle differently?)\"\n",
    "print(\"✅ Task 2: 902 / 226 split, seed 42\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "176e7b32",
   "metadata": {},
   "source": [
    "## Task 3: the baseline\n",
    "\n",
    "Fit a `DummyRegressor(strategy=\"mean\")` on the training set and write a function\n",
    "`report(model, X, y)` that returns a dictionary with `MAE`, `RMSE` and `R2` (use\n",
    "`mean_absolute_error`, `root_mean_squared_error`, `r2_score`). Save this helper for the final evaluation.\n",
    "\n",
    "For now, estimate the baseline's RMSE on the training set using 5-fold cross-validation:\n",
    "`cv = KFold(n_splits=5, shuffle=True, random_state=42)` and\n",
    "`cross_val_score(..., cv=cv, scoring=\"neg_root_mean_squared_error\")`.\n",
    "Store the mean as `baseline_cv_rmse`. Reuse this same splitter for every model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03730e5d",
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Task 3: fit the mean baseline, define report(), and estimate baseline_cv_rmse on training data\n",
    "# TODO: your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4b838f7",
   "metadata": {
    "tags": [
     "check"
    ]
   },
   "outputs": [],
   "source": [
    "assert hasattr(baseline, \"constant_\"), \"baseline must be a fitted DummyRegressor\"\n",
    "assert np.isclose(baseline.constant_.item(), y_train.mean()), \"fit the mean baseline on training targets only\"\n",
    "assert 2.0 < baseline_cv_rmse < 2.4, \"baseline CV RMSE should be about 2.2 logS\"\n",
    "print(f\"✅ Task 3: baseline training CV RMSE = {baseline_cv_rmse:.3f} logS\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8340190",
   "metadata": {},
   "source": [
    "## Task 4: linear regression\n",
    "\n",
    "Fit a plain `LinearRegression` on the training set. Also compute its 5-fold cross-validated RMSE on the\n",
    "training set with `cross_val_score(..., cv=cv, scoring=\"neg_root_mean_squared_error\")`, using\n",
    "the same `cv` splitter defined in Task 3.\n",
    "Store the fitted model as `linreg` and the mean CV RMSE as `linreg_cv_rmse`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8526973c",
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Task 4: LinearRegression, fitted on train, plus its 5-fold CV RMSE\n",
    "# TODO: your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a617038",
   "metadata": {
    "tags": [
     "check"
    ]
   },
   "outputs": [],
   "source": [
    "assert hasattr(linreg, \"coef_\"), \"linreg must be a fitted LinearRegression\"\n",
    "assert 1.1 < linreg_cv_rmse < 1.3, \"CV RMSE of linear regression should be about 1.19 logS\"\n",
    "print(\"✅ Task 4: linear regression CV RMSE ≈ 1.19\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1b685d9",
   "metadata": {},
   "source": [
    "## Task 5: ridge in a scaled pipeline, tuned by grid search\n",
    "\n",
    "Build `make_pipeline(StandardScaler(), Ridge())` and tune `alpha` with `GridSearchCV` over\n",
    "`[0.01, 0.1, 1, 10, 100, 1000]`, using the same `cv` and `scoring=\"neg_root_mean_squared_error\"`.\n",
    "Fit the search on the training set only. Store it as `ridge_search`. The pipeline parameter name is\n",
    "`ridge__alpha` (step name, two underscores, parameter)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92cbd2ae",
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Task 5: GridSearchCV over alpha for a StandardScaler + Ridge pipeline\n",
    "# TODO: your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "610fbbff",
   "metadata": {
    "tags": [
     "check"
    ]
   },
   "outputs": [],
   "source": [
    "assert hasattr(ridge_search, \"best_params_\"), \"ridge_search must be a fitted GridSearchCV\"\n",
    "assert \"ridge__alpha\" in ridge_search.best_params_, \"the grid must be over ridge__alpha\"\n",
    "assert 1.1 < -ridge_search.best_score_ < 1.3, \"best CV RMSE for ridge should be about 1.19 logS\"\n",
    "print(f\"✅ Task 5: ridge tuned, best alpha = {ridge_search.best_params_['ridge__alpha']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3a5dd37",
   "metadata": {},
   "source": [
    "Ridge barely improves on plain linear regression here: with five features and 900 molecules there is\n",
    "little overfitting to regularise away, so all small alphas give the same answer and only very large\n",
    "alphas hurt. That is itself a finding.\n",
    "\n",
    "## Task 6: k-nearest neighbours, tuned the same way\n",
    "\n",
    "Build `make_pipeline(StandardScaler(), KNeighborsRegressor())` and tune `n_neighbors` over\n",
    "`[1, 3, 5, 7, 11, 15, 21]` with the same `cv` and scoring. Store the search as `knn_search`.\n",
    "Why is the scaler essential here but almost cosmetic for ridge? (Think about what kNN computes.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "effd4bc4",
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Task 6: GridSearchCV over n_neighbors for a StandardScaler + KNeighborsRegressor pipeline\n",
    "# TODO: your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8e88fe9",
   "metadata": {
    "tags": [
     "check"
    ]
   },
   "outputs": [],
   "source": [
    "assert hasattr(knn_search, \"best_params_\"), \"knn_search must be a fitted GridSearchCV\"\n",
    "assert \"kneighborsregressor__n_neighbors\" in knn_search.best_params_, \"tune kneighborsregressor__n_neighbors\"\n",
    "assert -knn_search.best_score_ < 1.0, \"a scaled kNN should reach a CV RMSE below 1.0 logS (did you include the StandardScaler?)\"\n",
    "print(f\"✅ Task 6: kNN tuned, best k = {knn_search.best_params_['kneighborsregressor__n_neighbors']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c748e9d0",
   "metadata": {},
   "source": [
    "## Task 7: select by cross-validation, then evaluate on the test set\n",
    "\n",
    "First collect the fitted models and their mean CV RMSE on the training set. Choose `best_name` and\n",
    "`best_model` using only the lowest CV RMSE. Freeze that choice before looking at any test scores.\n",
    "\n",
    "Then collect, for each of the four models (baseline, linear, ridge, kNN), the test MAE, RMSE and R².\n",
    "Build a DataFrame `results` with one row per model and the columns `CV RMSE`, `test MAE`, `test RMSE`,\n",
    "`test R2`. These are the final held-out results, not another model-selection step.\n",
    "`GridSearchCV` refits the best pipeline on the full training set, so `ridge_search` and `knn_search`\n",
    "can be used directly as models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9289f44a",
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Task 7: choose best_name / best_model by training CV RMSE, then build the final test results table\n",
    "# TODO: your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "57f97a1c",
   "metadata": {
    "tags": [
     "check"
    ]
   },
   "outputs": [],
   "source": [
    "assert list(results.columns) == [\"CV RMSE\", \"test MAE\", \"test RMSE\", \"test R2\"], \"columns must be CV RMSE, test MAE, test RMSE, test R2\"\n",
    "assert best_name == results[\"CV RMSE\"].idxmin(), \"select the model using training CV RMSE, not test scores\"\n",
    "assert len(results) == 4, \"one row per model: baseline, linear, ridge, kNN\"\n",
    "assert results[\"test RMSE\"].min() < 1.0, \"the best model should reach a test RMSE below 1.0 logS\"\n",
    "assert (results[\"CV RMSE\"] - results[\"test RMSE\"]).abs().max() < 0.15, \"CV and test RMSE should agree to within about 0.1 for every model\"\n",
    "print(\"✅ Task 7: results table complete; CV and test agree\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a7f31d8",
   "metadata": {},
   "source": [
    "Read the table in two directions. Down the columns: every model beats the baseline, and the kNN\n",
    "model selected by training CV also performs best on this test set. Across the rows: CV and test RMSE\n",
    "differ by about 0.03 logS or less for the three learned models, and about 0.10 for the baseline.\n",
    "That is reassuring, but agreement alone does not rule\n",
    "out leakage. An unexpectedly good test score deserves investigation: leakage, an easier test set\n",
    "or sampling variation could all explain it. Do not change the chosen model using these results.\n",
    "\n",
    "## Task 8: parity plot of the selected model\n",
    "\n",
    "Use `best_model`, already selected by training CV in Task 7. Store its test predictions in `pred_best`\n",
    "and draw a parity plot: measured logS on the x axis, predicted on the y axis, with the diagonal\n",
    "`y = x`. Label the axes with units."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ced88975",
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Task 8: parity plot (measured vs predicted logS) for the model already selected by training CV\n",
    "# TODO: your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f50735ca",
   "metadata": {
    "tags": [
     "check"
    ]
   },
   "outputs": [],
   "source": [
    "assert len(pred_best) == len(y_test), \"pred_best must have one prediction per test molecule\"\n",
    "assert root_mean_squared_error(y_test, pred_best) < 1.0, \"pred_best should come from the best model (test RMSE < 1.0)\"\n",
    "print(f\"✅ Task 8: best model is '{best_name}'\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d434a1cf",
   "metadata": {},
   "source": [
    "## Task 9: the ten worst-predicted molecules\n",
    "\n",
    "Build a DataFrame `worst` with the columns `Compound ID`, `smiles`, `measured`, `predicted` and\n",
    "`residual` (measured − predicted) for the test molecules, sorted by absolute residual, and keep the\n",
    "ten largest. Then *look at the SMILES*. Do the badly predicted molecules have something in common\n",
    "that the five descriptors cannot see?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e432cf3",
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Task 9: the 10 test molecules with the largest absolute residual, with names and SMILES\n",
    "# TODO: your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a9ddb4e",
   "metadata": {
    "tags": [
     "check"
    ]
   },
   "outputs": [],
   "source": [
    "assert len(worst) == 10, \"keep exactly the ten worst molecules\"\n",
    "assert {\"Compound ID\", \"smiles\", \"measured\", \"predicted\", \"residual\"} <= set(worst.columns), \"missing columns\"\n",
    "assert worst[\"residual\"].abs().is_monotonic_decreasing, \"sort by absolute residual, largest first\"\n",
    "print(\"✅ Task 9: worst-ten table built; now read the SMILES\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bee1ae46",
   "metadata": {},
   "source": [
    "**What the solutions found.** Four of the ten worst molecules contain a phosphorus ester group\n",
    "(`P(=O)` or `P(=S)`): vamidothion, Dimecron (phosphamidon), triethyl phosphate and acephate are all\n",
    "*far more* soluble than predicted (residuals of +2.6 to +4.4 log units). The over-predicted group is\n",
    "polyhalogenated: chlorothalonil (four chlorines), hexachloro-1,3-butadiene, difluron and carbophenthion\n",
    "are *less* soluble than predicted. Neither pattern is visible to the five descriptors, which count\n",
    "rings, rotatable bonds, donors and polar surface area but know nothing about *which elements* are\n",
    "present or how lipophilic the molecule is. A lipophilicity term (logP) or a fingerprint that sees the\n",
    "phosphate and the halogens would be the next feature to add, which is exactly what Part II does.\n",
    "\n",
    "## Task 10: conclusion\n",
    "\n",
    "Complete the template below with your own numbers and observations (three to six sentences).\n",
    "\n",
    "> **Conclusion.** On a held-out test set of ___ molecules, the best model was ___ with a test RMSE of\n",
    "> ___ logS (MAE ___, R² ___), compared with ___ for the mean baseline and ___ for linear regression.\n",
    "> Cross-validation and test scores agreed to within ___, so the evaluation looks honest. The largest\n",
    "> errors were molecules containing ___, which suggests the descriptors are missing ___. A next step\n",
    "> would be ___."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d5d6578",
   "metadata": {},
   "source": [
    "**Conclusion (solutions).** The model selected by training cross-validation was a scaled\n",
    "k-nearest-neighbours regressor with $k = 7$, with a test RMSE of 0.91 logS on 226 held-out molecules (MAE 0.68, R² 0.82),\n",
    "compared with an RMSE of 2.17 for the mean baseline and 1.21 for linear regression; ridge regression\n",
    "(best $\\alpha = 10$) made little difference with these five descriptors. For the three learned models,\n",
    "cross-validation and test RMSE agreed to within about 0.03 logS, a useful consistency\n",
    "check alongside the training-only model selection. The largest errors were organophosphorus esters (under-predicted by 2.6–4.4\n",
    "log units) and polyhalogenated aromatics and alkenes (over-predicted by about 2), which suggests the\n",
    "five whole-molecule descriptors are missing element-specific and lipophilicity information. A next step\n",
    "would be to add a calculated logP and a structural fingerprint, and to re-evaluate on a scaffold split,\n",
    "since a random split can place close analogues in both training and test sets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bd719d5",
   "metadata": {},
   "source": [
    "## Optional extension (★★★): soluble or not?\n",
    "\n",
    "Many projects need a yes/no answer rather than a number. Define \"soluble\" as logS > −3 (a conventional\n",
    "cut-off used in several solubility papers, not a standard), build a scaled `LogisticRegression`\n",
    "pipeline on the same train/test split, and evaluate it with a confusion matrix, accuracy, and a ROC\n",
    "curve with its AUC. Compare the accuracy with a `DummyClassifier(strategy=\"most_frequent\")` baseline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff0036e9",
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Extension: classify soluble (logS > -3) vs insoluble with a scaled LogisticRegression\n",
    "# TODO: your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fbb1f1c1",
   "metadata": {
    "tags": [
     "solution"
    ]
   },
   "outputs": [],
   "source": [
    "# Extension: confusion matrix and ROC curve side by side\n",
    "# TODO: your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e83fd3cb",
   "metadata": {
    "tags": [
     "check"
    ]
   },
   "outputs": [],
   "source": [
    "assert hasattr(clf, \"predict_proba\"), \"clf must be a fitted classifier pipeline\"\n",
    "assert clf.score(X_test, y_test_cls) > 0.8, \"logistic regression should reach well over 80% accuracy\"\n",
    "assert auc > 0.85, \"ROC AUC should be above 0.85\"\n",
    "print(\"✅ Extension: classifier beats the majority-class baseline\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15111148",
   "metadata": {},
   "source": [
    "The classes are nearly balanced at this cut-off (about half of the molecules are \"soluble\"), so the\n",
    "majority baseline is only about 54% and the logistic model's ~88% accuracy and AUC of about 0.93 are a\n",
    "real gain. Look at the confusion matrix: are the mistakes symmetric, or does the model prefer one kind\n",
    "of error? If this were a screening cascade, which error would be more expensive?\n",
    "\n",
    "## Reflection questions\n",
    "\n",
    "1. The CV RMSE and the test RMSE agreed closely for every model. Describe one mistake in the workflow\n",
    "   (not in the model) that would have made the test RMSE *better* than the CV RMSE, and explain why\n",
    "   that would be a warning sign rather than good news.\n",
    "2. kNN beat the linear models comfortably. Episode 2.6 showed that kNN is also the model most helped\n",
    "   by duplicates and near-duplicates across the split. How would you check whether ESOL's test set\n",
    "   contains close analogues of training molecules, and what split would you use to find out how the\n",
    "   model does on genuinely new chemistry?\n",
    "3. The worst-predicted molecules shared chemical features the descriptors cannot represent. Propose\n",
    "   two new features (or one new representation) and say, for each, what kind of error you would expect\n",
    "   it to fix and what it might make worse."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}