Tensors, gradients, training loops and honest neural-network evaluation.
- A tensor has a shape, dtype and device; check all three at a model boundary.
- Keep the sample dimension first and make predictions and targets have identical shapes.
- Matrix multiplication combines features; elementwise multiplication does not.
- Fit preprocessing on training data before converting the arrays to tensors.
- Autograd applies the chain rule to the operations that produced the loss.
- backward computes gradients; the optimiser changes the parameters.
- Clear old gradients before each new optimisation step.
- A decreasing training loss verifies optimisation, not generalisation.
- A multilayer perceptron alternates learned linear maps with nonlinear activations.
- Stacking linear layers without nonlinearities still represents an affine function.
- Use raw logits with BCEWithLogitsLoss or CrossEntropyLoss; apply probability transforms for reporting.
- Network capacity is a modelling choice, not evidence of better generalisation.
- A mini-batch is a subset used for an update; an epoch is a pass through the training data.
- Call train during fitting and eval during evaluation, and disable gradients during evaluation.
- Weight batch losses by batch size when calculating an epoch average.
- Validation guides development; the test set stays untouched until the procedure is fixed.
- Early stopping chooses an epoch using validation data and must restore its saved weights.
- Dropout changes training behaviour; evaluation disables it.
- AdamW weight decay is a separate parameter-shrinkage operation.
- Seeds support repeatability; several seeds and honest splits assess robustness.
- A final test result estimates a frozen procedure and must not select its settings.