25 min of teaching15 min of exercisesbeginner
:.2f and :>8.pathlib.Path, test whether a file exists and list the files in a directory.Suppose you have just weighed out three compounds, dissolved each in 250 mL of buffer and want to know the molar concentration of every solution. On paper that is one division per sample. In a lab with hundreds of samples, a plate reader and a deadline, it is a small program. This episode collects the handful of Python features that such programs are built from, nothing more. If you can already write a loop and a function you will find most of it familiar; the point is to agree on the idioms this course uses before NumPy and pandas arrive in the next two episodes.
You should have completed the Chapter 0 self-check, so that the notebook you are reading this in can run the cells below. Programmers with Python experience can skim to the sections on f-strings, pathlib and tracebacks.
Every value in Python has a type, and the type decides what you can do with it. Four cover almost all scientific data:
(float, int, str, bool)There is also None, Python’s way of saying “no value here”: a measurement that was not taken, a result not yet computed. You will meet its tabular cousin, NaN, in episode 1.3.
Python converts between types only when you ask it to. Dividing two integers gives a float (7 / 2 is 3.5), but adding a string to a number is an error, and "3.5" read from a file is text until you call float("3.5") on it. Most of the puzzling errors in data work come from a value that has the wrong type, so when in doubt, ask type(x).
A list is an ordered sequence; you look things up by position. It is the right container when the order means something: the samples in the order you measured them, the time points of a kinetics run.
('buffer_A', 15.1)Position 0 is the first item and -1 is the last. A dictionary (dict) maps keys to values; you look things up by name. It is the right container when you would otherwise be scanning a table with your finger:
A dictionary is the periodic table reduced to the one column you need. atomic_mass["O"] is exactly what you do when you glance at the table on the lab wall: find the symbol, read off the mass. The same shape fits any lookup you do by name: pKa[acid], extinction_coefficient[dye], density[solvent]. In episode 1.3 you will see that a pandas Series is a dictionary that also knows how to do arithmetic.
Which to use? If you would naturally say “the third one”, use a list. If you would say “the one called oxygen”, use a dict. Adding to each looks different: masses_mg.append(11.0) extends a list; atomic_mass["S"] = 32.06 adds a key to a dict.
Lists are not vectors
masses_mg * 2 does not double the masses; it gives a list of six items, the original three repeated twice. Arithmetic on whole collections is exactly what NumPy adds in episode 1.2. Until then, loop.
A slice takes a contiguous part of a list: items[start:stop] gives the items from position start up to, but not including, stop. Leaving out start means “from the beginning”; leaving out stop means “to the end”.
([5, 10, 15], [0, 5], [20, 25, 30])Python indexes from 0 and slices are half-open intervals: items[a:b] is the set of positions \(\{i : a \le i < b\}\), so it holds exactly \(b - a\) items and items[:k] followed by items[k:] partitions the list without overlap. The convention looks odd for a day and then becomes the one you want, because lengths and differences of indices agree. Mathematicians who write \(x_1, \dots, x_n\) should remember that NumPy and pandas inherit the same convention: the first row is row 0.
Strings slice the same way, which is handy for sample codes: "PLATE07_B03"[-3:] is "B03".
To convert the three masses to concentrations you loop over them. The for statement visits each item in turn:
[2.7532611712461836, 2.1759644740494033, 3.352761587565917]That loop does only one thing: build a new list from an old one. Python has a one-line form for exactly that pattern, the list comprehensionlist comprehensionA one-line way to build a list from another sequence: [f(x) for x in items if cond].:
[2.7532611712461836, 2.1759644740494033, 3.352761587565917]Read it as “the value mass / molar_mass / volume_L, for each mass in masses_mg”. A comprehension can also filter: [m for m in masses_mg if m > 10] keeps only the larger masses. When a loop body grows beyond one expression, or needs to do something other than build a list, go back to a plain for loop. Readability wins.
To walk two lists in step, use zip; to count as you go, use enumerate:
0 buffer_A 2.7532611712461836
1 buffer_B 2.1759644740494033
2 buffer_C 3.352761587565917Those numbers have far too many digits. Fixing that is what f-strings are for, two sections from now.
The concentration calculation will be needed again, in another notebook, next month. Wrap it in a function:
def molar_concentration(mass_mg, molar_mass, volume_L=0.250):
"""Return the concentration in mmol/L of `mass_mg` of a compound dissolved in `volume_L`.
`molar_mass` is in g/mol. The default volume is the 250 mL used in this lab.
"""
return mass_mg / molar_mass / volume_L
molar_concentration(12.4, 18.015), molar_concentration(12.4, 18.015, volume_L=1.0)(2.7532611712461836, 0.6883152928115459)Three habits, each cheap now and valuable later:
help(molar_concentration) prints it, and so do the tooltips in your editor.volume_L=0.250) cover the common case and still let the caller override it.volume_L=1.0) say which number is which. molar_concentration(12.4, 18.015, 1.0) works too, but a month later you will not remember whether the third number was a volume or a temperature.Mutable defaults
Never use a list or dict as a default value (def f(results=[])): the same list is shared between every call, so results from one call leak into the next. Use results=None and create the list inside the function.
An f-stringf-stringA string literal prefixed with f that embeds expressions in braces, with optional formatting such as {x:.2f}. is a string with an f in front; anything inside braces is evaluated and inserted. After the expression, a colon introduces a format specification that says how to print it:
| spec | meaning | example |
|---|---|---|
:.2f |
fixed point, 2 decimals | 2.753 |
:.3g |
3 significant figures | 2.75 |
:>8 |
right-align in 8 characters | 0.28 |
:<10 |
left-align in 10 characters | buffer_A |
:, |
thousands separator | 1,128 |
:.1% |
percentage | 37.5% |
Specs combine: {conc:>8.2f} right-aligns the number in 8 characters with 2 decimals. That is enough to print a readable table:
sample mass (mg) conc (mmol/L)
buffer_A 12.4 2.75
buffer_B 9.8 2.18
buffer_C 15.1 3.35Headers use the same widths as the rows, which is what makes the columns line up. Units go in the header, always.
What does f"{1234.5678:>12,.1f}" print? Work out the width, the separator and the decimals, then run it.
Twelve characters wide, right-aligned, one decimal, with a comma. If you guessed right, you can read any format spec you will meet in this course.
Data lives in files, and file paths are where notebooks break when they move between machines: Windows uses backslashes, macOS and Linux use forward slashes, and "data" + "/" + "penguins.csv" is fragile on both. pathlibpathlibPython’s object-oriented way to handle file paths that works on every operating system. fixes this with a Path object that you join with /:
(PosixPath('data/penguins.csv'), True)exists() is the check to run before you try to read a file: a FileNotFoundError deep inside pandas is harder to understand than a clear message from you. A Path also knows its parts (penguins_file.name, .suffix, .parent), and it can list a directory:
iterdir() yields every entry; glob("*.csv") yields only those matching a pattern. Both give Path objects, so you can chain straight into .exists(), .stat().st_size or pd.read_csv(p).
Where does a relative path start?
Path("data") is relative to the current working directory of the Python process, usually the folder the notebook was launched from. In this course every notebook runs from the repository root, so data/penguins.csv always works. If exists() returns False when you are sure the file is there, print Path.cwd() and you will usually see why.
A module is a file of Python code that someone else has written and you can use. import math makes the standard library’s maths functions available as math.sqrt, math.log10 and so on; the dotted prefix says where the name comes from. Scientific Python has three conventions you will type thousands of times:
The as np part gives the module a short alias. These three abbreviations are universal: every example on the web, every colleague’s notebook and every episode here uses them, so use them too. from pathlib import Path is the other form: it pulls one name out of a module so that you write Path(...) rather than pathlib.Path(...).
Imports go at the top of a notebook or script, once. Re-running an import cell is harmless, and the kernelkernelThe Python process that runs notebook cells and remembers your variables between them. remembers imported names until you restart it.
from module import *
Importing everything with a star fills your namespace with hundreds of names you did not ask for, some of which silently shadow your own variables (sum, min, max). Import the module, or the specific names you need.
Things will go wrong, and Python’s error report, the tracebacktracebackPython’s error report: read it from the bottom (the error) up (where it happened)., is long enough that beginners often stop reading before the useful part. Run the next cell; it is meant to fail.
--------------------------------------------------------------------------- KeyError Traceback (most recent call last) Cell In[14], line 7 3 return sum(atomic_mass[symbol] * count for symbol, count in formula.items()) 4 5 6 formula_mass({"C": 6, "H": 12, "O": 6}) # glucose: works ----> 7 formula_mass({"Na": 1, "Cl": 1}) # sodium chloride: fails Cell In[14], line 3, in formula_mass(formula) 1 def formula_mass(formula): 2 """Return the molar mass in g/mol of a formula given as {symbol: count}.""" ----> 3 return sum(atomic_mass[symbol] * count for symbol, count in formula.items()) Cell In[14], line 3, in formula_mass.<locals>.<genexpr>(.0) ----> 3 def formula_mass(formula): 4 """Return the molar mass in g/mol of a formula given as {symbol: count}.""" 5 return sum(atomic_mass[symbol] * count for symbol, count in formula.items()) KeyError: 'Na'
Read it from the bottom up:
KeyError: 'Na'. A KeyError means a dictionary was asked for a key it does not have, and the key was 'Na'. That alone is usually enough to guess the cause: atomic_mass only holds H, C, N and O.formula_mass, at atomic_mass[symbol] (the generator expression inside sum(...) even gets a frame of its own, labelled <genexpr>); the top frame is the cell that called formula_mass, with an arrow at line 7. Each frame shows the file (or cell), the line number and the line of code.The fix is to add the missing elements, or better, to make the error message say what to do. atomic_mass.get(symbol) returns None instead of raising, and a one-line if can turn that into a message naming the element. Either way, the traceback told you what went wrong (missing key), where (inside formula_mass), and why (sodium is not in the table).
The three errors you will see most
KeyError (a dict, Series or DataFrame does not have that key or column; check spelling and df.columns), TypeError (an operation applied to the wrong type, such as "3" + 1), and ValueError (the type is right but the value is not, such as float("abc") or mismatched array shapes). NameError means you used a variable before the cell that defines it ran; restart the kernel and run the notebook top to bottom.
Mass of a molecule
Using the atomic_mass dictionary and the formula_mass function above, compute the molar mass of ethanol, C₂H₆O, and of urea, CH₄N₂O. Print each with two decimals and the unit. Then add sulfur (S, 32.06 g/mol) to the dictionary and compute the molar mass of methionine, C₅H₁₁NO₂S.
ethanol 46.07 g/mol
urea 60.06 g/mol
methionine 149.21 g/molThe formula is a dict because the natural question is “how many of element X?”, a lookup by name. The loop runs over a list of (name, formula) pairs because their order is what you want to print. Adding sulfur is one assignment; after it, formula_mass works for any formula over those five elements without being edited, which is the point of separating the data (the table) from the logic (the function).
A results table
Write a function print_table(names, values, unit, decimals=2) that prints a header row and one line per sample, with names left-aligned in 12 characters and values right-aligned in 10 characters with decimals decimal places. Call it on samples and concentrations with the unit "mmol/L", then again with decimals=4. Hint: a format spec can itself contain a variable, as in f"{x:>10.{decimals}f}".
def print_table(names, values, unit, decimals=2):
"""Print `names` and `values` as two aligned columns, with `unit` in the header."""
print(f"{'sample':<12}{unit:>10}")
for name, value in zip(names, values):
print(f"{name:<12}{value:>10.{decimals}f}")
print_table(samples, concentrations, "mmol/L")
print()
print_table(samples, concentrations, "mmol/L", decimals=4)sample mmol/L
buffer_A 2.75
buffer_B 2.18
buffer_C 3.35
sample mmol/L
buffer_A 2.7533
buffer_B 2.1760
buffer_C 3.3528The nested braces {decimals} are evaluated first, so the inner spec becomes .4f before the outer one is applied. Making decimals a default argument means the common case stays short while a one-off call can still ask for more precision. Notice that the header uses the same widths as the rows; if you change one, change the other, or better, store the widths in variables.
Inventory of the data directory
Using pathlib only (no pandas yet), list every .csv file under data/, and for each print its name, its size in kB with one decimal, and the number of lines it contains (open it with p.open() and count the lines with a loop or sum(1 for _ in f)). Make the function robust: if data/ does not exist, print a clear message instead of raising.
def inventory(directory="data"):
"""Print name, size and line count of every CSV file in `directory`."""
directory = Path(directory)
if not directory.exists():
print(f"no directory called {directory} under {Path.cwd()}")
return
print(f"{'file':<16}{'size (kB)':>12}{'lines':>8}")
for p in sorted(directory.glob("*.csv")):
with p.open(encoding="utf8") as f:
n_lines = sum(1 for _ in f)
print(f"{p.name:<16}{p.stat().st_size / 1000:>12.1f}{n_lines:>8}")
inventory()
inventory("no_such_dir")file size (kB) lines
esol.csv 96.7 1129
penguins.csv 15.2 345
no directory called no_such_dir under /siteglob("*.csv") returns only matching files, so README.md is skipped. The with block closes the file when you leave it, even if an error occurs. The line count is one more than the number of rows, because the first line is the header: penguins.csv has 345 lines for 344 penguins. Checking exists() first turns a cryptic error into a message that names the directory and the working directory, which is almost always what a confused user needs to see.
Everything here is scaffolding for the next two episodes. Lists of numbers become NumPy arraysNumPy arrayA fixed-type, n-dimensional grid of numbers that supports fast whole-array arithmetic. in episode 1.2, where arithmetic on whole collections finally works and a million concentrations take a millisecond. Dictionaries grow into pandas Series and DataFramesDataFramepandas’ table type: labelled rows and columns, each column with its own data type. in episode 1.3, where the atomic-mass lookup becomes a table of penguins and later, in Chapter 5, a table of molecular descriptors. pathlib is how every dataset in the course is loaded, and the traceback-reading habit is what you will use on the first day a model refuses to fit.