Computing¶
working pillars 5, 6, 8
Making code run, run fast, and run on someone else's machine.
Pages¶
| Page | Pillar | Covers |
|---|---|---|
| HPC and parallelism | 6 | Amdahl and Gustafson, roofline, MPI patterns, surface-to-volume, SLURM |
| GPU and CUDA | 6 | Execution model, memory hierarchy, coalescing, divergence, transfer cost |
| Data and I/O at scale | 8 | Parallel I/O, chunking, compression, checkpointing, data versioning |
| Python performance | 5 | Vectorisation, temporaries, Numba, JAX, the GIL, profiling |
| C++ and Fortran | 5 | Memory order, aliasing, interop, build systems |
| MLOps and deployment | 5 | Containers, Apptainer on HPC, Kubernetes, CI/CD, serving |
The theme¶
Every page here is about data movement. The roofline says most scientific kernels are memory-bound. Coalescing is data movement within a GPU. Chunking is data movement from disk. Loop order is data movement through cache. NumPy temporaries are extra passes over memory. Halo exchange is data movement between nodes.
Compute has been cheap for a long time; moving bytes has not. When a code is slower than expected, the productive first question is almost never "how many operations?" — it is "how many bytes, and how far?"
A second theme: measure¶
Each page repeats the same instruction in its own vocabulary — profile before optimising,
time I/O separately, check arithmetic intensity, look at p99 rather than the mean, use
py-spy on the running process. Intuition about performance is unreliable in a way that
intuition about correctness is not, because the hardware behaviour that dominates is
invisible in the source.
Connections¶
- Foundations › Linear solvers — usually the runtime.
- HPC — the hands-on companion: MPI, OpenMP and CUDA code, and batch scripts.
- Toolchains — the codes this runs.
- DL › PyTorch patterns — the ML-side equivalent.