Training in practice

You rarely compute the gradient on the whole dataset at once. You take a small random batch, say 32 or 128 examples, compute the loss on it, and update the weights. This is stochastic gradient descent. The noise in each batch actually helps training find good solutions.

One pass over the whole training set is an epoch. Small models may train for dozens of epochs. Very large language models often see most of their data only once.

Split your data into training, validation and test sets. The network learns from training data. You use validation data to choose settings and decide when to stop. You touch the test set once, at the end, to report how good the model really is.

If you keep tuning against the test set, it quietly becomes a second validation set and your final number is too optimistic.

Overfitting is when training loss keeps falling but validation loss starts rising. The network is memorising the training examples instead of learning the pattern. Plot both curves every time you train. That plot is your main debugging tool.

The usual fixes: get more data, or make more of it with augmentation (flips, crops, small colour changes for images). Add dropout, which randomly switches off neurons during training. Add weight decay, which keeps weights small. Stop early when validation loss stops improving. Or use a smaller model.

Neural networks are mostly large matrix multiplications. A GPU does thousands of these small sums in parallel, so training that takes a day on a laptop CPU can take minutes on a GPU.

Memory is the usual limit. If you get an out-of-memory error, lower the batch size first. Mixed precision, which stores numbers in 16 bits instead of 32, roughly halves memory and is one line in modern frameworks.