Evaluation done right

A model that scores well on its own training data has proved nothing. It may have memorised the answers. So you hold some data back: a training set to learn from, a validation set to tune on, and a test set you look at only once, at the end.

Cross-validation is a more careful version: split the data into five parts, train on four and test on the fifth, five times over. The average score is more trustworthy than one lucky split.

Leakage happens when information about the answer sneaks into the features. A model predicting hospital readmission that uses 'number of follow-up visits' is cheating: those visits happen after the event. It will look brilliant in testing and fail in real life.

Time is the most common source. If you predict the future, split by date: train on the past, test on later months. A random split lets the model peek at the future. Also fit cleaning steps, like scaling, on training data only.

If 1 in 100 transactions is fraud, a model that always says 'not fraud' is 99% accurate and completely useless. For rare events, look at precision (of the cases you flagged, how many were real?) and recall (of the real cases, how many did you catch?).

Which matters more depends on the cost of each mistake. Missing a cancer case is worse than a false alarm, so you favour recall. Blocking a good customer's card is costly too, so a bank balances both. For numbers, like prices, use error measures such as mean absolute error, in the same units as the target.

Scores hide patterns. Open a sample of wrong predictions and read them. You will often find a labelling error, a missing feature, or a group the model handles badly. Ten minutes of reading beats an hour of tuning.