ML Engineer · Step 5 of 8 · 8 min
Classic models that still win
You know when to reach for linear models, decision trees and gradient boosting, and why they beat deep learning on most tables.
Linear and logistic regression
Linear regression draws the best straight line through your data to predict a number, like a house price from its size. Logistic regression does the same job for yes-or-no questions and returns a probability.
They are fast, hard to break, and easy to explain: each feature gets a weight, so you can say 'each extra bedroom adds about this much'. Always try one first. It is your honest baseline.
Decision trees and random forests
A decision tree asks a series of yes-or-no questions: is income above X? Has the customer called support twice? It handles mixed data and odd patterns well, but one deep tree memorises the training data.
A random forest trains hundreds of trees on random slices of the data and lets them vote. The crowd is far more reliable than any single tree, and it needs little tuning.
Gradient boosting: the workhorse
Gradient boosting also builds many trees, but in sequence: each new tree focuses on the mistakes the earlier ones made. Libraries like XGBoost, LightGBM and CatBoost are the standard choice for tabular data, the rows-and-columns data most companies have.
On tables, a well-tuned gradient boosting model still usually beats a neural network, trains in minutes on a laptop and is cheaper to run. Deep learning wins on images, audio and text. Pick the tool for the data.
Tuning without fooling yourself
Each model has settings, called hyperparameters, such as tree depth or learning rate. Tune them with cross-validation on the training data, never on the test set. Small, honest gains are better than a big number you cannot reproduce.
Try it
Three models, one dataset
- 1.In a Colab or Kaggle notebook, load scikit-learn's breast cancer dataset (load_breast_cancer).
- 2.Ask a chatbot for code using the prompt below.
- 3.Compare the three scores. Then ask the chatbot why the gap is small or large on this dataset.
Write Python code that loads scikit-learn's breast cancer dataset, then uses 5-fold cross-validation to compare LogisticRegression (with scaling in a pipeline), RandomForestClassifier and HistGradientBoostingClassifier. Report mean recall and precision for each. Keep it short and commented.