How to Get Several Mediocre Models to Cooperate

A single model has a personality. It has things it is good at and things it consistently gets wrong. Ensembles work on a simple bet: if you combine models that fail in different ways, their mistakes partly cancel out and what survives is the signal.

The important word is different. Ten copies of the same model that all fail on the same samples will give you exactly nothing except a longer training time.

Bagging

Train the same kind of model many times, each on a random subset of the data, then average their votes. Each individual model is a bit unstable, but averaging smooths that out. Random Forest is bagging applied to decision trees, and it remains an outstanding default that is very hard to embarrass.

Boosting

Train models one after another, and let each new one focus on what the previous ones got wrong. Errors get progressively mopped up. XGBoost, LightGBM, and CatBoost all live here, and on tabular data they are still extremely tough to beat. If you have a spreadsheet-shaped problem and reach straight for a neural network, boosting will often quietly outperform you while using a fraction of the compute.

Stacking

This is my favourite, because it is the one that admits it does not know the answer in advance.

Instead of averaging your models with fixed weights, you train a second-level model (a meta-learner) whose only job is to learn how to combine the first-level predictions. It works out which base model to trust, and when.

Level 0: SVM, Random Forest, XGBoost, a small neural net | each produces predictions v Level 1: meta-learner learns how to weigh them

The critical detail is how you generate the level-0 predictions that feed the meta-learner. They must come from out-of-fold predictions. If you train a base model on some data and then let the meta-learner see that same model's predictions on that same data, the base model looks far more confident than it deserves to, and the meta-learner learns to trust a liar.

Base model predictions fed to the meta-learner must be out-of-fold. This is the whole ballgame. Get it wrong and stacking becomes an elaborate way to overfit.

When not to bother

Ensembles cost you interpretability, training time, and deployment simplicity. If a single model is within a point or two of your ensemble, and a clinician has to actually understand and trust the output, ship the single model. A result that nobody trusts enough to use is worth less than a slightly worse result that gets used.