Your Model Is Probably Cheating

There is a specific kind of joy in seeing 99% accuracy appear in your terminal. There is a specific kind of dread in realising, two weeks later, why it appeared.

Almost every time a result looks too good, the cause is the same. Information from your test set found a way into training. This is called leakage, and it is sneaky because nothing crashes and nothing warns you. The code runs. The number is beautiful. The model is useless.

The classic: scaling before splitting

This is the one that catches everybody at least once.

scaler.fit(X) # sees every sample, including test X = scaler.transform(X) X_train, X_test = split(X) # too late, the damage is done

When you fit the scaler on the full dataset, it learns the mean and variance of your test data. That information is now baked into your training features. Your model gets a small, invisible peek at the answers.

The fix is to fit only on training data, then apply that same fitted transform to the test set. A pipeline does this for you and is much harder to get wrong.

X_train, X_test = split(X) scaler.fit(X_train) # learns from training only X_train = scaler.transform(X_train) X_test = scaler.transform(X_test)

The subtle one: the same person in both sets

This is the version that matters most in medical data, and it is much easier to miss.

Suppose you have 40 participants and each one contributes 10 recordings. That is 400 samples. If you shuffle those 400 samples and split them randomly, the same person's recordings end up on both sides of the split. Your model does not have to learn anything about the condition you care about. It can just learn to recognise voices, or handwriting, or a particular sensor's quirks.

If a single participant appears in both training and test, you are not measuring whether your model generalises to new people. You are measuring whether it can recognise people it has already met.

The fix is to split by participant, not by sample. Every recording from person 17 goes to exactly one side. In scikit-learn this is GroupKFold or StratifiedGroupKFold, with the participant ID as the group.

The one nobody talks about: tuning on your test set

You try a model, check test accuracy, adjust a hyperparameter, check again, adjust again. After thirty rounds of this you have a great test score. You have also, slowly and politely, fitted your own decisions to the test set.

The test set is supposed to be used once, at the end. Everything before that should happen on a validation split. In practice I keep a holdout that I genuinely do not look at until the paper is almost written. It is uncomfortable. That discomfort is the point.

A quick checklist

Did any preprocessing step see the test data before the split?
Can one participant, patient, or session appear on both sides?
Did I do feature selection on the whole dataset?
Is my test set influencing decisions it should not be influencing?
Is the result suspiciously good compared with published work on similar data?

That last one deserves more respect than it usually gets. If your number is far above what the literature reports on comparable data, the most likely explanation is not that you are a genius. It is that something leaked. I say this with love, and from experience.