Accuracy Is a Liar | Madjda Khedimi

Imagine a dataset where 95 out of 100 people are healthy. I can write you a model right now that scores 95% accuracy. Here it is.

def predict(x): return "healthy"

That is the whole model. It has no parameters, it took me four seconds, and it beats plenty of published baselines on accuracy alone. It is also completely worthless, because it never finds a single sick person, which was the entire purpose.

Why accuracy breaks

Accuracy asks one question: what fraction of predictions were correct? When one class dominates, you can win that question by ignoring the minority class entirely. And in medical data the minority class is almost always the one you care about.

What to report instead

There is no single replacement, which is exactly why people cling to accuracy. You need a few numbers together.

Sensitivity (recall): of the people who actually have the condition, how many did you catch? This is the number that matters when missing a case is expensive.
Specificity: of the healthy people, how many did you correctly leave alone? This is what stops you from alarming everyone.
Precision: when you raise a flag, how often are you right?
F1: the harmonic mean of precision and recall. One number, but it hides the tradeoff you just made.
AUC-ROC: how well the model ranks positives above negatives, across every threshold. Useful, but optimistic when classes are heavily imbalanced.
AUC-PR: the same idea, but focused on the positive class. On imbalanced data this is often the more honest curve.

Report sensitivity and specificity together, always. One without the other is half a sentence.

And show the confusion matrix

Four numbers, no interpretation required. It takes one line to produce and it immediately reveals what your model is actually doing. If a paper hides its confusion matrix, I get curious about why.

The threshold is a decision, not a default

Most classifiers output a probability, and then somebody quietly compares it with 0.5. That 0.5 is not sacred. It is a choice about how much you would rather over-flag than under-flag.

In screening, catching a case late is usually far worse than a false alarm that gets resolved by a follow-up test. So you move the threshold, you accept more false positives, and you say so explicitly in the paper. What you should not do is leave it at 0.5 without comment and pretend the question never came up.