Transformers Are Not Magic on Small Data

Vision Transformers rewrote computer vision. They also came with a footnote that gets skipped a lot: they were trained on enormous datasets. When you bring one to a medical imaging problem with a few hundred samples, that footnote becomes the main story.

What a CNN assumes for free

A convolutional network has strong assumptions baked into its architecture. It assumes that nearby pixels are related, and that a feature is still the same feature if it moves somewhere else in the image. These assumptions are hard-coded, which means the model does not have to spend data learning them.

A Vision Transformer assumes almost none of that. It cuts the image into patches and lets attention figure out which patches relate to which. That freedom is exactly why it can outperform a CNN, because it is not constrained by assumptions that might be wrong. It is also exactly why it is hungry, because every relationship it does not assume, it has to learn from examples.

What this means in practice

On a small dataset you have three realistic options.

• Use a pretrained backbone. Do not train a ViT from scratch on 500 images. Take one pretrained on a large corpus and fine-tune it. You are borrowing all the data you do not have.
• Try a CNN first, honestly. Not as a strawman baseline you plan to beat, but as a real candidate. On small medical datasets a well-tuned ResNet is frequently the thing to beat, and sometimes it just wins.
• Consider a hybrid. Use the network as a feature extractor and put a simple classifier on top. With little data, a strong representation plus a simple decision rule can outperform an end-to-end model that has too many parameters and too few examples.

The augmentation surprise

Here is the part I did not expect when I started, and it is the reason I now test rather than assume.

Augmentation is supposed to be free performance. Rotate, flip, shift, and you have more data. But augmentation is not neutral. Every transformation makes a claim about what should not change the label. Flipping a photo of a cat gives you a cat. Flipping a medical drawing might destroy the exact directional information the diagnosis depends on.

So an augmentation that helps one architecture can actively hurt another, depending on what that architecture was relying on. Augmentation has to be matched to the model and to the data, not applied as a reflex.

Test it. Report what happened, including when it went the wrong way. A negative result that is clearly explained is more useful to the next person than another paper reporting that everything worked perfectly.