Self-supervised and supervised learning in medical imaging classification: addressing the hidden bias of workflow design

The rise of self-supervised learning (SSL) in medical imaging holds immense potential, particularly for leveraging unlabeled data and achieving surprising performance in scenarios with limited annotations. Our study shows that comparisons with traditional supervised learning (SL) are often confounded by differences in workflows, leading to potentially biased conclusions. The SL paradigm is typically employed with a one-stage training workflow, while the typical SSL linear evaluation workflow involves a two-stage process: pre-training a backbone with a projector, followed by fine-tuning a randomly initialized task-specific classification head replacing the projector. We show that the two-stage workflow, when applied to SL, can change the trained model performance. This is especially important when selecting the appropriate paradigm for medical imaging classification where the outcomes can have a clinical impact. We experimented with four medical imaging datasets, targeting age prediction and Alzheimer's disease diagnosis from brain MRI, pneumonia diagnosis from chest RX, and diagnosis of retina with choroidal neurovascularization from optical coherence tomography. For each dataset, we imposed different configurations of assumed label availability and class frequency distribution. For each configuration, we performed 30 experiments (5 for the larger dataset) and a robust statistical analysis of the results, which show that the different workflows can alter the trained model performance. This finding suggests that the typical comparisons between SL and SSL in literature may not solely reflect the learning paradigm itself but also the workflow, which is agnostic to the paradigm. In the field of medical imaging, where model performance directly impacts clinical decision-making and patient outcomes, ensuring fair and robust comparisons is critical. By addressing this overlooked bias, our work provides actionable insights to advance reliable methodologies, paving the way for more effective and trustworthy AI-driven solutions in healthcare.