Bigger Is Not Always More Accurate in Pathology AI: Major Test of 32 Foundation Models Reveals Generalization Gaps

The real challenge in digital pathology is not merely getting AI to score highly on one batch of slides, but whether it can remain accurate when used at another hospital, on a different tissue type, or for a different clinical question. A large-scale benchmark study published in *Nature Communications* compared 32 foundation models. The results showed that although pathology-specific models were generally strong, their performance varied by dataset and task, and no universally applicable champion emerged.

The study covered 41 whole-slide and image-patch tasks, using data from The Cancer Genome Atlas (TCGA), the Clinical Proteomic Tumor Analysis Consortium (CPTAC), and external and cross-domain datasets. The tasks included tumor classification, molecular subtyping, tumor staging, and biological pathway prediction. In other words, the AI models were required not only to recognize the appearance of tissue, but also to infer from stained slides information related to diagnosis, prognosis, or precision treatment.

The team divided the models into four categories: general-purpose vision, general-purpose vision-language, pathology-specific vision, and pathology-specific vision-language. In TCGA tasks, pathology-specific vision models such as Virchow2, Prov-GigaPath, H-optimus-0, and UNI frequently ranked among the leaders. Overall, models in this category outperformed pathology vision-language models and could compete with general-purpose vision models; however, the differences among the top models were often small and depended on the specific task.

Once the evaluation shifted to CPTAC and external data with different distributions, model rankings showed modest but consistently visible changes. This reflects the common problem of “domain shift” in pathology AI: even slight variations in staining procedures, scanning equipment, patient composition, and tissue sources can alter image features and cause a previously leading model to lose its advantage. Therefore, a high score on a single commonly used database cannot be directly equated with clinical reliability across hospitals.

The study also weakened the intuition that “the larger the model and the more data it has seen, the better it must be.” Neither model parameter count nor the volume of pretraining data could reliably predict performance on downstream tasks, particularly outside TCGA. Which tissues were represented in the training data, whether the cases were diverse, and how the model architecture handled pathology images may matter more than simply increasing scale; however, this comparison could not disentangle the causal effects of each factor.

A more practical clue came from combining models. The study used late fusion to integrate predictions from multiple high-performing models, improving overall performance across external datasets and different tissue types and indicating that the visual information learned by the models was complementary. The trade-offs were more complex deployment and greater computational requirements, while healthcare institutions would also need to determine how to trace and explain results when models disagree.

This study expanded from a 2025 preprint covering 31 models to a final paper comparing 32 models. The accompanying code, model data, and evaluation environment have also been made public and preserved with versioning, increasing the potential for reproducibility and subsequent testing. However, benchmark performance on retrospective public data is not prospective clinical validation. Before entering diagnostic and treatment workflows, models must still undergo independent testing at the target hospital and in the target patient population, with ongoing monitoring for data drift and the establishment of quality control, accountability, and the evidence required for regulatory oversight.