← Back to Home

A Change in Wording Changes the Cancer Stage: Study Reveals Instability in Medical AI Reasoning

Three language models were tested on clinical cases, and changes in prompt format alone could substantially alter staging accuracy. An examination of the models’ internal workings also found that methods that improve one model may not work for another.

By SURL BioNews

When AI assists with cancer staging, its answers should change in response to clinical evidence. Yet a new study found that changing the question format alone could cause substantial fluctuations in staging accuracy. When cases contained insufficient information, the models sometimes still assigned a definitive stage. For institutions preparing to introduce medical AI, this means that good performance during testing is not enough to demonstrate consistent performance in everyday use.

The study, published on October 31 in *npj Digital Medicine*, examined GPT-5, MedGemma-27B-Text-IT, and OpenBioLLM-Llama3-70B. According to the journal abstract, the team created 3,035 systematic perturbation tests from physician-validated oncology SOAP notes, short case vignettes, and MIMIC cases to assess whether the models could maintain consistent clinical reasoning when inputs changed. This figure represents test variants and should not be treated as a patient count.

The study reported that changes in prompt format alone produced variations of more than 50% in staging accuracy. However, the abstract did not provide the starting and ending accuracy values for each comparison or clarify whether this magnitude referred to a difference in percentage points. It therefore should not be interpreted as showing that all models experienced declines of more than 50 percentage points. Another warning sign was more direct: the models could produce definitive answers even when the clinical data were insufficient to support a judgment.

The team also used “sparse autoencoders” to examine how the models represented information internally. This analytical tool was trained on 50,000 MIMIC-IV clinical records totaling 1 billion tokens; tokens are the basic units models use to process text. These data were used to train the analytical tool, serving a different purpose from the case perturbation tests described above.

The analysis showed a hierarchy in MedGemma’s internal representations: higher-intensity features tended to encode word identity, while lower-intensity features carried contextual meaning. Information was distributed more evenly in OpenBioLLM. The study also found that these differences affected the effectiveness of retrieval and reasoning interventions, suggesting that an improvement method effective for one architecture could instead harm performance when transferred to another.

An earlier preprint of this work was updated in February, and its abstract described 355 oncology case perturbations. The journal version reported 3,035 and listed more case formats. The two versions support similar core conclusions, but their test scales should not be conflated. The journal currently provides an advance version that has passed peer review and awaits further editing. The abstract is also insufficient to determine the situations in which each model is most likely to make errors.

The researchers therefore argue that medical language models should undergo safety validation tailored to their respective architectures before deployment. These findings suggest that acceptance evaluations should examine not only overall accuracy but also consistency across different clinical record formats, missing information, and supporting measures. However, this study presents case-based tasks and analyses of model internals; it cannot be used to quantify the risk of harm in real clinical care. How to incorporate such testing into hospital procurement and regulatory requirements remains a question that must be answered before implementation.

References

  1. npj Digital Medicine
  2. PubMed Central / medRxiv