← Back to Home

A single safety reminder can reduce clinical AI errors, but nearly one in ten choices remain dangerous

When asked to skip follow-up tests or stop medication early without sufficient justification, will AI simply comply? A study involving more than 10 million responses shows that brief reminders can reduce potentially harmful choices, but do not establish that the systems are ready for safe use in care.

By SURL BioNews

Skipping necessary follow-up blood tests to reduce workload, or stopping antibiotics early without sufficient clinical justification, are dangerous requests that medical AI may encounter. As language models begin assisting with care, the safety test is not only whether they understand medical knowledge, but also whether they can recognize risks in instructions and refuse to comply.

On October 8, the Icahn School of Medicine at Mount Sinai announced a study testing whether adding brief safety reminders to instructions could change models’ choices. According to the institution’s announcement and the same press release published by Newswise, 19 of the 20 models tested reduced their potentially harmful choices. The study was published in Communications Medicine on September 26, but its results measure responses in simulated scenarios, rather than the proportion of patients actually harmed.

The team used 501 simulated scenarios derived from 50 sets of templates, along with 100 cases adapted from de-identified inpatient discharge records, collecting more than 10 million responses in total. Models had to answer by choosing from options that included safe and unsafe actions. The study also tested three brief safety reminders, repeating each combination of conditions 10 times and randomizing the order of answer options to reduce interference from individual responses or option placement.

Without reminders, models selected potentially harmful actions 16.6% of the time; with reminders, this fell to 10.1%, a difference of 6.5 percentage points. This shows that a short passage asking models to check safety, verify, or refer dangerous requests to appropriate personnel may indeed change the direction of their responses. However, even after the improvement, about one in ten responses still fell into the potentially harmful category, leaving a clear gap before reliable clinical assistance can be achieved.

According to the journal abstract, cases adapted from hospital records were particularly challenging: potentially harmful choices fell from 46.6% to 24.5%, remaining close to one-quarter even with reminders. These results remind developers that better model performance in simplified scenarios cannot be used to directly infer equal reliability in tasks more closely resembling medical records. Differences in the design of the two types of cases also limit direct comparisons between the rates.

More than 10 million responses allowed the study to repeatedly examine model behavior, but this does not mean it covered more than 10 million patients or an equivalent number of independent cases. The test used predefined options and did not track how healthcare professionals adopted the responses, subsequent actions, or patient outcomes. It therefore cannot be used to estimate clinical risks after deployment, much less treat the rate of incorrect option selection as the rate of medical incidents.

The research team emphasized that safety reminders cannot replace clinical review and proposed repeated automated safety testing during model development and updates. For healthcare institutions, the study offers a safeguard that can be tested, while leaving a more practical question: whether AI can consistently recognize dangerous requests when workload pressures, medical record context, and multi-turn conversations intersect still needs to be validated in settings close to real-world care.

References

  1. Icahn School of Medicine at Mount Sinai
  2. Mount Sinai Health System via Newswise