← Back to Home

As heart sound AI grew less accurate, why did human oversight fail to preserve diagnostic performance?

A randomized experiment involving 48 participants found that requiring participants to interpret recordings independently before seeing AI recommendations did not improve oversight. The laboratory study underscores the need for verifiable safeguards in medical AI.

By SURL BioNews

When physicians disagree with diagnostic AI, should they reexamine their own judgment or the system’s reliability? An experiment in heart sound interpretation found that as simulated AI became progressively less accurate, participants’ final interpretations also deteriorated. Requiring them to answer independently first did not improve the results. The study puts the safety promise of human oversight to a more concrete test.

Published on November 13 in *npj Digital Medicine*, the study randomly assigned 48 participants to two groups. Each participant interpreted 40 digital heart sound recordings, classifying them as normal or abnormal. The paper’s abstract collectively describes the participants as physicians, but the methods section specifies that 18 were final-year medical students, while the others included resident physicians and senior physicians. Coauthor Niels van Berkel’s publications page also lists the study and its abstract.

The groups differed in when they saw the AI recommendations: one group saw the recommendations before interpreting the recordings, while the other first classified the recordings independently and rated their confidence, then saw the recommendations and decided whether to revise their answers. The researchers reduced the AI’s accuracy across four stages to 90%, 60%, 50%, and 0%. In the final stage, they also deliberately put audio and visual cues out of sync, adding signs of system malfunction.

Overall, interpretation accuracy with AI assistance fell from approximately 81% to 58%. Participants did correct some erroneous recommendations, but during the stages when AI reliability was lower, human–AI performance remained below the baseline established before participants saw AI recommendations. The requirement to answer independently first did not provide the protective effect the researchers had anticipated.

Another clue came from changes in confidence and trust: in the group that interpreted the recordings independently first, confidence in their own answers declined before their trust in the AI did, even though their independent interpretation performance had not deteriorated. The authors suggest that the system may have been perceived as having greater authority, leading people to doubt themselves first. However, this remains an explanation requiring verification and cannot be treated as an established psychological mechanism.

These findings come from a controlled laboratory experiment and cannot be directly translated into a risk of clinical misdiagnosis. Participants had no contact with patients and lacked additional information such as vital signs. The setup, in which the AI deteriorated until every answer was wrong, also cannot represent the subtle, intermittent inaccuracies that real systems may exhibit. Nor can the study establish that all workflows requiring independent assessment first are ineffective.

The practical question it raises is: what standard must successful oversight meet, and who should continuously verify it? The authors argue that, alongside clinical judgment, system cues and monitoring at the hospital and developer levels also need to be considered. The effectiveness of these approaches remains to be tested, but safety design at least needs to demonstrate whether human–AI collaboration can preserve existing interpretation ability when the system becomes inaccurate.

References

  1. npj Digital Medicine
  2. Niels van Berkel