← Back to Home

As Auscultation AI Gradually Loses Accuracy, Can Human Review Catch Its Errors?

A heart sound interpretation experiment involving 48 participants found that as AI became less reliable, users’ diagnostic performance also declined. Requiring them to make their own judgments before seeing recommendations did not improve the results. The study shifts the safety question for clinical AI from whether human oversight exists to how it can be effective.

By SURL BioNews

If diagnostic AI starts making mistakes, can the person who reviews its results catch them in time? A cardiac auscultation study published in npj Digital Medicine raises a warning about this safeguard: as the system’s recommendations became progressively less accurate, participants’ interpretation performance also declined. Even retaining human authority over the final decision may not be enough to contain the effects of system failure.

This randomized experiment used digital heart sound interpretation as its setting, asking 48 participants to classify recordings as normal or abnormal, with abnormal classifications encompassing pathological blood flow sounds or heart murmurs. The study abstract refers to the participants collectively as physicians, but the methods section specifies that 18 were final-year medical students, while the remainder included residents and senior physicians. The results therefore reflect the performance of users with different levels of experience and cannot be directly equated with the abilities of cardiologists.

The research team arranged two workflows: one group saw the AI’s recommendation before interpreting the recording; the other had to submit its own classification before seeing the AI’s opinion and could then revise its answer. The latter approach sought to encourage users to think independently first, reducing the chance of being influenced by the machine’s answer. The authors’ publication page also confirms that this was the central design used to test whether human oversight could be effective.

The results showed that diagnostic performance worsened as AI reliability declined. Even when participants’ initial independent interpretations outperformed the AI at that point, the procedure requiring an initial judgment still did not improve the effectiveness of oversight. Another discrepancy emerged in subjective perceptions: participants’ confidence in their own judgments declined faster than their trust in the AI. This suggests that, when faced with increasingly unreliable recommendations, users may not adjust their reliance just as quickly, although this does not establish the psychological mechanisms at work in each individual.

Materials are also available for external scrutiny of the experimental procedure. The research team’s public GitHub repository provides the software used to run the experiment, participant instructions, audiovisual stimulus materials, and trial files, allowing other researchers to examine how recommendations were presented and interpretations were organized. These materials help researchers examine and reproduce the procedure, but the availability of the code does not itself mean that the study’s conclusions have been independently replicated.

This remains a small, controlled heart sound classification experiment. It did not measure outcomes of diagnosis or treatment in real patients, nor is it sufficient to infer that all medical AI will produce the same problems. For hospitals and developers, the practical issue it raises is that, beyond assigning people to review results, they must define who is responsible for detecting persistent inaccuracies, how warning signals can be obtained, and when to intervene. The study supports the need for further validation of these oversight processes; evidence from clinical settings is still needed to determine which measures are actually effective.

References

  1. Northern Sydney Local Health District Libraries, reporting npj Digital Medicine
  2. Niels van Berkel
  3. IKI-Lab on GitHub