Medical Technology · eu
As Diagnostic AI Becomes Less Accurate, Can Human Oversight Catch Its Errors? Heart Sound Experiment Reveals Limits
An experiment involving 48 participants interpreting heart sounds found that human–AI accuracy declined as AI recommendations became less accurate. Even requiring participants to make their own judgments first did not consistently preserve performance. The findings raise concerns about human oversight of medical AI, but further validation in real clinical settings is still needed.
Having humans make the final call when medical AI makes mistakes seems like an intuitive safeguard. But when a system is initially presented as highly accurate and then gradually becomes less accurate, can users notice in time? A randomized experiment published on November 13 in *npj Digital Medicine* found that although participants could correct some erroneous recommendations, their final interpretation performance still declined as AI reliability deteriorated. Even making an independent judgment before reviewing the AI recommendation did not provide consistent protection.
The study focused on cardiac auscultation: identifying recorded heart sounds as normal or abnormal, with abnormalities including pathological blood flow sounds and heart murmurs, rather than making a complete diagnosis of heart disease. Of the 48 participants, 18 were final-year medical students, 19 were resident physicians, and 11 were more senior physicians. Each participant interpreted 40 heart sound recordings. Participants were divided into two groups of 24: one reviewed the AI recommendation first, while the other recorded an independent judgment before reviewing the recommendation and deciding whether to revise it.
To control the degree of system inaccuracy, the research team manually modified the AI recommendations so that accuracy across four successive phases was 90%, 60%, 50%, and 0%. Participants’ average interpretation accuracy with AI assistance fell from 81.45% in the first phase to 58.33% in the final phase. Although human–AI performance exceeded the AI’s accuracy in the last three phases, it was lower than the performance of the independent-assessment-first group before reviewing the recommendations, suggesting that human intervention could offset only part of the loss.
Requiring independent thinking first was intended to prevent the AI answer from becoming the starting point for judgment, but no overall improvement was observed in this experiment. After the experiment, only 12.5% of participants mentioned in their responses that they had noticed a decline in system reliability. Even when explicitly asked what had changed about the system, the proportion rose to only 37.5%. This highlights a gap between two things: correcting an individual error does not necessarily mean recognizing that the system as a whole is malfunctioning.
The researchers suggested that users might first question their own judgment when they disagree with AI. In the independent-assessment-first group, performance without AI assistance remained stable, but confidence declined earlier; trust in AI showed a significant decline only in later phases. However, this remains a possible explanation for the observations and is insufficient to establish the underlying psychological mechanism.
The study was registered on AsPredicted in October 2023, with a statement at the time that no data had yet been collected and a plan to recruit 48 participants to compare the order in which AI recommendations were presented, different accuracy levels, and whether participants noticed deterioration. The public record allows the original plan to be checked, but the accuracy settings for the first two phases differed between the preregistration and the published paper. Interpretation should therefore distinguish between the planned design and what was actually implemented.
The experiment also has clear limitations: it used recordings and controlled recommendations, without real patient–physician interactions or complete clinical information. The degree of inaccuracy increased in a fixed sequence, and the final phase also introduced a mismatch between auditory and visual signals. As a result, order effects cannot be ruled out, and the effects of the various sources of interference in the final phase are difficult to disentangle. These findings cannot be directly translated into clinical misdiagnosis rates, nor can they represent the performance of all medical AI systems.
For the deployment and regulation of medical AI, the study raises a question: what standard should human oversight meet, and how can its effectiveness be demonstrated? The authors suggested considering system alerts and continuous monitoring at the hospital and developer levels, alongside clinicians’ judgment. Whether these arrangements can provide a more reliable safeguard still needs to be tested in real clinical workflows.