Medical Technology · eu
Can physicians spot when auscultation AI becomes increasingly inaccurate?
A randomized experiment involving 48 participants found that heart sound interpretation performance declined as AI reliability fell. Requiring participants to make their own judgments first did not improve overall results, highlighting the need for more concrete safety measures to support human oversight.
When medical AI makes mistakes, physicians are often expected to serve as the last line of defense. But judging whether a particular recommendation is trustworthy and recognizing that an entire system is becoming inaccurate are two different challenges. A study published on November 13 in *npj Digital Medicine* used a simulated cardiac auscultation experiment to show that human oversight may not be enough to stop progressively deteriorating diagnostic recommendations.
The study enrolled 48 physicians and final-year medical students, randomly assigning them to two groups. Each participant interpreted 40 heart sound recordings from the publicly available CirCor dataset, classifying them as normal or abnormal, with abnormalities including heart murmurs. One group saw the AI recommendation first; the other answered independently before viewing the recommendation and deciding whether to revise their answer.
The researchers deliberately manipulated the system’s outputs so that its accuracy fell sequentially to 90%, 60%, 50%, and 0%. In the final phase, they also introduced a mismatch between auditory and visual signals. This was therefore a controlled test of how people respond to a system becoming inaccurate, rather than a record of a clinical product naturally developing faults.
As the AI became less accurate, participants’ average accuracy after incorporating its recommendations fell from approximately 81% to 58%. Requiring independent interpretation first did not improve results overall. After the experiment, only 12.5% of participants spontaneously mentioned a decline in system reliability. Even when explicitly asked what had changed about the system, only 37.5% mentioned reduced reliability.
Participants could still correct some errors: during phases when AI reliability was lower, their final interpretations outperformed the AI itself but fell below the contemporaneous unaided performance of the group that answered independently first. This also highlights that assessing the effectiveness of oversight requires asking not only whether people outperform a faulty system, but also whether adding AI preserves their existing capabilities.
The findings have limited generalizability. This was a small experiment without real patient consultations or complete clinical information. The rapid deterioration in a fixed sequence, combined with the audiovisual mismatch in the final phase, also makes it difficult to disentangle the effects of individual factors. The findings cannot be directly extrapolated to estimate the risk of clinical misdiagnosis or patient harm.
On this basis, the authors argue that responsibility for safety needs to extend to hospitals and developers, combining system monitoring with clinical judgment. Validating medical AI requires more than measuring how accurately a model makes judgments. It also requires examining whether users and organizations can recognize and intervene in time when it begins to become inaccurate.