← Back to Home

More Accurate Diagnoses, More False Positives: The Clinical Trade-Offs of Reasoning AI

A study spanning 12 specialties found that reasoning AI was better at identifying primary diagnoses but also listed more incorrect diseases. Physician verification could improve performance, but gains on retrospective medical records still need validation in real clinical care.

By SURL BioNews

In the clinic, considering one more disease may prevent a missed diagnosis, but it may also lead a patient toward unnecessary tests and treatment. A study published on October 29 in Nature Communications reveals this dilemma for reasoning artificial intelligence: models more often identified the correct primary diagnosis, yet were also more likely to add incorrect diagnoses. For clinical use, the ability to get the answer right and the ability to control false positives must be assessed together.

The research team developed AIMS-LDR, a diagnostic benchmark based on long medical records, to compare six matched pairs of reasoning and general large language models across 12 specialties. According to i10X’s coverage of the same study, the data included 1,157 hospital admissions from MIMIC-IV and 1,821 cases from eICU, totaling 2,978 records. The models were required to make diagnostic judgments based on information such as medical histories, vital signs, laboratory results, and imaging reports.

The LDR-MIMIC results cited by i10X make the trade-off concrete: primary diagnosis accuracy was 52.8% for reasoning models and 38.7% for general models. However, the average number of false-positive diagnoses per case also rose from 1.09 to 1.47, an increase of approximately 34.8%. Here, false positives refer to diseases listed by the models that did not match the study’s reference answers; they do not represent the number of incorrect treatments patients actually received.

The study also found that the differential diagnoses proposed by reasoning models—the possible diseases for physicians to compare—were more clinically relevant, but their answers were less consistent across repeated runs. Scienmag’s coverage of the same paper noted that the pattern of higher accuracy accompanied by more false positives persisted across different prompt designs, model parameters, and evaluation methods, suggesting that a single prompting technique is unlikely to resolve the problem.

The team therefore tested a clear division of responsibilities: the model expanded the diagnostic possibilities, and physicians then checked their reliability. In a validation involving 15 physicians, this structured collaboration improved accuracy and reduced false positives. The results support incorporating physicians’ screening and judgment into the workflow, but the evidence still comes from controlled retrospective testing and has yet to demonstrate improved care outcomes for real patients.

Verification also takes time. According to i10X, physicians needed approximately 15 to 20 minutes per case to read the full reasoning, compared with approximately 5 to 8 minutes to review concise outputs. Bringing these tools into busy clinical settings will depend directly on whether physicians can quickly find evidence supporting a diagnosis and gaps in the reasoning.

The study’s limitations are equally clear: the main evaluation used a fixed list of candidate diseases and discharge diagnosis codes as the reference, making it potentially subject to constraints imposed by the list and coding errors. Existing medical records also cannot fully reproduce the process of physicians asking follow-up questions, ordering additional tests, and responding to changes in a patient’s condition. Prospective clinical studies are needed next to determine whether higher diagnostic accuracy can translate into fewer misdiagnoses and better patient outcomes in routine care.

References

  1. Nature Communications
  2. Scienmag
  3. i10X News