← Back to Home

AI learns to ask follow-up questions and select tests, with Microsoft study achieving 80% diagnostic accuracy in simulated cases

Starting with limited clues and then deciding what to ask and test next, AI diagnostic evaluations are beginning to more closely reflect clinical reasoning. But high scores on medical records remain some distance from demonstrably improving patient care.

By SURL BioNews

The difficulty of diagnosis often lies in what happens before the answer emerges: Which symptoms warrant follow-up questions, which test can narrow the possibilities, and when should the search stop? A Microsoft team’s study published in *Science* on October 22 allowed AI to gather clues step by step from limited information, assessing whether it could follow this path to a diagnosis. The results showed accuracy above 80% in both sets of simulated cases, but the study has yet to demonstrate that the system can deliver the same benefits at the bedside.

The study used a “sequential diagnosis” design. Unlike tests that provide a complete medical history at once and ask participants to select an answer, participants first read a brief clinical description, then choose to request information or order tests. The team’s Sequential Diagnosis Benchmark, SDBench, records whether the diagnosis is correct, what information was requested, and the estimated cost of tests, seeking to incorporate the reasoning process into the assessment.

The Microsoft AI Diagnostic Orchestrator (MAI-DxO) system tested in the study simulates discussions among multiple physicians: proposing possible diseases, challenging current judgments, and updating the differential diagnosis as new evidence emerges. According to Microsoft’s announcement, the system also weighs the potential information gained from tests against their costs to decide the next investigative step.

The paper’s abstract reports diagnostic accuracy of 83.2% on clinicopathological conference cases from *The New England Journal of Medicine* and 80.3% in an evaluation using MIMIC-IV-ED emergency department visit records. Generalist physicians participating in the comparison scored 36.1% and 46.8%, respectively. Microsoft added that the first set included 304 complex cases and that the physicians’ score of around 36% was achieved with access to external reference materials. These comparison conditions are essential context for understanding the gap.

The two datasets provided different testing scenarios: the journal cases tested reasoning about complex diseases, while the emergency department records extended the evaluation to more common reasons for seeking care. However, both still reconstructed the diagnostic process in controlled environments. These results cannot be directly interpreted as showing that AI outperforms all physicians, nor can they answer whether it can reduce missed diagnoses in actual practice, shorten waits for treatment, or improve patient outcomes. Test costs were also estimates and cannot be treated as savings already achieved by hospitals.

To allow other researchers to examine the method, Microsoft said it would release the benchmark and evaluation code for a subset of the journal cases, covering information provision, answer assessment, and cost estimation. This would help external teams reproduce the tests and compare other models. However, the release is limited to a subset of cases, so the boundaries of what the evaluation data and methods cover still require attention.

The next step will be research involving actual patients. Microsoft announced plans to work with Beth Israel Deaconess Medical Center in Boston to have patients interact directly with the system, then compare the results with physicians’ independent assessments. Results from this prospective study are still pending. For now, the 80% accuracy provides grounds for further validation; how AI should be integrated into diagnosis and treatment, when physician intervention is needed, and whether it can improve care remain questions to be answered one by one in clinical settings.

References

  1. Science / PubMed
  2. Microsoft AI