Medical Technology · global
Microsoft’s diagnostic AI achieves 83% accuracy in simulated cases; clinical benefits remain unproven
From asking follow-up questions about medical history to selecting tests, Microsoft’s system outperformed general practitioners in sequential diagnostic testing. But results from text-based cases still require validation before they can establish safety and effectiveness in emergency departments.
When symptoms have no clear cause, the challenge of diagnosis often lies in deciding what to ask next and which test to order. Microsoft researchers brought this reasoning process into simulated testing, where their diagnostic orchestration system achieved 83.2% accuracy on complex cases, outperforming general practitioners who could use reference tools. Published in Science on October 22, the study provides a measure of AI’s ability to gather evidence step by step, but has yet to demonstrate that it can improve care for real patients.
Microsoft calls the system MAI-DxO. It simulates a group of physicians with different roles, proposing possible diagnoses, challenging current hypotheses, and revising judgments as new information emerges. Reuters reporting that same day also described this approach: the system must choose its own questions and tests, rather than read a complete, already organized medical record and immediately provide an answer.
The study’s SDBench evaluation framework includes 304 complex cases from The New England Journal of Medicine and 325 records of emergency department visits resulting in hospital admission from the MIMIC-IV-ED database. Testing begins with limited clinical information, with additional clues provided only when participants request them. The evaluation tracks diagnostic accuracy, the evidence requested, and estimated testing costs.
On complex cases, general practitioners with access to reference materials achieved 36.1% accuracy; Microsoft’s announcement also stated that accuracy was approximately 22% without external reference resources. On emergency department cases, accuracy was 80.3% for the system and 46.8% for physicians. Both datasets showed the same pattern, but the comparators were individual general practitioners. These results therefore cannot establish that the system outperforms specialists in the relevant fields or clinical teams.
Cost results also depended on the settings. Medical Economics noted that without a budget limit, the system’s average testing costs were actually higher than physicians’ costs; only after cost constraints were introduced did it trade some accuracy for lower spending. Claims of being “more accurate and cheaper” must therefore be understood alongside the testing conditions. These figures estimate testing costs and do not account for patient discomfort, risks from invasive procedures, or time spent waiting for results.
The larger gap is that this remains a controlled, text-based simulation: it does not directly interpret images or pathology slides, and it is difficult to reproduce patients’ omission of symptoms, communication in clinical settings, and time pressure. The study did not evaluate treatment decisions, patient safety, or clinical outcomes. Even if diagnostic answers are more often correct, that does not directly establish whether the system might miss emergencies or help patients receive appropriate care sooner.
Microsoft said it would release the benchmark and evaluation code for some NEJM cases and begin a prospective study with Beth Israel Deaconess Medical Center in Boston, allowing patients to interact directly with the system and comparing its assessments with independent physician evaluations. According to Reuters, the collaboration will test its use for triage in emergency department waiting areas. As the system moves from simulated cases into this setting, the central question is whether its diagnostic advantage can translate into reliable care decisions.