Medical Artificial Intelligence · global
Microsoft diagnostic AI gets eight in ten simulated emergency cases right; clinical benefits remain unproven
A new study had AI and individual physicians ask questions, select tests and then make diagnoses step by step. AI held a clear advantage in the text-based evaluation, but simulated information, cost trade-offs and patient safety that has yet to be measured define what the results mean.
When a patient’s symptoms are still unclear, the challenge of diagnosis often lies in deciding what question to ask next and which test will help identify the cause. A study published in *Science* on October 22 brought this reasoning process into a text-based simulation. Microsoft’s diagnostic system, Diagnostic Orchestrator (DxO), achieved 80.3% diagnostic accuracy in an evaluation of emergency department cases, compared with 46.8% for individual generalist physicians. However, this gap does not yet demonstrate that AI performs better in actual care.
The study used a sequential diagnostic evaluation called SDBench, including 325 records from the MIMIC-IV-ED database of emergency department visits followed by hospital admission, along with 304 complex cases from *The New England Journal of Medicine* (NEJM). Participants initially received only a brief case summary and had to ask follow-up questions about the patient’s history, request tests and ultimately submit a diagnosis. On the NEJM cases, DxO achieved 83.2% accuracy, compared with 36.1% for physicians who could use search and reference materials.
DxO is designed like a virtual consultation: different roles propose possible causes, challenge current hypotheses and revise their judgments as new evidence emerges. According to *Medical Economics*, the main results used GPT-5 as the underlying model. This setup tests whether the system can gather clues in an orderly sequence, rather than simply identify the answer after reading a complete case.
However, not all clues in the simulation came from actual records. Official documentation explains that the AI “gatekeeper” responsible for answering questions prioritizes information available in the original case; if requested information is absent, it generates plausible simulated results. An AI judge also scores the final diagnosis against the case answer on a five-point scale, with a score of four or five considered correct. Accuracy here therefore reflects performance under specific simulation and scoring rules and cannot be equated directly with bedside diagnostic ability.
Higher accuracy also does not necessarily come with lower costs. *Medical Economics* reported that, without a budget cap, the estimated cost of tests requested by DxO was $4,979 per emergency department case, compared with $3,400 for physicians. Adding budget constraints reduced the system’s costs, but accuracy also declined. These figures estimate test costs using standardized prices; they are not actual medical expenditures and do not fully account for discomfort, risks associated with invasive tests or waiting times.
The larger gap remains between the evaluation and the clinic. This was a retrospective, text-only study that did not have the system directly interpret images or pathology slides, nor did it cover situations such as incomplete accounts from real patients. The comparison was with individual physicians, rather than matched specialists or multidisciplinary teams. The study assessed diagnosis alone and did not measure treatment choices, patient safety or health outcomes. According to disclosures cited by *Medical Economics*, all authors are Microsoft employees and hold equity in the company, and eight are also inventors on related patent applications, making subsequent independent validation particularly important.
Microsoft has publicly released the SDBench evaluation code, including the gatekeeper, judge and cost-estimation components, but access to the full cases remains subject to conditions: NEJM data require a license, while MIMIC data require completion of PhysioNet credentialing, training and a data use agreement. The public repository includes five synthetic demonstration cases. The company also announced plans for a prospective study with Beth Israel Deaconess Medical Center in which patients will interact directly with the system, followed by a comparison with physicians’ independent assessments. This will help test whether the advantages in simulated reasoning extend to real patient consultations; whether they can improve care outcomes still requires direct evidence.