Medical AI · global
Stopping the Record Before Diagnosis: Protege Launches an AI Diagnostic Reasoning Benchmark
Using 256 physician-reviewed cases, AI must identify diseases from medical records before the answers were documented. The test more closely reflects the information constraints of diagnosis, but whether it can shorten patients’ journeys to diagnosis still requires clinical evidence.
Clues to a disease may already be scattered across records of previous visits and laboratory reports, without yet forming a clear diagnosis. Can AI detect these signals? On October 21, Protege announced the Disease Odyssey benchmark, which cuts off medical records before a diagnosis is documented to test whether models can identify a developing disease, moving medical AI testing from medical knowledge toward interpreting actual medical records.
This retrospective benchmark includes 256 physician-accepted cases covering 129 ICD-10 diagnostic categories and compares four frontier models. According to its GitHub documentation, all data from the day the target diagnosis was first documented onward were removed; laboratory results were retained only if both the specimen collection date and the result date preceded the cutoff. This time boundary is intended to prevent models from answering with information that was not yet available at the time.
The same medical record was presented in both free-response and six-option multiple-choice formats, with each model answering three times in each format. Protege said all four models performed better on multiple-choice questions than on free-response questions; a separate company announcement launching the service summarized diagnostic accuracy as 51% to 67% and mentioned conditions including sleep apnea and anemia. These figures describe performance under specific case selection and scoring conditions and cannot be directly treated as accuracy in routine clinical visits.
The benchmark also revealed that a “wrong answer” is not always easy to define. Protege described a sleep apnea case in which none of the models’ free-response answers matched the reference diagnosis, yet physician reviewers considered the other diseases they proposed reasonably supported. A medical record may support several interpretations at once; requiring a model to identify only the single diagnosis documented later may therefore count reasonable clinical judgment as an error.
To allow outside scrutiny of the testing methods, Protege published the full prompts for both question formats on GitHub, along with the large language model prompts used to judge free-response answers and the scoring documentation. However, the medical records, reference diagnoses, answer options, physician reviews, and model outputs have not been released, and the benchmark is restricted to evaluation, with no use for training or fine-tuning. Publishing the question rules helps explain how performance is measured, but external researchers still cannot fully reproduce the results from the repository alone.
Disease Odyssey is also part of Protege’s newly launched independent evaluation service for medical AI. The company said the data come from its network of healthcare providers, that clinical experts review the questions and answers, and that it makes efforts to avoid overlap between test data and model training data. These are the evaluation provider’s descriptions of its processes and do not yet constitute external evidence that data overlap has been completely ruled out.
For patients, the real test lies beyond the scores: after AI suggests clues, does it lead to appropriate testing and reduce delays, and could it increase false alarms and unnecessary interventions? The current cases were selected and are not a representative sample of all clinical encounters, nor has physician–AI collaboration been tested in ongoing care. This benchmark provides a concrete starting point for examining diagnostic capabilities; whether it can improve patient outcomes still requires tracking how predictions change actual medical decisions.