Biomedical Artificial Intelligence · eu
How Reliable Is AI at Organizing Cancer Medical Records? Two Studies Test Data Extraction and Model Review
Research abstracts released by Flatiron suggest that AI can approach expert performance when extracting data from German prostate cancer records, and that another model can identify some errors. But missed errors and validation details that have yet to be disclosed still limit the research uses these data can support.
In a prostate cancer medical record, details about diagnosis timing, treatment history, and signs of disease progression are often scattered across different documents. Organizing them into fields suitable for research analysis is one concrete use for large language models. However, if a model misinterprets the disease course, subsequent analyses of treatment effectiveness or treatment patterns may also be affected. Beyond organizing data quickly, the question is whether it does so reliably.
Flatiron Health announced on November 12 that it would present two data validation studies at the 2026 European Society for Medical Oncology Congress on Artificial Intelligence and Digital Oncology in Berlin. The press release did not provide accuracy figures, but research abstracts released by the company the previous day offered some quantitative results. Both posters are scheduled for presentation on November 17; the authors’ affiliations for the German prostate cancer study include Charité and Flatiron.
The first study used Flatiron’s German research database, with a large language model extracting prostate cancer diagnosis details, treatments, castration resistance, and disease progression from routine care records. The team used a validation framework called VALID, with manually abstracted data as the evaluation benchmark, and also examined whether relationships among diagnosis dates, stage, and pathological grade aligned with clinical expectations.
According to the abstract, after prompt refinement and repeated error analysis, F1 scores for all required prostate cancer fields were within 5 percentage points of expert manual extraction performance. The F1 score combines precision and recall in assessing extraction results. Being “within 5 percentage points” does not mean accuracy reached 95%, nor does it directly indicate how many errors remain in each field.
The second pilot study used another large language model as a reviewer, checking original medical record documents from the UK and Germany against extraction guidelines to flag content where the first model may have made errors. In an evaluation of prostate-specific membrane antigen positron emission tomography (PSMA PET) reports, the review model’s recall for errors confirmed through human review was 60% in the UK and 75% in Germany. In other words, this check still missed some known errors, so the absence of a flag cannot be treated as a guarantee that the data are correct.
Another evaluation covering initial management, surgery, and recurrence events examined 12 fields. The review model flagged predictions with suspected errors in 9 of those fields; after the flagged content was corrected, F1 scores improved by 0.4 to 6.5 percentage points in 8 fields. This offers clues for prioritizing human review, but the abstract did not report the false-positive rate, so it is not yet possible to determine how much additional checking the review process would require.
The information currently available for verification comes mainly from company-released research abstracts and a conference preview. These have not yet provided sample sizes, absolute F1 scores for individual fields, or confidence intervals, nor have they demonstrated that the same performance would hold at other healthcare institutions. These results support further validation of automated medical record abstraction workflows, but they are not yet sufficient to show that such workflows can improve patient care or produce data that directly meet drug review requirements. To support drug development, it remains necessary to determine, for the specific research question, which errors could change the conclusions and which steps require expert involvement.