← Back to Home

Can AI-drafted imaging reports help reduce missed findings? Study of nearly 240,000 examinations shows promise and validation gaps

Mosaic has released an internal study claiming that its vision-language model can identify abnormalities omitted from original reports and improve interpretation efficiency. But the reported detection increase of about 40% uses a specific calculation, and more comprehensive clinical validation is needed to demonstrate patient benefit.

By SURL BioNews

An imaging report must both identify abnormalities and clearly document the findings. If AI can flag an overlooked pneumothorax, fracture, or abnormal catheter position before a physician signs the report, its value extends beyond saving typing time. On November 11, Mosaic Clinical Technologies released an internal retrospective study covering 237,905 examinations, claiming that its Cognita vision-language model identified expert-confirmed findings omitted from the original reports in all 50 categories of abnormalities tested. However, these results currently come from research data released by the company, and the device remains investigational.

The model is designed to read images and generate written observations to help draft radiology reports, which physicians then review, revise, and sign. According to two accompanying reports on the Cognita research page, the computed tomography component covered 71,707 examinations and 35 categories of abnormalities. The X-ray component included 100,000 chest examinations and 66,198 musculoskeletal and abdominal examinations, evaluating 15 categories of abnormalities in total. The study compared the model's output with original reports completed without AI.

The company's claimed average detection gain of about 40% needs to be understood in the context of the metric's definition. The X-ray study defined “relative detection gain” as the number of additional expert-confirmed positive findings identified by the model divided by the number of true-positive diagnoses in the original physician reports. It therefore does not mean that overall accuracy increased by 40 percentage points, nor that every patient would receive 40% more diagnoses. For example, the relative gain for pulmonary nodules or masses on chest X-rays was 57.5%, and for pneumothorax it was 46.5%. These figures describe the extent of additional findings and cannot be directly translated into confirmed lung cancer diagnoses or treatment benefits.

The large total number of examinations also does not mean that experts reinterpreted every image. The X-ray study first used an internally validated large language model to determine whether the AI and physician reports documented specified abnormalities. It then randomly sampled cases in which the two disagreed for review by U.S. board-certified radiologists, with 713 examinations manually reviewed in total. Reviewers were blinded to whether the written findings came from AI or the original report, helping reduce source bias. However, validation focused on discordant cases still requires attention to the possibility that both sides missed an abnormality and cannot be treated as providing an independent diagnostic standard for every case.

The assessment of report quality was smaller still. In an evaluation of 159 chest X-ray cases, 91.2% of AI drafts were judged to contain no errors, compared with 83.0% of original physician reports. In another 107 musculoskeletal and abdominal X-ray cases, the proportions with errors having potentially major clinical impact were 3.7% and 6.5%, respectively. These results offer preliminary indications of draft quality, but the samples are limited. They cannot support an inference that similar rates of improvement occurred across nearly 240,000 examinations, much less directly prove that final reports are necessarily safer after physicians use the drafts.

The efficiency data came from a separate component involving records of actual use for investigational purposes. The X-ray report stated that physicians who used a certain number of AI drafts achieved a 31% increase in efficiency, weighted by examination volume, compared with their own performance before adoption. The measure was time-based work units completed per minute, rather than a uniform 31% reduction in the time spent on each report. Such before-and-after comparisons can show changes in workflow, but they may still be influenced by case mix, experience using the system, and other workflow adjustments. They are not sufficient on their own to establish that AI caused all of the improvement.

This study moves the question from “Can AI interpret images?” to “Can physicians identify findings more completely and work more efficiently when AI drafts reports?” The next questions are whether the results can be reproduced across different hospitals and patient populations, and whether the additional findings actually change management and reduce harm. Both the Cognita research page and the papers state that the models remain investigational devices and may not be sold in the United States. Mosaic says it is seeking authorization from the U.S. Food and Drug Administration. At this stage, these are data on a clinical tool that warrant further testing, rather than conclusive evidence of improved patient outcomes.

References

  1. Mosaic Clinical Technologies
  2. Cognita Imaging
  3. Mosaic Clinical Technologies and Cognita Imaging