← Back to Home

Identifying Immunotherapy Toxicity in Medical Records: AI Assistance Cuts Review Time by 43.5%

Adverse reactions to cancer immunotherapy are often buried in scattered clinical notes. A multistage validation study shows that AI can help staff organize these clues faster and more consistently. However, a decline in performance during prospective testing also underscores that clinical benefits remain to be demonstrated.

By SURL BioNews

Safety monitoring for cancer immunotherapy sometimes requires piecing together information across pages of medical records: Is an adverse reaction occurring, how severe is it, and is it related to treatment? A study published on October 29 in Nature Cancer applied AI to this labor-intensive review process and found that AI assistance reduced staff members’ median time spent annotating medical records by 43.5%, while also improving interpretation accuracy and agreement among reviewers.

The study focused on immune-related adverse events during treatment with immune checkpoint inhibitors. The system extracted information from clinical text on the presence or absence, timing, and severity of six categories of events, along with physicians’ documented attribution to treatment and degree of certainty. This information can help organize safety data, but it still requires human verification; identifying descriptions in medical records is not equivalent to independently establishing causation.

Validation comprised retrospective testing, prospective silent deployment, and a staff user study. The March preprint of the same study, indexed by PubMed and hosted by Maastricht University, had already reported 263 medical records for retrospective testing and 884 for prospective testing. In retrospective testing, the F1 score was 0.92 for event detection and 0.66 for severity grading. F1 combines precision and recall; the gap between the two scores indicates that identifying events is easier than determining their severity.

The staff study in the final paper used a randomized crossover design, with 17 clinical research staff comparing manual and AI-assisted workflows, yielding 316 analyzable timed observations. Median annotation time per medical record fell from 428 seconds to 242 seconds. The 43.5% figure represents the reduction between the two groups’ medians; the earlier preprint abstract used approximately 40%, and reporting should follow the explicit definition in the final version.

Beyond the improvement in speed, the “exact match rate,” which required severity grades for all six categories of current events to match the reference answers, rose from 19.4% to 25.9%. Measures of agreement among reviewers increased from 0.22–0.51 to 0.82–0.85. However, the exact match rate remained relatively low, and greater agreement alone does not establish that every judgment was correct.

The three-month prospective silent deployment, which more closely reflected real-world data flows, presented another test: detection F1 fell to 0.72–0.79, below retrospective performance. Silent deployment means evaluating the system without intervening in care decisions. These results therefore support its ability to extract information from medical records, but do not yet establish that patients can receive earlier intervention or experience fewer hospitalizations and treatment interruptions.

All validation stages so far have taken place within the same healthcare system. External validation is still lacking to determine whether performance can be maintained across other hospitals, medical record formats, and cancer care settings. This study provides a concrete tool and validation pathway for organizing immunotherapy safety data. To become a reliable source of support for care decisions, it must still demonstrate that it can identify errors across different settings and deliver measurable patient benefits.

References

  1. Nature Cancer
  2. PubMed / National Library of Medicine
  3. Maastricht University