Biomedical Technology · global
When Medical AI Meets Fake Data Attacks: SAMD Preprint Attempts to Bring Risk Into the Design Stage Earlier
The safety of AI medical devices depends not only on whether the model is accurate, but also on whether the data has been tampered with before entering the model. An arXiv preprint proposes SAMD, a tool that attempts to use large language models to help identify attack paths that could lead to misdiagnosis or incorrect treatment.
AI medical devices are increasingly becoming part of the clinical setting: they read images, signals, or patient data, then pass results to physicians or care workflows. But if the data entering the model is altered at the moment of inference, the risk is not just algorithmic bias; the entire medical system may be led toward incorrect judgments.
A preprint submitted to arXiv on May 28 introduces SAMD, an automated tool for analyzing “false data injection” attack scenarios in AI/machine learning medical devices. The research team focuses not on training data contamination, but on attackers sending erroneous data into a model through vulnerable components during actual device use, potentially causing misdiagnosis or inappropriate treatment.
SAMD’s core approach is to treat an AI medical device as a control system and apply system safety analysis methods such as STPA-Sec to examine, one by one at the design stage, the relationships among sensors, software components, communication interfaces, and machine learning engines. The tool combines vulnerability databases with large language models to first identify the technical components in device documentation, then search for related known vulnerabilities, and finally generate possible attack paths and steps.
The research team used five AI/ML medical devices that have been FDA cleared in the United States as case studies. The preprint reports that SAMD achieved 100% precision in identifying target device technologies in the case documents; when retrieving known vulnerabilities related to those technologies, its precision was 63.2%; and when generating attack scenarios targeting machine learning models, the research team assessed their relevance and correctness at 95.3%, with a maximum time of about 191.64 seconds for a single case.
These figures suggest that SAMD is more like an early risk inventory tool than direct proof of clinical safety. Its validation is based on five cases and device documents, and is not equivalent to attack testing in a real hospital environment. Nor can it be inferred that FDA-cleared devices currently have vulnerabilities that can be immediately exploited. In particular, attack scenarios generated by large language models may help expand the range of possibilities engineers consider, but they still require item-by-item review by cybersecurity, medical device, and clinical experts.
This study brings to the foreground an issue easily obscured by the AI boom: regulation and validation of medical AI cannot only ask how a model performs on standard datasets. They must also ask whether data flows, device assembly methods, and real-world use settings leave openings that can be manipulated. As AI medical devices enter more diagnostic and treatment workflows, safety analysis at the design stage may become a harder-to-avoid part of regulatory and quality systems.
However, this paper is still a preprint, has not yet undergone peer review, and no independent external source has provided further corroboration of the same event. The actual value of SAMD will depend on whether it can maintain stable performance across more types of medical devices, more scenarios with uneven document quality, and more rigorous expert review.