Reasoning Across ECGs and Medical Records in One Pass: QoQ-Med3 Tests Multimodal Medical AI

Clinical judgment rarely relies on a single image or a single passage from a medical record. Physicians often need to piece together electrocardiograms, radiological images, laboratory records, and written reports, identifying where they corroborate or contradict one another. A study published in *npj Digital Medicine* introduces QoQ-Med3, an attempt to enable a single foundation model to work across these data formats instead of using a separate AI system for each type of examination.

The research team developed two versions: QoQ-Med3 and QoQ-Med3-MIMIC, the latter trained with MIMIC-IV data. The publicly released model is based on Qwen3-VL-8B-Instruct and has approximately 9 billion parameters. The model uses a reinforcement learning method with rewards adjusted for the clinical domain and data difficulty, aiming to prevent common image types or diseases from dominating training and to ensure that areas with less data, such as ultrasound and mammography, also receive sufficient learning signals.

The general version uses approximately 2.61 million samples from the CLIMB dataset, covering electrocardiograms as well as two-dimensional and three-dimensional data including chest X-rays, mammography, dermoscopy, pathology, fundus imaging, ultrasound, magnetic resonance imaging, and computed tomography. The MIMIC version additionally incorporates de-identified electronic medical records from intensive care units and emergency departments, enabling the model to answer questions within longer clinical contexts by integrating electrocardiographic signals, chest X-rays, and written records.

The paper reports that QoQ-Med3 achieved a balanced accuracy of 71.3% and an F1 score of 0.349 after results from multiple tasks were aggregated, outperforming the open and closed models included in the study overall, including GPT-4o. Improvements were particularly pronounced in ultrasound and mammography. However, these figures come from a combined evaluation across different datasets and classification tasks and cannot be directly interpreted as a 71.3% diagnostic accuracy rate in clinical settings. The F1 score also indicates considerable room for improvement in minority classes and complex tasks.

Cross-institutional data testing is another focus of the work. The version trained solely on public data was still transferable to held-out MIMIC-IV data not used in training, as well as private PMAP data from Johns Hopkins Hospital. The study also reports a 44.4% reduction after training in the hallucination rate caused by insufficient external information. These results support the conclusion that the model learned some transferable representations, but cross-dataset performance is not the same as reliability when dealing with real patients, differences in equipment, and incomplete records.

Both model versions, each with approximately 9 billion parameters, have been released under the MIT License, with support for multiple inference frameworks and containerized execution methods to help other teams reproduce and evaluate the work. At the same time, the research team explicitly positions them as research previews: current evidence comes primarily from retrospective data and predefined question-answering or classification tasks. Prospective clinical trials, workflow evaluations involving collaboration with physicians, and comprehensive validation of calibration, population bias, privacy, and responsibility for errors are still lacking, making the models unsuitable for direct clinical deployment.