Let Electronic Health Records “Run Through” a Clinical Trial First: Five AI Agents Collaborate to Design Studies and Reproduce Treatment Effects

Before a clinical trial actually recruits patients, the research team must repeatedly refine the inclusion criteria, efficacy endpoints, and statistical methods. EmulatRx, developed by a team at Weill Cornell Medicine, seeks to let artificial intelligence first use de-identified electronic health records to “rehearse” this process, helping researchers estimate which designs are feasible and whether existing healthcare data can reproduce known treatment effects.

The system does not rely on a single chatbot to handle everything. Instead, five specialized large language model agents collaborate to generate trial protocols, construct electronic health record study cohorts, conduct clinical plausibility reviews, perform causal analyses, and oversee the overall workflow, respectively. The purpose of this division of labor is to retain records of the inputs, judgments, and corrections at every step, allowing researchers to trace how results were produced rather than merely receiving an answer that is difficult to verify.

The researchers evaluated a total of 20 “target trial emulations.” Ten acute-disease studies used the MIMIC-IV critical care database, while another 10 chronic-disease studies used data from the INSIGHT Clinical Research Network, covering five healthcare systems in New York City. The paper states that EmulatRx reproduced treatment effects reported in previous studies across multiple analyses, suggesting that it could potentially be used for pretrial feasibility assessments and protocol comparisons; however, this consistency does not mean that the system has demonstrated that a treatment is effective.

In the trial-protocol parsing stage, the GPT-4o-based “Trialist” agent achieved an F1 score of 95.4%. The model comparison in the study also showed that GPT-4o outperformed the tested local models in clinical relevance, reasoning, and SQL query generation. However, this result measures performance on specific tasks and cannot be directly interpreted as a 95.4% clinical accuracy rate for the entire workflow, nor does it demonstrate that the same advantage extends to other hospitals, medical-record formats, or disease areas.

The most practical use of this type of tool may be to identify problems such as overly narrow recruitment criteria, insufficient sample sizes, missing data, or covariate imbalances at an earlier stage, and to generate preliminary analyses from existing records for expert review. The team has archived version 1.0.0 of EmulatRx on Zenodo and linked it to a public code repository, providing external researchers with an entry point for reviewing and reproducing the implementation; Cornell University’s technology-transfer materials indicate that a provisional patent application has been filed for the related technology.

However, real-world data carry biases arising from treatment choices, record quality, and differences among patients. Even when causal-inference methods adjust for measured factors, they cannot guarantee the elimination of unrecorded confounding variables; language models that generate erroneous criteria or query code may also quietly alter the study cohort. The research team also acknowledges that standardized end-to-end evaluation benchmarks are currently lacking and that validation in more healthcare databases is still required.

EmulatRx is therefore better viewed as a design and auditing aid for clinical research teams than as an arbiter that automatically validates treatment efficacy. For it to enter routine trial planning, cross-institutional validation, version and audit management, medical-record privacy governance, and evidence standards acceptable to regulators must be established in addition to continuous human oversight; genuine prospective randomized trials remain a crucial part of answering questions about treatment effects and safety.