Evaluating an AI-Powered Patient Simulator for Communication Skills Training: One-Year Randomized Controlled Trial
Journal of Medical Internet Research ·
Background: Clinical communication skills are a core yet complex competency in medical education and practice. However, early-stage medical students often struggle to apply communication frameworks in practice due to limited opportunities for structured simulation training. Generative AI (GenAI)–supported training has the potential to offer a scalable and innovative approach for clinical communication skills training. Objective: This study examined the longitudinal development of medical students’ clinical communication skills over 1 academic year and investigated whether AI-powered simulation contributed to differential improvement relative to conventional training. In addition, the study explored motivational, behavioral, and affective mechanisms associated with communication skills development. Methods: A nonblinded randomized controlled design was used at a medical school in the United Kingdom, with students assigned either to the control group (n=28, 56%) or to the AI group (n=22, 44%), who practiced with SimPatient, a patient-simulator chatbot accessed through a web browser. Communication skills performance was assessed using standardized objective structured clinical examinations (OSCE) scores relative to the pass marks to enable comparison across time points. Paired t tests and difference-in-differences (DID) analyses were conducted between groups and over time to examine changes in communication skills. Latent change score modeling was used to identify predictors of communication skills change. Sentiment analysis was conducted on 6120 comments from AI-student conversations. Finally, a system quality survey was used to assess participants’ feedback on the AI-powered simulator. Results: Latent change score modeling showed that gains in intrinsic motivation and behavioral learning were significant predictors of students’ communication skills change. The AI group showed a greater descriptive increase in communication skills than the control group, yet the between-group difference was not statistically significant. Within the AI group, consultations whose lexical tone trended more strongly toward resolution (consultation tone) were descriptively associated with communication skills improvement; given the small subsample (n=17), this finding is reported as exploratory. The AI group’s confidence level in AI use was negatively associated with communication skills improvement, while the group nevertheless reported a generally high level of satisfaction with the system. Conclusions: In this study, medical students’ clinical communication skills increased over the academic year, with this development most strongly associated with their motivation and behavioral engagement; the AI-supported group did not differ significantly from conventional training. The simulator responded appropriately and provided a usable environment for practicing foundational communication behaviors but was limited in capturing the affective complexity, vulnerability, and narrative unpredictability of real clinical encounters. In conclusion, this study suggests that GenAI holds promise for supporting foundational clinical communication training and providing positive practice environments, thereby fostering motivation and behavioral engagement.