Evaluating the Performance of Five Leading Large Language Models in Generating Health Education Materials for Patients With Infertility: Comparative Analysis
Journal of Medical Internet Research ·
Background: Patients with infertility often have inadequate reproductive health literacy (RHL), while existing health education materials may be difficult to read, overly complex, and insufficiently tailored to individual needs. In recent years, large language models (LLMs) have created new opportunities to improve health education through their ability to generate and interpret medical information. However, their applicability and content quality in infertility education remain insufficiently evaluated. Objective: This study systematically evaluated the readability and content quality of LLM-generated health education materials for patients with infertility, identified model strengths and limitations, and provided evidence to support their standardized use in reproductive health. Methods: This study selected DeepSeek-R1, ChatGPT (GPT-5.2 Plus), Kimi 2.5, Qwen 3, and Gemini 3 as the study subjects. Based on the theoretical framework of RHL, this study developed a standardized question list comprising 6 dimensions and 93 items, and used unified prompts to generate health education texts in both Chinese and English. In this study, the Datayze Readability Analyzer was used to assess readability via the Flesch Reading Ease, Gunning Fog Index, and Flesch-Kincaid Grade Level (FKGL). Meanwhile, the content quality was evaluated in a double-blinded manner by reproductive medicine experts across 3 dimensions: accuracy, comprehensiveness, and humanistic care. Repeated-measures analysis of variance was used to compare performance across the LLMs, and Pearson correlation analysis was conducted to examine the associations between readability and content quality. A radar chart was used to evaluate the overall performance of each model. Results: The 5 LLMs differed significantly in both text readability and content quality. In terms of readability, Gemini 3 and Qwen 3 performed best, with significantly higher FRE scores than the other 3 models. DeepSeek-R1 and Kimi 2.5 performed least favorably, with FKGL scores corresponding to a ninth-grade reading level. ChatGPT (GPT-5.2 Plus) showed intermediate performance. In terms of content quality, Kimi 2.5 performed best across accuracy, comprehensiveness, and humanistic care, followed by DeepSeek-R1. ChatGPT (GPT-5.2 Plus) performed least favorably. Analysis revealed significant negative correlations between readability and content comprehensiveness for texts generated by Kimi 2.5, indicating a trade-off between these 2 attributes. The composite radar chart analysis showed that Kimi 2.5 achieved the highest score (11,594.33), whereas ChatGPT (GPT-5.2 Plus) had the lowest score (5555.83). Conclusions: Different LLMs showed significant differences in performance in generating health education texts for patients with infertility. In some models, readability was negatively correlated with content comprehensiveness. Under the conditions of this study, Kimi 2.5 demonstrated strong overall performance. However, no model achieved the best performance across all evaluation dimensions. Future applications of LLMs in reproductive health education will require domain-specific optimization, context-specific adaptation, and systematic quality control to support more accurate, comprehensive, and patient-centered communication.