Real-World Use and Expert-Guided Optimization of a Large Language Model–Based Pediatric Health Assistant: Retrospective Observational Study
Journal of Medical Internet Research ·
Background: Pediatric care resources are limited and unevenly distributed in many health systems. Caregivers increasingly turn to large language models (LLMs) for children’s health information, but real-world evidence on LLM use and response quality in general pediatric care is limited. Objective: This study aimed to characterize real-world use, consultation needs, and response quality of Xiaoyi, an LLM-based pediatric health assistant, and to evaluate the impact of expert-guided optimization on response quality. Methods: This study collected all consultation sessions submitted between April 7 and December 31, 2025, to Xiaoyi, a pediatric health assistant co-developed by Children’s Hospital of Chongqing Medical University and Zuoyi Tech. Temporal patterns of consultation activity were characterized; all sessions were annotated for pediatric subspecialty, reason for consultation, and presenting symptoms through LLM-assisted labeling with pediatrician validation. A multidisciplinary panel of 40 pediatricians evaluated Xiaoyi’s response quality over 2 rounds using predefined criteria established by expert consensus: an initial evaluation informed system optimization (prompt revision, knowledge-base and tool expansion, and validation rules) and a follow-up evaluation assessed its impact. In each round, 1000 sessions randomly sampled from the corresponding pre- or postoptimization period (2000 in total) were rated at the session level (overall quality and clinical appropriateness) and the turn level (6 predefined deficiency categories). Categorical ratings were compared using the chi-square test, and turn-level deficiency rates were compared using generalized estimating equations to account for within-session correlation. Results: Caregivers of 8266 children (median age 4.2, IQR 1.3-8 years; 4705/8266, 56.9% male) from 31 of 34 provincial-level divisions in China initiated 16,074 sessions, of which 7046 (43.8%) occurred outside routine clinic hours (08:00-17:30). Queries spanned 31 subspecialties, most frequently child health care (2025/16,074, 12.6%), pulmonology (1903/16,074, 11.8%), and gastroenterology (1833/16,074, 11.4%). The leading reasons for consultation were symptom consultation (7845/16,074, 48.8%), medical report interpretation (2531/16,074, 15.7%), and triage guidance (2515/16,074, 15.6%); among symptom consultation sessions, fever (2719/16,074, 34.7%) and cough (2349/16,074, 29.9%) were most common. After expert-guided optimization, the number of sessions rated good (n=1000) in overall quality (the response fully resolving the caregiver’s question with actionable information) increased from 652 (65.2%) to 743 (74.3%), and the number rated clinically appropriate increased from 614 (61.4%) to 874 (87.4%, both P <.001). The turn-level deficiency rate declined from 19.4% (790/4073 turns) to 6.9% (236/3419 turns; odds ratio 0.30, 95% CI 0.24-0.37; P <.001), with suboptimal clinical recommendations (not optimally tailored to the child’s age or clinical context) remaining the most frequent category. Conclusions: This real-world study suggests that an LLM-based health consultation assistant may complement existing pediatric care, with expert-guided optimization substantially improving response quality. However, the persistence of deficiencies highlights the need for continued monitoring and refinement to ensure clinical reliability. Trial Registration: