A Multistage Fine-Tuning and Empathy Alignment Approach for an Emotional Support Dialogue System: Development and Evaluation Study
Journal of Medical Internet Research ·
Background: Large language models (LLMs) are increasingly being used for text-based emotional support. However, general-purpose LLMs may not consistently recognize emotional cues, adapt supportive strategies to contexts, or generate responses aligned with psychologically informed communication. The limited availability of large-scale, structured Chinese counseling data further constrains the development of models for context-sensitive emotional support. Objective: We developed and evaluated YuanChat, a family of specialized LLMs for context-sensitive emotional support; assessed whether E-URIAL improved response quality; and compared YuanChat with base LLMs. Methods: We constructed 2 datasets from anonymized real-world counseling sessions: ECD-3 (104,897 counselor response turns) for fine-tuning and (9413 manually annotated counselor responses). Annotations covered emotional support conversation stages (exploration, comfort, and action), counseling techniques (clarification, support, and challenge), and empathic effects. We developed 2 YuanChat variants using GLM-4-9B-Chat and Huozi 3.5 as backbone models using parameter-efficient fine-tuning with low-rank adaptation (LoRA). We then developed E-URIAL, an inference-time framework combining predicted emotion and support strategy labels with retrieved labeled exemplars. Given the dialogue history through the help seeker’s latest utterance, each model generated a counselor-style response. We compared YuanChat with GPT-4, ERNIE 4.0, CharacterGLM, GLM-4-9B-Chat, and Huozi 3.5. The evaluation comprised (1) automatic evaluation on in-domain and external datasets, assessing empathy identification and empathic expression; (2) evaluation of empathy-related situational reasoning using an expert-developed multiple-choice benchmark; (3) evaluation by clinical experts (n=8) of empathic attunement, support strategy appropriateness, perceived helpfulness, and safety and professional boundary adherence; and (4) a complementary evaluation by volunteer master’s students in clinical and counseling psychology (n=50) of response fluency, empathy, and perceived helpfulness. Results: Fine-tuning increased YuanChat-Huozi3.5’s empathic expression score from 0.57 to 0.68; empathy identification score increased from 0.54 to 0.59. In expert evaluation, YuanChat-Huozi3.5 received a higher empathic attunement score than the base Huozi3.5 model (1.279 vs 0.327; mean difference 0.952, 95% CI 0.746‐1.152). Using professional counselor responses as the reference, the mean differences for YuanChat-Huozi3.5 were 0.148, 0.079, 0.039, and 0.002 in empathic attunement, supportive appropriateness, perceived helpfulness, and safety and professional boundary adherence, respectively. No statistically significant differences were detected between YuanChat-Huozi3.5 and the counselor references (all Holm-adjusted ≥.29). Higher scores were observed when E-URIAL was applied. When E-URIAL was applied with 3 retrieved exemplars, YuanChat-Huozi3.5 achieved an empathy identification score of 0.69 versus 0.59 without E-URIAL, an empathic expression score of 0.79 versus 0.68, and a perceived helpfulness score of 0.74 versus 0.61. Conclusions: The findings indicate that fine-tuning on real-world counseling data and annotating models with structured emotional support labels improved context-sensitive empathic responding in LLMs. Without updating model parameters, E-URIAL further improved empathy identification and empathic expression across multiple models. Together, these methods provide a scalable, theory-informed approach to enhancing the emotional support capabilities of LLMs.