Personality Inference From Semistructured Interviews Using Large Language Models: Comparative Evaluation Study
Journal of Medical Internet Research ·
Background: Personality traits are central to understanding human behavior and life outcomes, yet their assessment remains challenging due to biases and limited expressiveness in traditional self-report inventories. The emergence of large language models (LLMs) creates new opportunities to infer personality from naturalistic text, but it remains unclear whether these systems can reliably predict validated personality traits or whether their predictions primarily reflect superficial linguistic patterns. Objective: This study evaluates the ability of LLM-based methods to predict continuous Big Five personality scores (1‐5 Likert scale) from semistructured interview transcripts and compares the performance of different modeling paradigms. Methods: We analyzed semistructured interview transcripts from 518 participants, each paired with validated self-reported Big Five scores measured using the 10-item Big Five Inventory (BFI-10). We compared four paradigms: (1) zero-shot and chain-of-thought prompting with GPT-5 mini; (2) parameter-efficient fine-tuning (low-rank adaptation) applied to open-source RoBERTa-base and LLaMA-2-7B models; (3) supervised fine-tuning of GPT-4.1 mini across two independent training runs; and (4) embedding-based regression using 4 pretrained encoders (all-mpnet-base-v2, E5-large-v2, text-embedding-3-small, and text-embedding-3-large) with a multioutput ridge regression. Results: Supervised fine-tuning yielded the strongest overall performance among the evaluated approaches. In the best-performing run, correlations reached =0.718 for conscientiousness and =0.631 for agreeableness. Neuroticism showed moderate and consistent performance across the 2 fine-tuning runs, whereas extraversion was less stable across runs. Embedding-based regression, particularly with text-embedding-3-large, achieved competitive results for conscientiousness and agreeableness, although it remained below supervised fine-tuning. Prompt-based methods showed weak and inconsistent performance across traits. Openness was generally difficult to predict, although RoBERTa-base with embedding-based low-rank adaptation fine-tuning showed modest but statistically significant performance, indicating trait- and method-dependent variability. Conclusions: LLM-based approaches exhibit trait-dependent performance in predicting personality from naturalistic dialogue, with fine-tuning substantially improving correspondence with validated psychometric scores. These findings suggest that personality inference from naturalistic dialogue is feasible but depends strongly on modeling strategy, trait domain, and the availability of supervised fine-tuning with validated personality measures.