Miscarriage Risk Prediction for Women With Polycystic Ovary Syndrome Who Achieve Clinical Pregnancy After Fresh Embryo Transfer: Multicenter Development, Geographic External Validation, and Prospective Temporal Validation Study
Journal of Medical Internet Research ·
Background: Women with polycystic ovary syndrome (PCOS) who achieve clinical pregnancy following fresh in vitro fertilization (IVF)/intracytoplasmic sperm injection (ICSI) carry an elevated miscarriage risk. However, individualized risk stratification after clinical pregnancy confirmation but before outcome determination remains challenging in clinical practice. Objective: This study aimed to develop, externally and prospectively validate over time, and interpret a machine learning–based risk stratification model for women with PCOS who had achieved a confirmed clinical pregnancy after fresh IVF/ICSI. Methods: This multicenter study included 8911 clinical pregnancy IVF/ICSI cycles across development (n=7241; miscarriages: n=973), external validation (n=1170; miscarriages: n=146), and prospective temporal validation (n=500; miscarriages: n=62) cohorts. Prespecified predictors contained no observed missing values; training-fold median imputation was kept only as a leakage-safe fallback. A total of 13 candidate machine learning and traditional statistical algorithms were compared internally using repeated stratified 5-fold cross-validation (5 repeats) within the development cohort; preprocessing and hyperparameter tuning were performed exclusively in training folds. A prelocked Extreme Gradient Boosting (XGBoost) model was carried forward unchanged for external and prospective temporal validation. Cross-fitted logistic recalibration of averaged XGBoost out-of-fold probabilities was evaluated as a secondary development-sample diagnostic analysis. Model performance encompassed discrimination, precision-recall, calibration, threshold-based metrics, decision curve analysis, risk stratification, and Shapley additive explanations–based interpretability. Results: Miscarriage rates were 13.4% (973/7241), 12.5% (146/1170), and 12.4% (62/500) for the development, external validation, and prospective temporal validation cohorts, respectively. In repeated cross-validation, XGBoost (area under the receiver operating characteristic curve [AUC] 0.885, 95% CI 0.874-0.896) and LightGBM (AUC 0.882, 95% CI 0.871-0.893) showed comparable discrimination; the paired XGBoost-LightGBM DeLong test was not significant (Holm-adjusted P =.07). Raw XGBoost out-of-fold probabilities overestimated absolute risk (mean predicted risk 30.3% vs observed risk 13.4%); cross-fitted recalibration within the development cohort improved calibration but was not propagated to validation cohorts. The locked XGBoost model achieved AUCs of 0.787 (95% CI 0.748-0.827; Brier score 0.099) in external validation and 0.763 (95% CI 0.709-0.816; Brier score 0.104) in prospective temporal validation, showing cross-cohort performance attenuation. Risk-quartile stratification using fixed development-cohort cutoffs showed consistent event enrichment across validation cohorts, but positive predictive values remained modest. Shapley additive explanations analysis identified anti-Müllerian hormone, endometrial thickness on the day of human chorionic gonadotropin administration, and basal luteinizing hormone as leading predictors. Conclusions: The locked XGBoost model using routine preoutcome variables may support miscarriage risk stratification and counseling for women with PCOS with clinical pregnancy after fresh IVF/ICSI. Given cross-cohort discrimination loss and calibration drift, it should serve only as a risk-ranking tool, not for diagnosis or treatment selection. Local calibration monitoring is mandatory before any real-world clinical deployment.