External Validation of Postpartum Hemorrhage Prediction Models Using Electronic Health Record Data.

Objective: A recent study leveraging machine learning methods found that postpartum hemorrhage (PPH) can be predicted accurately at the time of labor admission in the U.S. Consortium for Safe Labor (CSL) dataset, with a C-statistic as high as 0.93. These CSL models were developed in older data (2002-2008) and used an estimated blood loss (EBL) of ≥1,000 mL to define PPH. We sought to externally validate these models using a more recent cohort of births where blood loss was measured using quantitative blood loss (QBL) methods.

Study Design: Using data from 5,261 deliveries between February 1, 2019 and May 11, 2020 at a single tertiary hospital, we mapped our electronic health record (EHR) data to the 55 predictors described in previously published CSL models. PPH was defined as QBL ≥1,000 mL within 24 hours after delivery. Model discrimination and calibration of the four CSL models were measured using our cohort. In a secondary analysis, we fit new models in our study cohort using the same predictors and algorithms as the original CSL models.

Results: The original study cohort had a substantially lower rate of PPH, 4.8% (7,279/228,438) versus 25% (1,321/5,261), possibly due to differences in measurement. The CSL models had lower discrimination in our study cohort, with a C-statistic as high as 0.57 (logistic regression). Models refit in our study cohort achieved better discrimination, with a C-statistic as high as 0.64 (random forest). Calibration improved in the refit models as compared with the original models.

Conclusion: The CSL models' accuracy was lower in a contemporary EHR where PPH is assessed using QBL. As institutions continue to adopt QBL methods, further data are needed to understand the differences between EBL and QBL to enable accurate prediction of PPH.

Key Points: · Machine learning methods may help predict PPH.. · EBL models do not generalize when QBL is used.. · Blood loss estimation alters model accuracy..