Development and internal validation of a model predicting severe maternal morbidity using pre-conception and early pregnancy variables: a population-based study in Ontario, Canada

Background Improvement in the prediction and prevention of severe maternal morbidity (SMM) - a range of life-threatening conditions during pregnancy, at delivery or within 42 days postpartum - is a public health priority. Reduction of SMM at a population level would be facilitated by early identification and prediction. We sought to develop and internally validate a model to predict maternal end-organ injury or death using variables routinely collected during pre-pregnancy and the early pregnancy period. Methods We performed a population-based cohort study using linked administrative health data in Ontario, Canada, from April 1, 2006 to March 31, 2014. We included women aged 18–60 years with a livebirth or stillbirth, of which one birth was randomly selected per woman. We constructed a clinical prediction model for the primary composite outcome of any maternal end-organ injury or death, arising between 20 weeks’ gestation and 42 days after the birth hospital discharge date. Our model included variables collected from 12 months before estimated conception until 19 weeks’ gestation. We developed a separate model for parous women to allow for the inclusion of factors from previous pregnancy(ies). Results Of 634,290 women, 1969 experienced the primary composite outcome (3.1 per 1000). Predictive factors in the main model included maternal world region of origin, chronic medical conditions, parity, and obstetrical/perinatal issues – with moderate model discrimination (C-statistic 0.68, 95% CI 0.66–0.69). Among 333,435 parous women, the C-statistic was 0.71 (0.69–0.73) in the model using variables from the current (index) pregnancy as well as pre-pregnancy predictors and variables from any previous pregnancy. Conclusions A combination of factors ascertained early in pregnancy through a basic medical history help to identify women at risk for severe morbidity, who may benefit from targeted preventive and surveillance strategies including appropriate specialty-based antenatal care pathways. Further refinement and external validation of this model are warranted and can support evidence-based improvements in clinical practice. Supplementary Information The online version contains supplementary material available at 10.1186/s12884-021-04132-6.

Background Severe maternal morbidity (SMM) covers a range of conditions along the continuum to maternal death during pregnancy or within 42 days after delivery [1]. Maternal morbidity is a substantial public health concern [2] whose incidence is rising in Canada and the US [3]. These patterns are driven by multiple risk factors including delaying childbearing, use of assisted reproductive technologies, rising rates of obesity, and Caesarean delivery [4]. Because of the low prevalence of maternal mortality in many industrialized countries, data covering several years are required to compute precise estimates of prevalence and risk factors, thus complicating the use of maternal mortality as a population health indicator [2,5]. Consequently, SMM has received increasing attention as an indicator of perinatal health and obstetric care [6,7].
As the focus in industrialized countries such as Canada has shifted towards 'near miss' events as a means to improving the health and quality of care for pregnant women [5], prediction of SMM has been identified as a critical research gap in obstetrics [4]. Many maternal characteristics are known pre-conception or early in pregnancy and are strong risk factors for the development of SMM [2,8]. Therefore, a combination of such factors may reliably predict its onset, enabling evidencebased and rational early triage of high-risk women for enhanced surveillance and subspecialty-based care.
Advances in maternal morbidity risk prediction include a US obstetric comorbidity index [9], which was externally validated within a Canadian population, resulting in modest discrimination (C-statistic of 0.66, 95% confidence interval [CI] 0.65-0.67) [10]. That index included variables that both preceded, and were simultaneous with, the onset of SMM, making it a useful research tool for identifying the burden of morbidity but less so for clinical prediction. Others have developed models focused on specific subtypes of maternal morbidity, such as cardiovascular-related conditions [7]. Models predicting maternal mortality include the Collaborative Integrated Pregnancy High-dependency Estimate of Risk (CIPHER) model (C-statistic 0.82, 95% CI 0.81-0.84) and the Maternal Severity Index (C-statistic 0.83, 95% CI 0.80-0.85) [11], both developed among women either already critically ill or hospitalized, and mostly later in gestation.
Since SMM predominantly arises around birth or early postpartum [1], the ideal timeframe for prediction is before or early in pregnancy to facilitate effective preventive strategies such as referral to high-risk centres or shared-care antenatal care pathways [12,13]. Existing models do not enable these latter steps, nor do they account for important pre-pregnancy factors, such as maternal infertility and its treatment, which are associated with SMM [14]. Additionally, existing prediction efforts did not consider prior adverse pregnancy outcomes among parous women. We therefore undertook the current study to develop and internally validate a clinical prediction model of SMM, defined as a composite of maternal end-organ injury or death, using readily available factors ascertained pre-pregnancy and prior to 20 weeks' gestation in a population-based study in Ontario -Canada's most populous and multi-ethnic province.

Methods
The use of data in this project was authorized under section 45 of Ontario's Personal Health Information Protection Act, which does not require review by a Research Ethics Board. We followed the Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guideline for reporting of prediction studies [15].

Population and data sources
All women with a pregnancy lasting beyond 20 weeks' gestation, and who delivered within an Ontario hospital between April 1, 2006 and March 31, 2014, were identified within the Better Outcomes Registry & Network (BORN) databases [16]. Data beyond 2014 were not available in these datasets. The BORN registry captures over 99% of hospital births in the province, and has been validated for data completeness and accuracy [17,18]. We used the Registered Persons Database, the Immigration, Refugees and Citizenship Canada's Permanent Resident Database, the Ontario Health Insurance Plan (OHIP) outpatient claims database, and the Canadian Institute for Health Information (CIHI) Discharge Abstract Database to capture maternal demographics, pre-existing health conditions and diagnoses and procedures documented during a hospitalization (see Table S1 for variables and diagnostic codes used to develop the study cohort). The datasets were linked using unique encoded identifiers and analysed at ICES -a not-for-profit provincial including appropriate specialty-based antenatal care pathways. Further refinement and external validation of this model are warranted and can support evidence-based improvements in clinical practice.
Keywords: Women's health, Epidemiology, Obstetrics, Gynaecology research entity that houses a large network of health administrative databases (https:// www. ices. on. ca/).
We excluded ectopic pregnancies, pregnancies resulting in abortion or miscarriage or ending before 20 weeks' gestation. We randomly sampled one birth (live-or stillbirth) per woman to avoid potential within-person correlations among women with multiple pregnancies (Table S1; Figure S1).

Study outcomes
The primary composite outcome was maternal end-organ injury or death arising between 20 weeks' gestation and up to 42 days after the index birth hospital discharge date. The list of conditions used to define maternal end-organ injury was based on the model developed by Bateman [9] and validated by Metcalfe [10], comprising 20 diagnoses and procedures, and consistent with Canadian perinatal surveillance definitions for SMM and death [19][20][21] ( Table S1).
A secondary outcome was all-cause maternal mortality, from birth until 365 days postpartum, since previous work has shown a persistent increase in mortality risk beyond the early postpartum period [22,23].

Candidate predictors, variable selection, and coding
Demographic, medical and obstetric factors known to be associated with an increased risk of SMM were considered as candidate predictors. These included: estimated maternal age at conception (continuous, categorical, and squared terms); residential income quintile; world region of origin (Table S2), as a proxy for both maternal birthplace and ethnicity; attendance at a first-trimester prenatal care visit; pre-pregnancy body mass index (BMI); parity; multiple gestation; infertility; infertility treatment; placental disorders (e.g., placenta praevia, placenta accreta); and pre-existing medical conditions coded within 12 months before the estimated date of conception (Table S1). Substantial missing data were noted only for the variable pre-pregnancy BMI (63.8%). We tested models in which BMI was modelled as a continuous variable and where missing values were assigned the median BMI (24.2 kg/m 2 ). We further tested models in which BMI was divided into the following categories: < 18.5 kg/m 2 , 18.5-24.9 kg/m 2 (reference category), 25-29.9 kg/m 2 , > 30 kg/ m 2 , and missing. Certain categorical variables with a low frequency in the cohort were combined with other similar variables (e.g., pre-existing cardiovascular conditions; placental conditions and anomalies). Variables were also assessed for collinearity by checking the variance inflation factor (VIF), and where collinear (VIF > 5), the most commonly reported variable was selected [24].
In the model restricted to the sub-cohort of parous women, in addition to the above variables, we included complications coded in any previous pregnancy as predictors (Table S1).
Possible interactions between variables were assessed and included if statistically significant at alpha = 0.10 [25].

Statistical analysis Descriptive statistics
We used standardized differences to contrast births with and without the primary composite outcome of maternal end-organ injury or death, with a value > 0.10 indicating an important difference in baseline characteristics [26].

Model discrimination
Among the entire cohort, a logistic regression model was fit using the final selected variables to predict the primary composite outcome of maternal acute end-organ injury or death from 20 weeks' gestation until 42 days postpartum. A backward elimination method was applied for variable selection, with predictor evaluation based on a balance of the model's C-statistic, clinical influence, and statistical significance. For continuous predictor variables such as age in which non-linear associations with the outcome were observed, a quadratic (squared) term was added to the model. Model discrimination was expressed as a C-statistic and its 95% CI, as well as visual detection of a receiver operating curve (ROC). We considered a C-statistic of < 0.5 to be not useful, 0.5 to 0.6 poor, 0.6 to 0.7 moderate, and ≥ 0.7 as good [27].

Model internal validation
To arrive at an optimism-corrected C-statistic, we used a bootstrapping approach, with 500 bootstrap samples selected from the original cohort, with replacement [28] -an approach known to produce stable estimates with low risk of bias [29]. The optimism-corrected C-statistic was defined as the C-statistic from the original data minus the optimism value [30].

Model calibration
Model calibration was assessed by visual inspection of calibration plots of observed vs. expected probabilities of the outcome, where a 45-degree line denotes good calibration, and a slope of 1 indicates perfect agreement between observed and expected events [31].

Risk classification
We used a risk classification table and computed likelihood ratios (LRs) [32] with associated 95% CI to assess the main model's ability to stratify the population as low or high-risk. We divided the population into five groups of predicted probability: very low risk (< 1.5 per 1000), low risk (1.5 to 3 per 1000), intermediate risk (3 to 5 per 1000), high risk (5 to 15 per 1000) and very high risk (> 15 per 1000). These cut-offs were chosen based on the overall incidence of our primary outcome of 3.1 per 1000, which we assumed to reflect the risk among the majority of the cohort. Positive LRs of > 5 and > 10 were interpreted as moderately or very useful "rule-in" tests, while values between 0.2 and 0.5, and < 0.1 were considered moderately and very useful "rule-out" tests [33].

Funding
This study was supported by funding from the Canadian Institutes of Health Research (grant number 15139).

Results
After sampling one birth per woman from among 853,517 eligible births, the total cohort comprised 634,290 births ( Figure S1). The primary outcome of end-organ injury or death from 20 weeks' gestation up to 42 days postpartum occurred in 1969 women (3.1 per 1000), including 62 deaths (0.1 per 1000). Women who experienced the primary outcome were older, more likely to have a preexisting medical condition, and to have had infertility treatment ( Table 1).

Model discrimination and internal validation Overall cohort
In the overall cohort (n = 634,290), variables significantly associated with the composite outcome of maternal endorgan injury or death included maternal age, low income, world region of origin, high BMI, pre-existing medical conditions, and placental disorders (Table S3), which contributed to the final model. Attendance at a firsttrimester antenatal visit and parity were inversely associated with the composite outcome. The corresponding model C-statistic was 0.68 (95% CI 0.66-0.69) (Fig. 1). There was minimal overfitting of the model, with mean optimism of 0.0055 (95% CI 0.0050-0.0061), and an optimism-corrected C-statistic of 0.67 (95% CI 0.66-0.68). Model discrimination was unchanged when BMI was included, either as a categorical variable with "missing" as a separate category, or as a continuous variable imputed with the median value for BMI. We tested 300 pairwise interactions, of which 13 interactions were statistically significant. The main model including interaction terms resulted in similar model discrimination as the main model (C-statistic 0.69, 95% CI 0.68-0.70), however this model included unstable estimates. Therefore, the model without interaction terms was chosen as the most balanced and efficient model. All-cause mortality from birth until 365 days postpartum occurred in 194 women over the study time period (0.3 per 1000). The final multivariable model for all-cause mortality no longer retained world region of origin, parity, previous spontaneous abortion, and several medical comorbidities (Table S4). Major psychiatric conditions and alcohol and substance use newly emerged as predictors. The corresponding C-statistic was 0.70 (95% CI 0.66-0.74) ( Figure S2). However, this model was slightly over-fitted, and the optimism-corrected C-statistic was 0.67 (95% CI 0.63-0.71).
The risk classification table for the main model, dividing the cohort according to the five categories of predicted risk of acute end-organ injury or death (Table 3) demonstrated the capacity of this model to classify women who are at very low risk (−LR 0.41, 95% CI 0.33-0.52) and those at very high risk of the outcome (+LR 8.58, 95% CI 7.32-10.05), but was less useful in classifying women in intermediate risk categories.

Model fit and calibration
Visual inspection of the calibration plots in the entire cohort suggested good agreement between observed and expected events for the primary outcome, with slightly worse calibration for mortality ( Figure S3a-b). Among parous women, model calibration for maternal end-organ injury or death improved from the base model to models including variables measured pre-pregnancy and in a previous pregnancy (Figure S3c-e).

Main findings
We have shown that a model based on variables available pre-pregnancy and in early pregnancy can moderately discriminate women destined for a severely morbid event or death from those likely to have uncomplicated pregnancies. Predictive variables retained in the final models included demographic, obstetric, and other medical risk factors. Notably, attendance at the first trimester visit with a care provider -a measure of good prenatal care -was inversely associated with the risk of SMM. Inclusion of prior pregnancy factors, which have not been incorporated in previous predictive models for SMM, further enhanced model performance, in keeping with the importance of clinical obstetrical history. Our model displayed good calibration, indicating that a combination of routinely measured pre-pregnancy and early pregnancy factors can estimate the absolute risk of acute endorgan injury or death with reasonable accuracy. Using this model effectively increased the probability of identifying a very high-risk woman with this outcome by 40%, and reduced the probability in someone considered very low-risk by 20% [33], but was less useful in classifying women in intermediate risk categories. This suggests that additional clinical, laboratory, or paraclinical factors are needed to accurately predict morbidity in all women, and further, that a certain proportion of these events are truly sudden and unpredictable.

Strengths and limitations
The models in this study relied on information that is routinely known at the time of the first antenatal visit, using variables that were temporally remote from when most maternal morbid events arise -largely around the time of birth [1]. Moreover, our source population comprised all pregnancies from gestational week 20. However, our datasets had few routinely collected clinical measures, such as blood pressure and haemoglobin For women with a missing body mass index, or an implausible value < 10 or > 50 kg/m 2 , a median body mass index of 24.2 kg/m 2 was used g Women with a missing body mass index, or an implausible value < 10 or > 50 kg/m 2 , were removed h Includes chronic congestive heart failure, congenital heart disease, pulmonary hypertension, coronary artery disease, cardiac dysrhythmia, chronic rheumatic heart diseases, or non-incident stroke or myocardial infarction within the previous 365 days i Includes sickle cell disease and HIV or glucose concentrations, or first-trimester screening biomarkers. In the prediction of preterm preeclampsia, for example, a model that contained a combination of clinical and paraclinical variables (including placental biomarkers) performed better than with either set of variables in isolation [34]. BMI was incomplete in our dataset, as is common in most administrative data sources. However, the proportion in any given BMI category and those with missing values was not appreciably different among women with and without the outcome. Furthermore, there were few substantial differences in other baseline characteristics between those with missing vs. non-missing BMI (Table S8). Thus, while the contribution of BMI to the outcome may not have been well represented in our models, this unlikely changed the overall model performance. In addition, clinical practice around identification and management of SMM has evolved over time. It is plausible that the strength of different risk factors for SMM may have changed across the study period as well (e.g., use of lower-risk IVF strategies). However, we used a constant definition for the study outcomes, and any changes in clinical practice patterns would not affect the internal validity of our models. Prediction models are often used to estimate an individual's absolute risk of a serious adverse event that might be mitigated with the use of a particular therapy, while avoiding subjecting individuals at low predicted risk to potential harmful effects of such therapy [35]. In obstetrics, serious adverse events are rare, with limited options for targeted prevention. We acknowledge, therefore, the limitations afforded by the C-statistic to discriminate between individuals with and without a rare adverse event, in which a high false positive rate might be justified [36]. The LRs add clinical meaning to the model and serve as a foundation for what might be considered reasonable predictability of rare but catastrophic obstetric events. The high LR of the model for women with very high predicted risk despite the rarity of the outcome in this group speaks to the potential utility of the model as a screening tool.

Interpretation
The models in this study relied on information that is routinely known at the time of the first antenatal visit, and that is temporally remote from when most morbid events arise -around the time of birth [1]. Our main model shows the potential utility of harnessing data in early pregnancy to predict a variety of later adverse maternal outcomes. Consistent with previous research  on postnatal mortality [23], our model for all-cause mortality showed substance use, alcohol use, and psychiatric conditions to be significant predictors of death up to 365 days postpartum. SMM rates have stagnated within Western nations, yet evidence-based strategies to reduce their burden are lacking [1]. Despite the possibility for early identification and prevention of some forms of SMM, current practice guidelines do not incorporate recommendations for prediction of severe morbidity, and use narrow crude definitions to identify such events [37]. Further refinement of clinical prediction models and the eventual development of a clinical risk calculator may help to inform early triage of women for enhanced surveillance or referral to subspecialty care or shared-care antenatal pathways -decisions that at present rely principally on clinical judgment. In developing and refining our study's model in external cohorts, investigators should consider adding first-trimester placental biomarkers and other maternal biomarkers alongside routinely measured clinical variables, such as blood pressure and weight. The incorporation of such variables may facilitate prediction of the whole of severe morbidity as well as cause-specific outcomes, and better inform individualized and targeted prevention [38].

Conclusion
In conclusion, a model developed using pre-pregnancy and early pregnancy predictors available within administrative datasets had moderate prediction of maternal acute end-organ injury or death, and as such shows significant promise in the early clinical prediction of SMM. The addition of factors from a prior pregnancy among parous women slightly improved the model performance. Enhancement of these models, using direct clinical measures, and by external validation or using machine learning, is needed.
Additional file 1: Table S1. Variables used to define cohort entry and exclusion criteria, primary outcome, and variables considered for each clinical prediction model. Table S2. List of countries used to define World region of origin among immigrant women included in the study. Table S3. Adjusted odds ratio of maternal end-organ injury or death arising between 20 weeks' gestation and up to 42 days after birth, in association with variables measured pre-pregnancy and in the index pregnancy prior to 20 weeks' gestation. Table S4. Adjusted odds ratio of all-cause maternal death arising between birth and up to 365 days after birth, in association with variables measured pre-pregnancy and in the index pregnancy prior to 20 weeks' gestation.  maternal end-organ injury or death arising between 20 weeks' gestation and up to 42 days after birth in association with variables measured in the index pregnancy prior to 20 weeks' gestation. Table S6. Adjusted odds ratio of maternal end-organ injury or death arising between 20 weeks' gestation and up to 42 days after birth in association with variables measured in the index pregnancy prior to 20 weeks' gestation and prepregnancy. Table S7. Adjusted odds ratio of maternal end-organ injury or death arising between 20 weeks' gestation and up to 42 days after birth in association with variables measured in the index pregnancy prior to 20 weeks' gestation, pre-pregnancy, and in any previous pregnancy. Table S8. Baseline characteristics of the study population, according to missing BMI vs. non-missing BMI. Figure S1. Flow diagram of the creation of the study cohort. Figure S2. Receiver operating characteristic curve showing discrimination of the clinical prediction model for all-cause maternal death. Figure S3. a. Calibration plot of predicted (x-axis) and observed (y-axis) deciles of probability of acute end organ injury or death, entire cohort of 634,290 births. b. Calibration plot of predicted (x-axis) and observed (y-axis) deciles of probability of all-cause maternal death, entire cohort of 634,290 births. c. Calibration plot of predicted (x-axis) and observed (y-axis) deciles of probability of acute end organ injury or death, early index pregnancy factors, sub-cohort of 333,435 births among parous women. d. Calibration plot of predicted (x-axis) and observed (y-axis) deciles of probability of acute end organ injury or death, pre-pregnancy and early index pregnancy factors, sub-cohort of 333,435 births among parous women. e. Calibration plot of predicted (x-axis) and observed (y-axis) deciles of probability of acute end organ injury or death, pre-pregnancy, previous pregnancy, and early index pregnancy factors, sub-cohort of 333,435 births among parous women.