Skip to main content
Download PDF

Development and internal validation of a model predicting severe maternal morbidity using pre-conception and early pregnancy variables: a population-based study in Ontario, Canada

BMC Pregnancy and Childbirth volume 21, Article number: 679 (2021) Cite this article

Abstract

Background

Improvement in the prediction and prevention of severe maternal morbidity (SMM) - a range of life-threatening conditions during pregnancy, at delivery or within 42 days postpartum - is a public health priority. Reduction of SMM at a population level would be facilitated by early identification and prediction. We sought to develop and internally validate a model to predict maternal end-organ injury or death using variables routinely collected during pre-pregnancy and the early pregnancy period.

Methods

We performed a population-based cohort study using linked administrative health data in Ontario, Canada, from April 1, 2006 to March 31, 2014. We included women aged 18–60 years with a livebirth or stillbirth, of which one birth was randomly selected per woman. We constructed a clinical prediction model for the primary composite outcome of any maternal end-organ injury or death, arising between 20 weeks’ gestation and 42 days after the birth hospital discharge date. Our model included variables collected from 12 months before estimated conception until 19 weeks’ gestation. We developed a separate model for parous women to allow for the inclusion of factors from previous pregnancy(ies).

Results

Of 634,290 women, 1969 experienced the primary composite outcome (3.1 per 1000). Predictive factors in the main model included maternal world region of origin, chronic medical conditions, parity, and obstetrical/perinatal issues – with moderate model discrimination (C-statistic 0.68, 95% CI 0.66–0.69). Among 333,435 parous women, the C-statistic was 0.71 (0.69–0.73) in the model using variables from the current (index) pregnancy as well as pre-pregnancy predictors and variables from any previous pregnancy.

Conclusions

A combination of factors ascertained early in pregnancy through a basic medical history help to identify women at risk for severe morbidity, who may benefit from targeted preventive and surveillance strategies including appropriate specialty-based antenatal care pathways. Further refinement and external validation of this model are warranted and can support evidence-based improvements in clinical practice.

Peer Review reports

Background

Severe maternal morbidity (SMM) covers a range of conditions along the continuum to maternal death during pregnancy or within 42 days after delivery [1]. Maternal morbidity is a substantial public health concern [2] whose incidence is rising in Canada and the US [3]. These patterns are driven by multiple risk factors including delaying childbearing, use of assisted reproductive technologies, rising rates of obesity, and Caesarean delivery [4]. Because of the low prevalence of maternal mortality in many industrialized countries, data covering several years are required to compute precise estimates of prevalence and risk factors, thus complicating the use of maternal mortality as a population health indicator [2, 5]. Consequently, SMM has received increasing attention as an indicator of perinatal health and obstetric care [6, 7].

As the focus in industrialized countries such as Canada has shifted towards ‘near miss’ events as a means to improving the health and quality of care for pregnant women [5], prediction of SMM has been identified as a critical research gap in obstetrics [4]. Many maternal characteristics are known pre-conception or early in pregnancy and are strong risk factors for the development of SMM [2, 8]. Therefore, a combination of such factors may reliably predict its onset, enabling evidence-based and rational early triage of high-risk women for enhanced surveillance and subspecialty-based care.

Advances in maternal morbidity risk prediction include a US obstetric comorbidity index [9], which was externally validated within a Canadian population, resulting in modest discrimination (C-statistic of 0.66, 95% confidence interval [CI] 0.65–0.67) [10]. That index included variables that both preceded, and were simultaneous with, the onset of SMM, making it a useful research tool for identifying the burden of morbidity but less so for clinical prediction. Others have developed models focused on specific subtypes of maternal morbidity, such as cardiovascular-related conditions [7]. Models predicting maternal mortality include the Collaborative Integrated Pregnancy High-dependency Estimate of Risk (CIPHER) model (C-statistic 0.82, 95% CI 0.81–0.84) and the Maternal Severity Index (C-statistic 0.83, 95% CI 0.80–0.85) [11], both developed among women either already critically ill or hospitalized, and mostly later in gestation.

Since SMM predominantly arises around birth or early postpartum [1], the ideal timeframe for prediction is before or early in pregnancy to facilitate effective preventive strategies such as referral to high-risk centres or shared-care antenatal care pathways [12, 13]. Existing models do not enable these latter steps, nor do they account for important pre-pregnancy factors, such as maternal infertility and its treatment, which are associated with SMM [14]. Additionally, existing prediction efforts did not consider prior adverse pregnancy outcomes among parous women. We therefore undertook the current study to develop and internally validate a clinical prediction model of SMM, defined as a composite of maternal end-organ injury or death, using readily available factors ascertained pre-pregnancy and prior to 20 weeks’ gestation in a population-based study in Ontario – Canada’s most populous and multi-ethnic province.

Methods

The use of data in this project was authorized under section 45 of Ontario’s Personal Health Information Protection Act, which does not require review by a Research Ethics Board. We followed the Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guideline for reporting of prediction studies [15].

Population and data sources

All women with a pregnancy lasting beyond 20 weeks’ gestation, and who delivered within an Ontario hospital between April 1, 2006 and March 31, 2014, were identified within the Better Outcomes Registry & Network (BORN) databases [16]. Data beyond 2014 were not available in these datasets. The BORN registry captures over 99% of hospital births in the province, and has been validated for data completeness and accuracy [17, 18]. We used the Registered Persons Database, the Immigration, Refugees and Citizenship Canada’s Permanent Resident Database, the Ontario Health Insurance Plan (OHIP) outpatient claims database, and the Canadian Institute for Health Information (CIHI) Discharge Abstract Database to capture maternal demographics, pre-existing health conditions and diagnoses and procedures documented during a hospitalization (see Table S1 for variables and diagnostic codes used to develop the study cohort). The datasets were linked using unique encoded identifiers and analysed at ICES – a not-for-profit provincial research entity that houses a large network of health administrative databases (https://www.ices.on.ca/).

We excluded ectopic pregnancies, pregnancies resulting in abortion or miscarriage or ending before 20 weeks’ gestation. We randomly sampled one birth (live- or stillbirth) per woman to avoid potential within-person correlations among women with multiple pregnancies (Table S1; Figure S1).

Study outcomes

The primary composite outcome was maternal end-organ injury or death arising between 20 weeks’ gestation and up to 42 days after the index birth hospital discharge date. The list of conditions used to define maternal end-organ injury was based on the model developed by Bateman [9] and validated by Metcalfe [10], comprising 20 diagnoses and procedures, and consistent with Canadian perinatal surveillance definitions for SMM and death [19,20,21] (Table S1).

A secondary outcome was all-cause maternal mortality, from birth until 365 days postpartum, since previous work has shown a persistent increase in mortality risk beyond the early postpartum period [22, 23].

Candidate predictors, variable selection, and coding

Demographic, medical and obstetric factors known to be associated with an increased risk of SMM were considered as candidate predictors. These included: estimated maternal age at conception (continuous, categorical, and squared terms); residential income quintile; world region of origin (Table S2), as a proxy for both maternal birthplace and ethnicity; attendance at a first-trimester prenatal care visit; pre-pregnancy body mass index (BMI); parity; multiple gestation; infertility; infertility treatment; placental disorders (e.g., placenta praevia, placenta accreta); and pre-existing medical conditions coded within 12 months before the estimated date of conception (Table S1). Substantial missing data were noted only for the variable pre-pregnancy BMI (63.8%). We tested models in which BMI was modelled as a continuous variable and where missing values were assigned the median BMI (24.2 kg/m2). We further tested models in which BMI was divided into the following categories: < 18.5 kg/m2, 18.5–24.9 kg/m2 (reference category), 25–29.9 kg/m2, > 30 kg/m2, and missing. Certain categorical variables with a low frequency in the cohort were combined with other similar variables (e.g., pre-existing cardiovascular conditions; placental conditions and anomalies). Variables were also assessed for collinearity by checking the variance inflation factor (VIF), and where collinear (VIF > 5), the most commonly reported variable was selected [24].

In the model restricted to the sub-cohort of parous women, in addition to the above variables, we included complications coded in any previous pregnancy as predictors (Table S1).

Possible interactions between variables were assessed and included if statistically significant at alpha = 0.10 [25].

Statistical analysis

Descriptive statistics

We used standardized differences to contrast births with and without the primary composite outcome of maternal end-organ injury or death, with a value > 0.10 indicating an important difference in baseline characteristics [26].

Model discrimination

Among the entire cohort, a logistic regression model was fit using the final selected variables to predict the primary composite outcome of maternal acute end-organ injury or death from 20 weeks’ gestation until 42 days postpartum. A backward elimination method was applied for variable selection, with predictor evaluation based on a balance of the model’s C-statistic, clinical influence, and statistical significance. For continuous predictor variables such as age in which non-linear associations with the outcome were observed, a quadratic (squared) term was added to the model. Model discrimination was expressed as a C-statistic and its 95% CI, as well as visual detection of a receiver operating curve (ROC). We considered a C-statistic of < 0.5 to be not useful, 0.5 to 0.6 poor, 0.6 to 0.7 moderate, and ≥ 0.7 as good [27].

Model internal validation

To arrive at an optimism-corrected C-statistic, we used a bootstrapping approach, with 500 bootstrap samples selected from the original cohort, with replacement [28] – an approach known to produce stable estimates with low risk of bias [29]. The optimism-corrected C-statistic was defined as the C-statistic from the original data minus the optimism value [30].

Model calibration

Model calibration was assessed by visual inspection of calibration plots of observed vs. expected probabilities of the outcome, where a 45-degree line denotes good calibration, and a slope of 1 indicates perfect agreement between observed and expected events [31].

Risk classification

We used a risk classification table and computed likelihood ratios (LRs) [32] with associated 95% CI to assess the main model’s ability to stratify the population as low or high-risk. We divided the population into five groups of predicted probability: very low risk (< 1.5 per 1000), low risk (1.5 to 3 per 1000), intermediate risk (3 to 5 per 1000), high risk (5 to 15 per 1000) and very high risk (> 15 per 1000). These cut-offs were chosen based on the overall incidence of our primary outcome of 3.1 per 1000, which we assumed to reflect the risk among the majority of the cohort. Positive LRs of > 5 and > 10 were interpreted as moderately or very useful “rule-in” tests, while values between 0.2 and 0.5, and < 0.1 were considered moderately and very useful “rule-out” tests [33].

Funding

This study was supported by funding from the Canadian Institutes of Health Research (grant number 15139).

Results

After sampling one birth per woman from among 853,517 eligible births, the total cohort comprised 634,290 births (Figure S1). The primary outcome of end-organ injury or death from 20 weeks’ gestation up to 42 days postpartum occurred in 1969 women (3.1 per 1000), including 62 deaths (0.1 per 1000). Women who experienced the primary outcome were older, more likely to have a pre-existing medical condition, and to have had infertility treatment (Table 1).

Table 1 Baseline characteristics of the study population, according to whether a woman had the composite outcome of maternal end-organ injury or death between 20 weeks’ gestation and up to 42 days after birth. All data are shown as a number (%) unless otherwise stated
Full size table

The most frequent factors contributing to end-organ injury or death were acute heart failure (40.6%), need for assisted ventilation (29.2%), acute renal failure (12.0%) and shock (10.1%) (Table 2).

Table 2 Occurrence of maternal end-organ injury or death between 20 weeks’ gestation and up to 42 days after birth, and the ranking of the most prevalent morbidity indicators
Full size table

Model discrimination and internal validation

Overall cohort

In the overall cohort (n = 634,290), variables significantly associated with the composite outcome of maternal end-organ injury or death included maternal age, low income, world region of origin, high BMI, pre-existing medical conditions, and placental disorders (Table S3), which contributed to the final model. Attendance at a first-trimester antenatal visit and parity were inversely associated with the composite outcome. The corresponding model C-statistic was 0.68 (95% CI 0.66–0.69) (Fig. 1). There was minimal overfitting of the model, with mean optimism of 0.0055 (95% CI 0.0050–0.0061), and an optimism-corrected C-statistic of 0.67 (95% CI 0.66–0.68). Model discrimination was unchanged when BMI was included, either as a categorical variable with “missing” as a separate category, or as a continuous variable imputed with the median value for BMI. We tested 300 pairwise interactions, of which 13 interactions were statistically significant. The main model including interaction terms resulted in similar model discrimination as the main model (C-statistic 0.69, 95% CI 0.68–0.70), however this model included unstable estimates. Therefore, the model without interaction terms was chosen as the most balanced and efficient model.

Fig. 1
figure 1

Receiver operating characteristic curve showing discrimination of the clinical prediction model for maternal end-organ injury or death. Legend: Outcomes are those arising between 20 weeks’ gestation and 42 days after birth, using variables measured pre-pregnancy, and in the index pregnancy prior to 20 weeks’ gestation. Predictor variables and adjusted odds ratios are shown in Table S3. Analysed is the entire cohort of 634,290 births. C-statistic for Area Under the Curve = 0.68 (95% CI 0.66–0.69)

Full size image

All-cause mortality from birth until 365 days postpartum occurred in 194 women over the study time period (0.3 per 1000). The final multivariable model for all-cause mortality no longer retained world region of origin, parity, previous spontaneous abortion, and several medical comorbidities (Table S4). Major psychiatric conditions and alcohol and substance use newly emerged as predictors. The corresponding C-statistic was 0.70 (95% CI 0.66–0.74) (Figure S2). However, this model was slightly over-fitted, and the optimism-corrected C-statistic was 0.67 (95% CI 0.63–0.71).

The risk classification table for the main model, dividing the cohort according to the five categories of predicted risk of acute end-organ injury or death (Table 3) demonstrated the capacity of this model to classify women who are at very low risk (−LR 0.41, 95% CI 0.33–0.52) and those at very high risk of the outcome (+LR 8.58, 95% CI 7.32–10.05), but was less useful in classifying women in intermediate risk categories.

Table 3 Risk classification comparing predicted and observed risks of the outcome using five groups of predicted probability, and associated likelihood ratios in each group. Data are from main model predicting acute end organ injury or death from 20 weeks' gestation until 42 days after birth (n = 634,290)
Full size table

Sub-cohort of parous women

In the sub-cohort of 333,435 parous women, the aforementioned variables significantly associated with end-organ injury or death persisted, as did the addition of an unplanned Caesarean delivery and severe organ injury in a previous birth (Table S7). The C-statistic was 0.61 (95% CI 0.59–0.63) when limited to variables from the index pregnancy (Fig. 2a), rising to 0.69 (95% CI 0.67–0.70) after adding pre-pregnancy predictors (Fig. 2b), and 0.71 (95% CI 0.69–0.73) when including the variables from a previous pregnancy (Fig. 2c). We noted minimal overfitting for each model. With optimism-corrected C-statistics of 0.60 (95% CI 0.58–0.62), 0.68 (95% CI 0.66–0.70), and 0.70 (95% CI 0.69–0.72), respectively.

Fig. 2
figure 2

Receiver operating characteristic curve showing the discrimination of the clinical prediction model for maternal end-organ injury or death. Legend: Outcomes are those arising between 20 weeks’ gestation and 42 days after birth using variables measured in the index pregnancy prior to 20 weeks’ gestation (a); the index pregnancy prior to 20 weeks’ gestation and pre-pregnancy (b); the index pregnancy prior to 20 weeks’ gestation, pre-pregnancy, and in a previous pregnancy (c). Predictor variables and odds ratios are shown in Tables S5 (a), S6 (b), and S7 (c). Analysed is the cohort of 333,435 births among parous women. a C-statistic for Area Under the Curve = 0.61 (95% CI 0.59–0.63). b C-statistic for Area Under the Curve = 0.69 (95% CI 0.67–0.70). c C-statistic for Area Under the Curve = 0.71 (95% CI 0.69–0.73)

Full size image

Model fit and calibration

Visual inspection of the calibration plots in the entire cohort suggested good agreement between observed and expected events for the primary outcome, with slightly worse calibration for mortality (Figure S3a-b). Among parous women, model calibration for maternal end-organ injury or death improved from the base model to models including variables measured pre-pregnancy and in a previous pregnancy (Figure S3c-e).

Discussion

Main findings

We have shown that a model based on variables available pre-pregnancy and in early pregnancy can moderately discriminate women destined for a severely morbid event or death from those likely to have uncomplicated pregnancies. Predictive variables retained in the final models included demographic, obstetric, and other medical risk factors. Notably, attendance at the first trimester visit with a care provider – a measure of good prenatal care – was inversely associated with the risk of SMM. Inclusion of prior pregnancy factors, which have not been incorporated in previous predictive models for SMM, further enhanced model performance, in keeping with the importance of clinical obstetrical history. Our model displayed good calibration, indicating that a combination of routinely measured pre-pregnancy and early pregnancy factors can estimate the absolute risk of acute end-organ injury or death with reasonable accuracy. Using this model effectively increased the probability of identifying a very high-risk woman with this outcome by 40%, and reduced the probability in someone considered very low-risk by 20% [33], but was less useful in classifying women in intermediate risk categories. This suggests that additional clinical, laboratory, or paraclinical factors are needed to accurately predict morbidity in all women, and further, that a certain proportion of these events are truly sudden and unpredictable.

Strengths and limitations

The models in this study relied on information that is routinely known at the time of the first antenatal visit, using variables that were temporally remote from when most maternal morbid events arise – largely around the time of birth [1]. Moreover, our source population comprised all pregnancies from gestational week 20. However, our datasets had few routinely collected clinical measures, such as blood pressure and haemoglobin or glucose concentrations, or first-trimester screening biomarkers. In the prediction of preterm preeclampsia, for example, a model that contained a combination of clinical and paraclinical variables (including placental biomarkers) performed better than with either set of variables in isolation [34]. BMI was incomplete in our dataset, as is common in most administrative data sources. However, the proportion in any given BMI category and those with missing values was not appreciably different among women with and without the outcome. Furthermore, there were few substantial differences in other baseline characteristics between those with missing vs. non-missing BMI (Table S8). Thus, while the contribution of BMI to the outcome may not have been well represented in our models, this unlikely changed the overall model performance. In addition, clinical practice around identification and management of SMM has evolved over time. It is plausible that the strength of different risk factors for SMM may have changed across the study period as well (e.g., use of lower-risk IVF strategies). However, we used a constant definition for the study outcomes, and any changes in clinical practice patterns would not affect the internal validity of our models.

Prediction models are often used to estimate an individual’s absolute risk of a serious adverse event that might be mitigated with the use of a particular therapy, while avoiding subjecting individuals at low predicted risk to potential harmful effects of such therapy [35]. In obstetrics, serious adverse events are rare, with limited options for targeted prevention. We acknowledge, therefore, the limitations afforded by the C-statistic to discriminate between individuals with and without a rare adverse event, in which a high false positive rate might be justified [36]. The LRs add clinical meaning to the model and serve as a foundation for what might be considered reasonable predictability of rare but catastrophic obstetric events. The high LR of the model for women with very high predicted risk despite the rarity of the outcome in this group speaks to the potential utility of the model as a screening tool.

Interpretation

The models in this study relied on information that is routinely known at the time of the first antenatal visit, and that is temporally remote from when most morbid events arise – around the time of birth [1]. Our main model shows the potential utility of harnessing data in early pregnancy to predict a variety of later adverse maternal outcomes. Consistent with previous research on postnatal mortality [23], our model for all-cause mortality showed substance use, alcohol use, and psychiatric conditions to be significant predictors of death up to 365 days postpartum.

SMM rates have stagnated within Western nations, yet evidence-based strategies to reduce their burden are lacking [1]. Despite the possibility for early identification and prevention of some forms of SMM, current practice guidelines do not incorporate recommendations for prediction of severe morbidity, and use narrow crude definitions to identify such events [37]. Further refinement of clinical prediction models and the eventual development of a clinical risk calculator may help to inform early triage of women for enhanced surveillance or referral to subspecialty care or shared-care antenatal pathways – decisions that at present rely principally on clinical judgment. In developing and refining our study’s model in external cohorts, investigators should consider adding first-trimester placental biomarkers and other maternal biomarkers alongside routinely measured clinical variables, such as blood pressure and weight. The incorporation of such variables may facilitate prediction of the whole of severe morbidity as well as cause-specific outcomes, and better inform individualized and targeted prevention [38].

Conclusion

In conclusion, a model developed using pre-pregnancy and early pregnancy predictors available within administrative datasets had moderate prediction of maternal acute end-organ injury or death, and as such shows significant promise in the early clinical prediction of SMM. The addition of factors from a prior pregnancy among parous women slightly improved the model performance. Enhancement of these models, using direct clinical measures, and by external validation or using machine learning, is needed.

Availability of data and materials

The dataset from this study is held securely in coded form at ICES. While data sharing agreements prohibit ICES from making the dataset publicly available, access may be granted to those who meet pre-specified criteria for confidential access, available at www.ices.on.ca/DAS. The full dataset creation plan and underlying analytic code are available from the authors upon request, understanding that the computer programs may rely upon coding templates or macros that are unique to ICES and are therefore either inaccessible or may require modification.

References

  1. Ray JG, Park AL, Dzakpasu S, et al. Prevalence of SMM and factors associated with maternal mortality in Ontario, Canada. JAMA Netw Open. 2018;1(7):e184571.

    Article  Google Scholar 

  2. Lindquist A, Knight M, Kurinczuk JJ. Variation in SMM according to socioeconomic position: a UK national case-control study. BMJ Open. 2013;3(6):e002742.

    Article  Google Scholar 

  3. Dayan N, Fell DB, Guo Y, et al. SMM in women with high BMI in IVF and unassisted singleton pregnancies. Hum Reprod. 2018;33(8):1548–56.

    Article  CAS  Google Scholar 

  4. D'Alton ME, Bonanno CA, Berkowitz RL, et al. Putting the “M” back in maternal-fetal medicine. Am J Obstet Gynecol. 2013;208(6):442–8.

    Article  Google Scholar 

  5. van Roosmalen J, Zwart J. Severe acute maternal morbidity in high-income countries. Best Pract Res Clin Obstet Gynaecol. 2009;23(3):297–304.

    Article  Google Scholar 

  6. Firoz T, Chou D, von Dadelszen P, et al. Measuring maternal health: focus on maternal morbidity. Bull World Health Organ. 2013;91(10):794–6.

    Article  Google Scholar 

  7. Malhame I, Danilack VA, Raker CA, et al. Cardiovascular SMM in pregnant and postpartum women: development and internal validation of risk prediction models. BJOG. 2021;128(5):922–32.

  8. Aoyama K, Pinto R, Ray JG, et al. Association of maternal age with SMM and mortality in Canada. JAMA Netw Open. 2019;2(8):e199875.

    Article  Google Scholar 

  9. Bateman BT, Mhyre JM, Hernandez-Diaz S, et al. Development of a comorbidity index for use in obstetric patients. Obstet Gynecol. 2013;122(5):957–65.

    Article  Google Scholar 

  10. Metcalfe A, Lix LM, Johnson J-A, Currie G, Lyon AW, Bernier F, et al. Validation of an obstetric comorbidity index in an external population. BJOG. 2015;122:1748–55.

    Article  CAS  Google Scholar 

  11. Aoyama K, D'Souza R, Pinto R, et al. Risk prediction models for maternal mortality: a systematic review and meta-analysis. PLoS One. 2018;13(12):e0208563.

    Article  CAS  Google Scholar 

  12. Menard MK. Toward achieving risk-appropriate maternity care: maternal morbidity prediction. Obstet Gynecol. 2019;134(2):213–5.

    Article  Google Scholar 

  13. Macones GA. Understanding and reducing serious maternal morbidity: a step in the right direction. Obstet Gynecol. 2013;122(5):945–6.

    Article  Google Scholar 

  14. Dayan N, Joseph KS, Fell DB, et al. Infertility treatment and risk of SMM: a propensity score-matched cohort study. CMAJ. 2019;191(5):E118–27.

    Article  Google Scholar 

  15. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015;350:g7594.

    Article  Google Scholar 

  16. BORN Ontario. Legacy data elements. 2019. https://www.bornontario.ca/en/data/legacy-data-elements.aspx. Accessed 13 Sept 2019.

    Google Scholar 

  17. Dunn S, Lanes A, Sprague AE, et al. Data accuracy in the Ontario birth registry: a chart re-abstraction study. BMC Health Serv Res. 2019;19(1):1001.

    Article  Google Scholar 

  18. Dunn S, Bottomley J, Ali A, Walker M. 2008 Niday Perinatal Database quality audit: report of a quality assurance project. Chronic Dis Inj Can. 2011;32(1):32–42.

    CAS  PubMed  Google Scholar 

  19. Joseph KS, Liu S, Rouleau J, Kirby RS, Kramer MS, Sauve R, et al. SMM in Canada, 2003 to 2007: surveillance using routine hospitalization data and ICD-10CA codes. J Obstet Gynaecol Can. 2010;32(9):837–46.

    Article  CAS  Google Scholar 

  20. Main EK, Abreo A, McNulty J, et al. Measuring SMM: validation of potential measures. Am J Obstet Gynecol. 2016;214(5):643.e641–10.

    Article  Google Scholar 

  21. Roberts CL, Cameron CA, Bell JC, Algert CS, Morris JM. Measuring maternal morbidity in routinely collected health data: development and validation of a maternal morbidity outcome indicator. Med Care. 2008;46(8):786–94.

    Article  Google Scholar 

  22. Grigoriadis S, Wilton AS, Kurdyak PA, et al. Perinatal suicide in Ontario, Canada: a 15-year population-based study. CMAJ. 2017;189(34):E1085–92.

    Article  Google Scholar 

  23. Ray JG, Zipursky J, Park AL. Injury-related maternal mortality. Am J Obstet Gynecol. 2018;219(3):307–8.

    Article  Google Scholar 

  24. Hair JF, Black WC, Babin BJ, Anderson RE, Tatham RL. Multivariate data analysis. Upper Saddle River: Pearson Prentice Hall; 2006.

    Google Scholar 

  25. Steyerberg EW. Clinical prediction models: A practical approach to development, validation, and updating. New York: Springer; 2009.

  26. Austin PC. Using the standardized difference to compare the prevalence of a binary variable between two groups in observational research. Commun Stat Simul Comput. 2009;38(6):1228–34.

    Article  Google Scholar 

  27. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29.

    Article  CAS  Google Scholar 

  28. Steyerberg EW. Mathematics and statistics. Clinical prediction models: a practical approach to development, validation, and updating. New York: Springer; 2009.

    Book  Google Scholar 

  29. Steyerberg EW, Harrell FE Jr, Borsboom GJ, Eijkemans MJ, Vergouwe Y, Habbema JD. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 2001;54(8):774–81.

    Article  CAS  Google Scholar 

  30. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15(4):361–87.

    Article  Google Scholar 

  31. Steyerberg EW, Vickerrs AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for some traditional and novel measures. Epidemiology. 2010;21:128–38.

    Article  Google Scholar 

  32. Deeks JJ, Altman DG. Diagnostic tests 4: likelihood ratios. Br Med J. 2004;329(7458):168–9.

    Article  Google Scholar 

  33. McGee S. Simplfying likelihood ratios. J Gen Intern Med. 2002;17(8):647–50.

    Article  Google Scholar 

  34. Akolekar R, Syngelaki A, Poon L, Wright D, Nicolaides KH. Competing risks model in early screening for preeclampsia by biophysical and biochemical markers. Fetal Diagn Ther. 2013;33(1):8–15.

    Article  Google Scholar 

  35. Alba AC, Agoritsas T, Walsh M, et al. Discrimination and calibration of clinical prediction models: users’ guides to the medical literature. JAMA. 2017;318(14):1377–84.

    Article  Google Scholar 

  36. Romero-Brufau S, Huddleston JM, Escobar GJ, Liebow M. Why the C-statistic is not informative to evaluate early warning scores and what metrics to use. Crit Care (London, England). 2015;19:285.

    Article  Google Scholar 

  37. American College of O, Gynecologists, the Society for Maternal-Fetal M, Kilpatrick SK, Ecker JL. SMM: screening and review. Am J Obstet Gynecol. 2016;215(3):B17–22.

    Article  Google Scholar 

  38. Davidson AJ, Park AL, Ray JG. Navigating SMM using big data: green, yellow, and red flags for researchers. Obstet Med. 2019;12(3):105–6.

    Article  Google Scholar 

Download references

Acknowledgements

Dr. Dayan would like to acknowledge Dr. Ugochinyere Ukah for review of the manuscript and suggestions for model refinement.

Funding

This study was supported by funding from the Canadian Institutes of Health Research (grant number 15139). The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. Dr. Dayan is supported by the Fondation de Recherche du Québec – Santé (FRQS) Chercheur Boursier Clinicien award. This study was also supported by ICES, which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). Parts of this material are based on data and information compiled and provided by MOHLTC and CIHI. The analyses, conclusions, opinions and statements expressed herein are solely those of the authors and do not reflect those of the funding or data sources; no endorsement is intended or should be inferred. This Study is based in part on data provided by Better Outcomes Registry and Network (“BORN”), part of the Children’s Hospital of Eastern Ontario. The interpretation and conclusions contained herein do not necessarily represent those of BORN Ontario. Parts or whole of this material are based on data and/or information compiled and provided by Immigration, Refugees and Citizenship Canada (IRCC) current to May 2017.  However, the analyses, conclusions, opinions and statements expressed in the material are those of the author(s), and not necessarily those of IRCC.

Author information

Authors and Affiliations

  1. Department of Medicine and Research Institute, McGill University Health Centre, 5252 de Maisonneuve West, 2B.40, Montreal, QC, H4A 3S5, Canada

    Natalie Dayan & Gabriel D. Shapiro

  2. Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Purvis Hall, 1020 Pine Ave West, Montreal, QC, H3A 1A2, Canada

    Natalie Dayan, Gabriel D. Shapiro & Olga Basso

  3. ICES, 2075 Bayview Ave, Toronto, ON, M4N 3M5, Canada

    Jin Luo, Jun Guan, Alison L. Park & Joel G. Ray

  4. School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Centre for Practice-Changing Research Building, Room L-1154, 401 Smyth Road, Ottawa, ON, K1H 8L1, Canada

    Deshayne B. Fell

  5. Departments of Medicine and Obstetrics and Gynecology, University of Toronto, 123 Edward St., suite 1200, Toronto, ON, M5G 1E2, Canada

    Carl A. Laskin & Joel G. Ray

  6. TRIO Fertility, 655 Bay St, Toronto, ON, M5G 2K4, Canada

    Carl A. Laskin

Authors
  1. Natalie Dayan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gabriel D. Shapiro
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jin Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jun Guan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Deshayne B. Fell
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Carl A. Laskin
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Olga Basso
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Alison L. Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Joel G. Ray
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

ND, AP and JGR designed the study and interpreted analyses. ND wrote the manuscript. JL and JG assisted with study design and cohort creation and performed data analyses. GDS contributed to study design and data analysis interpretation. DBF, CL, OB, and AP assisted with study design and data interpretation. All authors revised the manuscript critically for important intellectual content, approved the version to be published and agreed to be accountable for all aspects of the work.

Corresponding author

Correspondence to Natalie Dayan.

Ethics declarations

Ethics approval and consent to participate

The use of data in this project was authorized under section 45 of Ontario’s Personal Health Information Protection Act, which does not require review by a Research Ethics Board, or individual patient consent. ICES approved the study and all analyses of raw data were conducted at ICES. No other administrative permissions were required.

Consent for publication

Not applicable.

Competing interests

The authors declare they have no conflicts of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Table S1.

Variables used to define cohort entry and exclusion criteria, primary outcome, and variables considered for each clinical prediction model. Table S2. List of countries used to define World region of origin among immigrant women included in the study. Table S3. Adjusted odds ratio of maternal end-organ injury or death arising between 20 weeks’ gestation and up to 42 days after birth, in association with variables measured pre-pregnancy and in the index pregnancy prior to 20 weeks’ gestation. Table S4. Adjusted odds ratio of all-cause maternal death arising between birth and up to 365 days after birth, in association with variables measured pre-pregnancy and in the index pregnancy prior to 20 weeks’ gestation. Table S5. Adjusted odds ratio of maternal end-organ injury or death arising between 20 weeks’ gestation and up to 42 days after birth in association with variables measured in the index pregnancy prior to 20 weeks’ gestation. Table S6. Adjusted odds ratio of maternal end-organ injury or death arising between 20 weeks’ gestation and up to 42 days after birth in association with variables measured in the index pregnancy prior to 20 weeks’ gestation and pre-pregnancy. Table S7. Adjusted odds ratio of maternal end-organ injury or death arising between 20 weeks’ gestation and up to 42 days after birth in association with variables measured in the index pregnancy prior to 20 weeks’ gestation, pre-pregnancy, and in any previous pregnancy. Table S8. Baseline characteristics of the study population, according to missing BMI vs. non-missing BMI. Figure S1. Flow diagram of the creation of the study cohort. Figure S2. Receiver operating characteristic curve showing discrimination of the clinical prediction model for all-cause maternal death. Figure S3. a. Calibration plot of predicted (x-axis) and observed (y-axis) deciles of probability of acute end organ injury or death, entire cohort of 634,290 births. b. Calibration plot of predicted (x-axis) and observed (y-axis) deciles of probability of all-cause maternal death, entire cohort of 634,290 births. c. Calibration plot of predicted (x-axis) and observed (y-axis) deciles of probability of acute end organ injury or death, early index pregnancy factors, sub-cohort of 333,435 births among parous women. d. Calibration plot of predicted (x-axis) and observed (y-axis) deciles of probability of acute end organ injury or death, pre-pregnancy and early index pregnancy factors, sub-cohort of 333,435 births among parous women. e. Calibration plot of predicted (x-axis) and observed (y-axis) deciles of probability of acute end organ injury or death, pre-pregnancy, previous pregnancy, and early index pregnancy factors, sub-cohort of 333,435 births among parous women.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dayan, N., Shapiro, G.D., Luo, J. et al. Development and internal validation of a model predicting severe maternal morbidity using pre-conception and early pregnancy variables: a population-based study in Ontario, Canada. BMC Pregnancy Childbirth 21, 679 (2021). https://doi.org/10.1186/s12884-021-04132-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12884-021-04132-6

Keywords

BMC Pregnancy and Childbirth

ISSN: 1471-2393

Contact us