- Research article
- Open Access
Prediction of pre-eclampsia in nulliparous women using routinely collected maternal characteristics: a model development and validation study
BMC Pregnancy and Childbirth volume 20, Article number: 23 (2020)
Guidelines recommend identifying in early pregnancy women at elevated risk of pre-eclampsia. The aim of this study was to develop and validate a pre-eclampsia risk prediction model for nulliparous women attending routine antenatal care “the Western Sydney (WS) model”; and to compare its performance with the National Institute of Health and Care Excellence (NICE) risk factor-list approach for classifying women as high-risk.
This retrospective cohort study included all nulliparous women who gave birth in three public hospitals in the Western-Sydney-Local-Health-District, Australia 2011–2014. Using births from 2011 to 2012, multivariable logistic regression incorporated established maternal risk factors to develop and internally validate the WS model. The WS model was then externally validated using births from 2013 to 2014, assessing its discrimination and calibration. We fitted the final WS model for all births from 2011 to 2014, and compared its accuracy in predicting pre-eclampsia with the NICE approach.
Among 12,395 births to nulliparous women in 2011–2014, there were 293 (2.4%) pre-eclampsia events. The WS model included: maternal age, body mass index, ethnicity, multiple pregnancy, family history of pre-eclampsia, autoimmune disease, chronic hypertension and chronic renal disease. In the validation sample (6201 births), the model c-statistic was 0.70 (95% confidence interval 0.65–0.75). The observed:expected ratio for pre-eclampsia was 0.91, with a Hosmer-Lemeshow goodness-of-fit test p-value of 0.20. In the entire study sample of 12,395 births, 374 (3.0%) women had a WS model-estimated pre-eclampsia risk ≥8%, the pre-specified risk-threshold for considering aspirin prophylaxis. Of these, 54 (14.4%) developed pre-eclampsia (sensitivity 18% (14–23), specificity 97% (97–98)). Using the NICE approach, 1173 (9.5%) women were classified as high-risk, of which 107 (9.1%) developed pre-eclampsia (sensitivity 37% (31–42), specificity 91% (91–92)). The final model showed similar accuracy to the NICE approach when using lower risk-threshold of ≥4% to classify women as high-risk for pre-eclampsia.
The WS risk model that combines readily-available maternal characteristics achieved modest performance for prediction of pre-eclampsia in nulliparous women. The model did not outperform the NICE approach, but has the advantage of providing individualised absolute risk estimates, to assist with counselling, inform decisions for further testing, and consideration of aspirin prophylaxis.
Antenatal guidelines recommend routine risk assessment for pre-eclampsia in early pregnancy and low dose aspirin prophylaxis for women at elevated risk [1,2,3,4]. However, approaches for using established maternal risk factors to classify a woman’s risk of pre-eclampsia vary; and Australian guidelines do not offer an explicit approach [1, 2].
The National Institute of Health and Care Excellence (NICE) in the United Kingdom lists moderate and high-risk factors for pre-eclampsia and recommends prophylaxis for women with one or more high-risk factors, or two or more moderate-risk factors . The listed risk factors are readily available maternal characteristics, so this approach has the practical advantage of being widely accessible, including in low-resource settings. However, its predictive performance has not yet been adequately validated in Australian women. Internationally, validation studies of the NICE approach have reported poor to moderate predictive performance [5,6,7,8,9,10,11]. One Australian validation study has reported good sensitivity for prediction of pre-eclampsia requiring delivery before 37 weeks, but was limited by a small sample size (n = 543, sensitivity 75% (95% confidence interval (CI) 35–97%), false positive rate 22%) .
A limitation of risk factor list approaches such as the NICE approach is that common risk factors such as older maternal age and higher body mass index (BMI) are dichotomised as ‘present’ or ‘absent’ ignoring any relationship between the level of these factors and pre-eclampsia risk. A risk prediction model that combines maternal risk factors, and includes all informative numerical values, has the potential advantage of providing an individualised estimate of pre-eclampsia risk that can assist patient counselling and inform clinical management decisions, rather than limiting categorisation to high-risk versus not high-risk. Although there has been a major research effort to develop risk prediction models for pre-eclampsia in early pregnancy, this research has largely focused on the use of specialised tests such as uterine artery Doppler and serum biomarkers  which are not available in low-resource settings, nor routinely used in all public antenatal care clinics in Australia.
Globally, pre-eclampsia is reported in 1–8% of pregnancies [14,15,16,17]. In Australia, the average rate of pre-eclampsia was 3.3% of mothers who gave birth between 2000 and 2008, declining from 4.6% in 2000 to 2.3% in 2008 . Internationally, nulliparity is classified as a moderate-risk factor [3, 4]. However, in a recent Australian study, we observed the incidence of pre-eclampsia was lower among nulliparous women than all women (parous and nulliparous) (2.5% vs 3.5% respectively) .
A woman’s prior pregnancy history of pre-eclampsia is one of the strongest risk factors for pre-eclampsia . More challenging, is identifying women at high-risk of pre-eclampsia in their first pregnancy. Accurate identification and appropriate management of this group has the potential to provide clinical benefits for current and subsequent pregnancies. However, few studies of pre-eclampsia risk assessment tools have targeted this group.
The aims of this study are to develop and validate a pre-eclampsia risk prediction model for nulliparous women that can be used at the first antenatal visit using routinely collected maternal characteristics; and compare its performance with the NICE approach to inform the development of an Australian strategy for pre-eclampsia risk assessment and prevention.
Study design, setting and data source
We conducted a multi-hospital retrospective cohort study of nulliparous women giving birth between 1 January 2011 and 31 December 2014 at three public hospitals (Auburn, Blacktown/Mount-Druitt and Westmead) in the Western Sydney Local Health District (WSLHD). We included all nulliparous women with no previous pregnancies in the study sample. Study data were extracted from the ObstetriX database held by the hospital maternity units. The ObstetriX database collects information for all women attending their first antenatal visit to the discharge of mothers and their babies from the hospital .
We excluded women with missing information for pre-eclampsia, parity or candidate risk factors for pre-eclampsia. We also excluded women who were prescribed antiplatelet therapy in the first trimester given the effectiveness of these agents for preventing pre-eclampsia .
The primary outcome was the development of pre-eclampsia of any severity or timing. During the study period, pre-eclampsia was defined as hypertension with new onset of significant proteinuria ≥20 weeks’ gestation . Secondary outcomes were early-onset pre-eclampsia (requiring delivery < 34 weeks’ gestation) and preterm pre-eclampsia (requiring delivery < 37 weeks’ gestation).
We extracted maternal socio-demographic characteristics (age, country-of-birth, primary language spoken at home and socioeconomic status classified from postcode using the Index of Relative Socio-economic Advantage and Disadvantage from the Socio-Economic Indexes for Areas (SEIFA)) , risk factors for pre-eclampsia (listed below); and study outcomes from the ObstetriX database.
For development of the Western Sydney (WS) risk model, we selected 12 candidate risk factors: maternal age, body mass index (BMI), autoimmune disease, chronic hypertension, chronic renal disease, diabetes mellitus (type 1 or 2), multiple (multi-fetal) pregnancy, family history of pre-eclampsia, conception method, ethnicity, socio-economic status, and smoking status. These candidate risk factors were identified from the antenatal guidelines [1,2,3,4]; with the addition of conception methods and smoking status which were identified from a systematic review of published risk models . We categorised ethnicity into two groups based on country of birth and primary language spoken at home (Australian/New Zealand-born English speakers; immigrants and non-English speakers). We categorised socioeconomic status into two groups using the SEIFA index (most disadvantaged SEIFA 1–2; most advantaged SEIFA 3–5).
For validation of NICE approach, we classified women with ≥1 high-risk factors or ≥ 2 moderate-risk factors as meeting the criteria of high-risk for aspirin prophylaxis , and refer to this group herein as “screen-positive”. All NICE-listed risk factors relevant for nulliparous women are collected in the ObstetriX database.
Women with missing values for study variables were excluded from the analysis requiring that variable.
We assessed the distribution of risk factors measured as continuous variables (age, BMI) visually by plotting a probability distribution curve. We performed a descriptive analysis of maternal risk factors by assessing the frequency of categorical variables as a percentage in all women, then separately for women who developed pre-eclampsia and those that did not.
Model development and validation
We split the study sample into two groups for model development and temporal validation by year of infant birth (model development sample 2011–2012, validation sample 2013–2014). For model development, we used a two-stage approach. First, to optimize prediction of pre-eclampsia from age, BMI and other NICE-listed moderate-risk factors we developed a WS ‘base’ model by excluding women with NICE-listed high-risk factors (autoimmune disease, chronic hypertension, chronic renal disease and diabetes (type 1 or 2)). The approach optimizes the model for use for the large majority of women who do not have high-risk factors; and would be sufficient in settings where women with high-risk factors are already referred for further testing and management. Second, we developed a WS ‘full’ model for use in all women, by introducing women with high-risk factors into the development sample, retaining the base model risk score, and estimating coefficients for the high-risk factors. We internally validated the model in the development sample then externally validated it in the validation sample to assess the potential for model overfitting. If the model fit was satisfactory, we planned to refit the model predictors in the entire study sample to develop a WS ‘final’ model. Further details of these analyses are given below.
WS base model
To develop the WS base model, we included the following candidate predictors in a multivariable regression model: maternal age in years, BMI in kg/m2, socioeconomic status (high vs low), conception method (assisted, by use of medications such as clomiphene or fertilization procedures including intrauterine insemination, in-vitro fertilization and intracytoplasmic sperm injection, vs natural conception), smoking status (current smokers vs non-smokers), multiple pregnancy (yes vs no); and family history of pre-eclampsia (yes vs no), and ethnicity (Australian/New Zealand-born English-speakers vs immigrants and non-English speakers).
To consider how to deal with the factors measured on a continuous scale (maternal age, BMI) in the model, we graphically examined their relationship with logit pre-eclampsia using a cubic splines approach. We assessed each factor and possible interactions between factors such as maternal age and multiple pregnancy by inspecting their effect size and p-value. We manually excluded factors that did not contribute to the model. We then developed the WS base model using the final predictors with no further stepwise procedures.
We performed internal validation of the model using the bootstrapping sampling technique to assess potential overfitting of the regression coefficients . The mean c-statistic (corresponds to the area-under-the curve (AUC)) of the bootstrapping models was compared with the WS base model using the following formula: AUC = 0.5*(Dxy + 1), where: Dxy is Somer’s D. A well fitted model will report minimal optimism. We planned to adjust the regression coefficients by the resulting shrinkage factor if required .
To externally validate the base model, we applied the model algorithm in the validation sample. As described above, the base model was developed use in women without high-risk factors, thus we excluded women with high-risk factors from this analysis. We calculated the predicted probability of pre-eclampsia for each individual woman by calculating the (log odds (Y)) and the odds ratio (ExpY) for pre-eclampsia and using the following equation: Probability = odds/1 + odds and presented the distribution of predicted probabilities in a histogram. We assessed model discrimination by calculating the AUC and 95% CI. We assessed model calibration in this sample using the Hosmer-Lemeshow goodness-of-fit test, with p-value < 0.05 indicating poor calibration . We also calculated the ratio of observed: expected pre-eclampsia events and graphically assessed calibration by plotting observed risks on the y-axis against predicted risks on the x-axis for subgroups of patients categorized by their predicted probabilities (1- < 2%, 2- < 3%, 3- < 4%, 4- < 5%, 5- < 8%, ≥8%) .
WS full model
To develop the WS full model, we introduced women with NICE-listed high-risk factors (autoimmune disease, chronic hypertension, chronic renal disease, diabetes) into the model development sample. We developed a multivariable regression model in this sample by retaining the WS base model risk score (Y) and adding the four high-risk factors listed above as additional predictors. We manually excluded high-risk factors that were not strongly or statistically significantly associated with pre-eclampsia. We followed the same approach outlined above for the base model to undertake internal and external validation of the full model to assess potential model overfitting.
WS final model
After assessment of over-fitting and calibration of the model in the validation sample, we refitted the WS base and full model in the entire study sample to develop the final WS model. First, we fitted the WS base model in women without high-risk factors. We retained the base model risk score and refitted the WS final model in the entire study sample to estimate the ß-coefficients for the high-risk factors and a new intercept. We presented the intercept and beta (log odd ratio) estimates and 95% CI for the intercept and each predictor.
Given the model is intended to be used to provide pre-eclampsia risk estimates to inform clinical decisions, we also assessed model sensitivity (95% CI), specificity (95% CI), positive predictive value (PPV, 95% CI), negative predictive value (NPV, 95% CI), positive likelihood ratio (LR) and negative LR to predict pre-eclampsia at specified cut-points determined by the risk thresholds for classifying high- vs low-risk. For our primary analysis, we used ≥8% as the risk threshold to classify high-risk as recommended by the United States Preventive Services Task Force (USPSTF) for commencing aspirin prophylaxis based on the prevalence of pre-eclampsia in trials demonstrating the effectiveness of aspirin  and from a publication recommending a 6–10% risk threshold for informing aspirin decisions . We also examined the final model performance at 2, 3, 4, 5 and 8% risk thresholds. We also reported model sensitivity at 5 and 10% fixed false positive rates (FPRs) to allow comparison with published models identified from our previous systematic review . For these analyses, we classified women as ‘true positive’ if they had a model-predicted risk above the cut-point and developed pre-eclampsia; false positive (predicted risk at/above cut-point and no pre-eclampsia); true negative (predicted risk below cut-point, no pre-eclampsia) or false negative (predicted risk below cut-point and pre-eclampsia).
In a secondary analysis, we assessed the discrimination of the WS final model to predict early-onset pre-eclampsia and preterm pre-eclampsia by estimating the AUC and sensitivity and specificity at ≥8% risk threshold in the entire study sample.
Model comparison with NICE approach
We compared the performance of the WS base and final models with the NICE approach by assessing the sensitivity and specificity, PPV, NPV, positive LR and negative LR of the NICE approach to predict pre-eclampsia in women without high-risk factors for comparison with the WS base model; and all women for comparison with the final model. For these comparisons, we assessed model sensitivity by fixing model specificity at the same level as the NICE approach. We report the model risk threshold that corresponds to this specificity level. For both the WS final model and NICE approach, we also calculated the number needed to treat (NNT) and the number needed to screen (NNS)  to avoid one pre-eclampsia event under a strategy where women classified as high-risk are recommended aspirin. For each approach, the NNT was calculated by applying a RR reduction of 10% for aspirin reported from the Perinatal Antiplatelet Review of International Studies (PARIS) individual participant data meta-analysis of randomized controlled trials  to the ‘baseline’ risk of pre-eclampsia observed for women classified as high-risk. The NNS was calculated by dividing the NNT by the proportion of pregnant women who were classified as high-risk using the approach.
We performed a secondary analysis to assess the performance of the NICE approach for predicting preterm versus term pre-eclampsia (delivery ≥37 weeks’ gestation) and early-onset versus late-onset pre-eclampsia (delivery ≥34 weeks’ gestation) by estimating the OR and 95% CI using multinomial logistic regression and reporting a p-value for the Wald Chi Square test for the hypothesis of no difference in approach performance between the pre-eclampsia subgroups (preterm versus term; and early-onset versus late-onset pre-eclampsia).
We used SPSS version 25 and SAS version 9.3 statistical software and R for all analyses. The R rms package was used for model internal validation (bootstrapping). A p-value of < 0.05 was regarded as statistically significant for all analyses.
We created an Excel spreadsheet to present the WS final model as a risk prediction calculator that can be used in the clinic to provide women with an individualised estimate of their probability of pre-eclampsia . We followed the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines to report our methods and findings .
A total of 12,793 nulliparous women gave birth in the three hospital sites during the study period. After exclusion of 358 women with missing information on risk factors and 40 women who received aspirin in the first trimester, 12,395 women were included for model development (n = 6194) and validation (n = 6201) (Fig. 1).
Participant characteristics are summarized in Table 1. Maternal age ranged from 14 to 46 years (mean 27.1, SD 4.9). Overall, 802 (6.5%) women were ≥ 35 years, 1494 (12.1%) were obese (BMI at first visit ≥30 kg/m2) and 308 (2.5%) had a multiple pregnancy. Two-thirds of women were immigrants or non-English speakers.
Pre-eclampsia incidence was 2.4% (n = 293). Forty-six (0.4%) women had early-onset pre-eclampsia and 115 (0.9%) women had preterm pre-eclampsia.
WS base model
After exclusion of 133 women with high-risk factors, the WS base model was developed in 6061 women. Age was modelled as a continuous spline linear from 27 years; and BMI was modelled as a continuous spline linear from 26.3 kg/m2 (Additional file 1: Figure S1). After inspection of effect size and p-value for each factor, three factors (conception method, smoking and socioeconomic status) were manually removed. The five factors included in the base model were: maternal age, BMI, ethnicity, multiple pregnancy, and family history of pre-eclampsia (Additional file 2: Table S1).
Internal validation by bootstrapping sampling indicated very small optimism (Dxy 0.009) (Additional file 2: Table S2). The optimism corrected performance estimate of Dxy was 0.3149, which corresponds to an AUC of 0.66, which was similar to the apparent model performance (AUC 0.66).
After exclusion of 137 women with high-risk factors, 6064 women were included in the external validation sample for the base model. In this sample, the AUC was 0.68 (0.62–0.73) indicating modest discrimination. Base model predictions in the validation sample ranged between 1 and 49% (Interquartile range (IQR): 1.2–2.2%, Additional file 1: Figure S2). Model calibration was good (Hosmer and Lemeshow goodness-of-fit test X2 = 6.87; p = 0.44); observed:expected ratio of pre-eclampsia events = 0.91. The calibration plot showed an acceptable level of calibration, however, at predicted probabilities for pre-eclampsia higher than 8%, it overestimated the risk of pre-eclampsia (Fig. 2).
WS full model
The full model was built in the entire model development sample (n = 6194). Of the four high-risk factors considered, diabetes (type 1 or 2) was eliminated from the model. The eight factors included in the full model were: maternal age, BMI, ethnicity, multiple pregnancy, and family history of pre-eclampsia, autoimmune disease, chronic hypertension and chronic renal disease (Additional file 2: Table S1).
Internal validation of the full model by bootstrapping sampling indicated very small optimism (Dxy 0.0087) (Additional file 2: Table S2). The optimism corrected performance estimate of Dxy was 3.716000e-01, which corresponds to an AUC of 0.69 which is similar to the apparent model performance (AUC 0.69).
In the external validation sample of 6201 women, the AUC was 0.70 (0.65–0.75) indicating good discrimination. Full model predictions in the validation sample ranged between 1 and 86% (IQR: 1.3–2.3%, Additional file 1: Figure S3). Model calibration was good (Hosmer and Lemeshow goodness-of-fit test X2 = 9.90; p = 0.20); observed:expected ratio of pre-eclampsia events = 0.91. The calibration plot showed an acceptable level of calibration, although as observed for the base model, at predicted probabilities for pre-eclampsia higher than 8%, it overestimated the risk of pre-eclampsia (Fig. 3).
WS final model
Given the model did not demonstrate over-fitting and was adequately calibrated in the validation sample, the model was refitted in the entire study sample to produce the WS final model.
When applied to the entire study sample, the AUC was 0.70 (0.66–0.73) indicating good discrimination (Additional file 1: Figure S4). The WS final model classified 374 (3%) of women at ≥8% risk of pre-eclampsia (Table 2). At this risk threshold, 54 (14.4%) women develop pre-eclampsia with model sensitivity 18% (14–23%), specificity 97% (97–98%), PPV 14% (11–18%), and NPV 98.0% (97.8–98.3%) The NNT was 69 and NNS was 2295. The performance of the model at lower risk thresholds is shown in Table 2. At fixed 5% FPR, corresponding to a 5.3% risk threshold, the model classified 6% of women to be at high-risk of pre-eclampsia and the sensitivity was 30% (25–36). At a fixed 10% FPR, corresponding to a 3.5% risk threshold, the model classified 11% of women to be at high-risk of pre-eclampsia and the sensitivity was 40% (95% CI 35–46).
The WS final model can be used to calculate a woman’s probability of developing pre-eclampsia as follows:
Y = − 7.786 + 0.052 * maternal age in years from age 27 years + 0.078 * BMI in kg/m2 from 26.3 kg/m2 + 0.525 if Australian/New Zealand born English-speaker + 1.318 if multiple pregnancy + 1.740 if family history of pre-eclampsia + 1.512 if autoimmune disease + 1.545 if chronic hypertension + 1.494 if chronic renal disease.
Odds = Exp Y (final prediction score)
Pre-eclampsia probability = Odds / (1 + Odds)
A ‘WS pre-eclampsia risk prediction tool’ has been created as an Excel spreadsheet that can be used in the clinic to perform these calculations automatically using information entered about a woman’s risk factors (Additional file 3). To illustrate, if a 24 year-old Australian nulliparous woman presents for her first antenatal visit with a BMI of 25 kg/m2 and no other risk factor, inputting this value into the Excel spreadsheet gives an estimate of the probability of pre-eclampsia of 1.7% (as shown in Fig. 4). If the same woman also has a family history of pre-eclampsia, addition of this information into the tool gives a revised probability of pre-eclampsia of 8.9%.
WS model prediction of early-onset and preterm pre-eclampsia
For prediction of early-onset pre-eclampsia, the WS final model had an AUC of 0.72 (95% CI 0.63–0.81). At the ≥8% risk threshold, 17 (4.5%) women developed pre-eclampsia with a model sensitivity of 37% (25–51%) and specificity 97.1% (96.8–97.4%) (Additional file 2: Table S3). At a fixed 5% FPR, corresponding to a 5.6% risk threshold, model sensitivity was 43% (95% CI 30–58). At a fixed 10% FPR, corresponding to a 3.7% risk threshold, model sensitivity was 48% (95% CI 34–62).
For prediction of preterm pre-eclampsia, the WS final model had an AUC of 0.74 (95% CI 0.68–0.79). At the ≥8% risk threshold, 36 (9.6%) women developed pre-eclampsia with a model sensitivity of 31% (24–40%) and specificity 97% (97–98%) (Additional file 2: Table S3). At a fixed 5% FPR, model sensitivity was 44% (95% CI 36–53). At a fixed 10% FPR, model sensitivity was 50% (95% CI 41–59).
The NICE approach
Using the NICE approach in the subset of women without NICE-listed high-risk factors (n = 12,125, pre-eclampsia events = 260), 903 (7.4%) of women were classified screen-positive with a sensitivity of 28% (23–34%) and specificity 93% (93–94%) (Table 3). Among women classified screen-positive, 74 (8.2%) developed pre-eclampsia. For comparison, at the same specificity, the WS base model predicted 7.5% of women in this sample at ≥3.9% risk and achieved similar sensitivity (29%; 24–35%) to the NICE approach. The NNT for women classified by the model at the ≥3.9% risk threshold was 120 and for NICE was 122. The NNS for the model was 1595, compared to 1639 for women using the NICE approach.
Using the NICE approach in the entire study sample including women with high-risk factors (n = 12,395, pre-eclampsia events = 293), 1173 (9.5%) of women were classified screen-positive with a sensitivity of 37% (31–42%), specificity 91% (91–92%) (Table 3). Among women classified screen-positive, 107 (9.1%) developed pre-eclampsia. For comparison, at the same specificity, the WS final model predicted 9.7% of women in this sample at ≥3.8% risk, and achieved a similar sensitivity (38%; 33–44%) to the NICE approach. The NNT for women classified by the model at the ≥3.8 risk threshold was 108 and for NICE was 110. The NNS for the model was 1107, compared to 1158 using the NICE approach.
The accuracy of the NICE approach was higher to predict early-onset and preterm pre-eclampsia (Additional file 2: Table S4).
The WS risk model combines eight routinely collected maternal characteristics to estimate the probability of pre-eclampsia in early pregnancy for women attending antenatal care for their first pregnancy. The model demonstrated good calibration for risk prediction overall, but model accuracy was modest when using the pre-specified 8% risk threshold to predict which women will develop pre-eclampsia. While very few women who did not develop pre-eclampsia had a model-predicted risk above 8% (specificity 97%, 97–98%), only one in five women who developed pre-eclampsia had a risk estimate above this level (sensitivity 18, 95% CI 14–23%). Model sensitivity was higher when assigning a lower risk threshold to classify women as high-risk. For example, one in three women who developed pre-eclampsia had a predicted risk ≥4% (sensitivity 36% (31–41%), specificity 92% (91–92%)). The observed risk of pre-eclampsia was 10% in this group of women.
In contrast to the WS model algorithm, the NICE approach classifies all nulliparous women as moderate-risk, and those with any of the eight additional moderate or high-risk factor/s as high-risk. Based on our study sample, nearly 10% of nulliparous women screened ‘positive’ using the NICE approach, with a sensitivity and specificity for prediction of pre-eclampsia similar to the WS model when at a 3.8% risk threshold (NICE sensitivity 37% (95% CI 31–42%), specificity 91% (91–92%)).
The sensitivity of the WS model was higher for prediction of early-onset and preterm pre-eclampsia than any-onset pre-eclampsia with similar specificity. While these findings might suggest some maternal risk factors are more strongly associated with early-onset and pre-term pre-eclampsia than late-onset pre-eclampsia, an alternative explanation may be that women with risk factors for pre-eclampsia may be more likely to have an induced birth before 40 weeks, thus removing the risk of pre-eclampsia at term. Our finding that the median gestational age at birth for women classified as high-risk (≥1 high-risk factors listed by NICE guideline) was 38.6 weeks’ gestation compared to 39.5 weeks’ gestation for those not classified as high-risk provides some support for this explanation.
The model includes all NICE-listed risk factors except diabetes (type 1 or 2) which did not contribute to the model with an adjusted odds ratio of 0.71 (95% CI 0.22–2.31). One possible interpretation of these findings is that women with well managed diabetes are not at higher risk of pre-eclampsia. Evidence from a clinical trial of women with type 1 diabetes reported each 1% decrement in HbA1C value before pregnancy and at the first antenatal visit was associated with a lower risk of pre-eclampsia supports this as a possible explanation . However, few women in our study sample (< 1%) had diabetes recorded and the wide 95% confidence interval does not exclude the possibility of diabetes as a predictor of pre-eclampsia.
Comparison with existing evidence
Despite the potential clinical benefits of targeting nulliparous women for pre-eclampsia prevention, most validation studies of risk assessment tools do not report performance separately for nulliparous women [7,8,9,10,11,12]. Two pre-eclampsia risk prediction models based on routinely collected maternal factors identified from our previous systematic review  have been assessed in nulliparous women [6, 31]. Both included more risk factors than the WS model and NICE approach. First, North et al. (2011) recruited 3529 singleton nulliparous women from five centres in Australia, New Zealand, Ireland, and the United Kingdom (UK) and developed a model that included 11 clinical predictors and mean arterial pressure . Model calibration was only reported in the development sample. Model accuracy was not assessed at a clinically-defined risk threshold. At the study-defined cut-off of 25% FPR, model sensitivity (61% (54–68%)  was higher than our WS model sensitivity (50% (44–56%) at the same cut-off. This study adds to evidence that combining blood pressure measurements to maternal factors improves prediction of pre-eclampsia in early pregnancy . Unfortunately, blood pressure measurements were not available in the ObstetriX database to allow us to assess the impact on WS model performance.
The second study reported by Wright et al. (2015) assessed the accuracy of a model based on nine maternal factors (maternal age, height, interpregnancy interval, method of conception, chronic hypertension, systemic lupus erythematosus or antiphospholipid syndrome, weight, family history of pre-eclampsia and diabetes mellitus) and ethnicity (white, Afro-Caribbean, South Asian, East Asian and mixed)) . The study included 59,947 singleton nulliparous women recruited from two hospitals in UK. The Wright model was internally validated by 5-fold cross validation. The authors assessed model accuracy at the risk cut-off corresponding to the FPR estimated for the NICE approach and reported a model sensitivity of 31% (28.8–33.3%) at 11.5% FPR which compared favourably to the sensitivity of the NICE approach in the same sample (24.8%, 22.7–26.9%) . The WS final model achieved higher sensitivity to predict pre-eclampsia at the same cut-off (sensitivity 40% (35–46) at 10% FPR).
The Wright model was recently validated in a sample of 4184 nulliparous women recruited from single hospital in the UK  to predict preterm pre-eclampsia (preterm pre-eclampsia rate = 0.7%, and compared to the NICE approach. The authors reported higher sensitivity than the original study to predict preterm pre-eclampsia (57.1%; 95% CI 37.5–74.8% vs 35.9; 31.5–40%), with slightly lower FPR (8.8% vs 11.5%), and similar performance to the NICE approach . In comparison, the WS final model showed slightly lower sensitivity to predict preterm pre-eclampsia with narrower confidence interval than the validation study of the Wright model (50%; 41–59% vs 57.1%; 37.5–74.8%) at 8.8% FPR. The accuracy of the NICE approach to predict preterm pre-eclampsia was similar to our present study in Australian women.
One other Australian study has assessed the NICE approach, although the authors did not report results separately for nulliparous women and the sample size of 543 women did not allow precise estimates of performance . We have previously assessed the NICE approach in nulliparous women enrolled in international trials of aspirin prophylaxis who had a relatively high-risk of pre-eclampsia (4.8%) . In this trial sample, the NICE approach had a lower sensitivity (8.9, 10.1%, respectively) and higher specificity (97.2, 96.6%, respectively) than the present study . However, trial data were not available for BMI and not adequately reported for family history which may have resulted in an underestimation of the NICE approach performance . Our present study findings highlight the importance of BMI as a risk factor with obesity (BMI ≥30 kg/m2) recorded for 12% women; and we observed a high rate of pre-eclampsia (7%) for this group.
Two international studies have reported on the performance of the NICE approach in nulliparous women, although both excluded women with multiple pregnancies [6, 8]. Both studies reported slightly lower estimates of sensitivity and specificity than the present study. First, a UK study reported the NICE approach classified 12% of nulliparous women as high-risk (similar to 10% in the present study) with a sensitivity of 25% (95% CI 23–27%) and specificity (88%) for prediction of any-onset pre-eclampsia (overall pre-eclampsia rate 2.8%) . Second, a study with smaller sample that included women from the UK, Ireland, New Zealand and Australia reported a sensitivity of 31% and a specificity 88% (calculated from data tabulated) for prediction of preterm pre-eclampsia (overall pre-eclampsia rate 1.3%) . Together with our findings, these results provide consistent evidence that the NICE approach correctly classifies approximately one-quarter to one-third of women who will develop pre-eclampsia as high-risk.
The main limitation of our study is that we collected study data retrospectively which might lead to under-ascertainment of study variables . Under-ascertainment of risk factors may result in underestimation of their predictive performance. Another limitation is that we did not include births from private hospitals to assess whether hospital setting, or maternal factors related to delivery in a private hospital, impacts the risk of developing pre-eclampsia which would reduce the applicability of the WS model to women giving birth outside public hospitals. Finally, no women in the study sample had more than three high-risk factors recorded, thus model predictions cannot be extended to such women.
The main intended benefit of pre-eclampsia risk assessment is to correctly identify high-risk women who will benefit from prophylaxis and/or further testing and management by a specialist to prevent the condition or its complications. Although we demonstrate the predictive performance of the WS model and NICE approach is modest compared to that reported for strategies using tests such as uterine artery Doppler and serum biomarkers , our findings support the clinical value of these approaches in settings where such specialised tests are not routinely available. For example, the NICE approach can be recommended for nulliparous women, if one considers managing eleven women as high-risk for every one woman who will develop pre-eclampsia is acceptable (PPV 9% (8–11%)). Acceptability will depend on the level of concern about the potential adverse effects of prophylactic agents, such as aspirin . In settings where specialised tests are available, the NICE approach may have a role to select high-risk women for further testing. Under this strategy, our estimate of NPV at 98% indicates that for every 50 women classified as low-risk using the NICE approach, one woman will develop pre-eclampsia and miss the opportunity for further testing.
Compared to the NICE approach, the WS model has the advantage of providing an estimate of the probability of pre-eclampsia to assist patient counselling and guide clinical practice decisions about prophylaxis and/or referral for further testing. In particular, the model allows risk prediction based on maternal age and BMI, which are more common than high-risk factors such as chronic hypertension, whereas the NICE approach is restricted to using a single different cut-off for these two factors. We provide the model algorithm in an Excel spreadsheet that can be readily incorporated into the first antenatal clinic visit. At a population level, setting the risk threshold of 4% to guide decisions such as aspirin prophylaxis appeared appropriate with a 10% observed risk of pre-eclampsia in this group of women.
The WS model should be validated outside of the WSLHD before implementing for nulliparous women in other settings. For example, our findings that immigrant women are at lower risk of pre-eclampsia may not apply outside of Australia.
Further research is needed to investigate whether the inclusion of individual ethnic groups may improve model prediction in Australia’s highly culturally diverse antenatal population, and to investigate the value of including maternal blood pressure measurement in the model.
In settings such as Australia, where tests such as uterine artery Doppler and serum biomarkers are available for pre-eclampsia risk assessment but not yet widely used, broader implementation will depend on cost-effectiveness relative to risk assessment without these tests. Our findings of the accuracy of simple risk tools will be valuable to inform these cost-effectiveness analyses. The WS model can also be used as a foundation on which to build a specialised risk prediction model.
A risk prediction model that incorporates guideline-listed risk factors and ethnicity provides modest performance for predicting pre-eclampsia in Australian nulliparous women. The model did not outperform the NICE approach, but has the advantage of providing individualised risk estimates over the NICE guideline to assist patient counselling and inform decisions for further testing and prophylaxis.
Availability of data and materials
The data and analyses that support the findings of this study are available on request from the co-author SL; email: email@example.com. The data are not publicly available due to WSLHD regulations.
Adjusted odds ratio
Area under the curve
Body mass index
False positive rate
National Institute for Health and Clinical Excellence
Number needed to screen
Number needed to treat
Negative predictive value
Perinatal Antiplatelet Review of International Studies
Positive predictive value
Receiver operating characteristics
Socio-economic Advantage and Disadvantage from the Socio-Economic Indexes for Areas
Society of Obstetric Medicine of Australia and New Zealand
Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis
United States Preventive Services Taskforce
Western Sydney Local Health District
SOMANZ (Society of Obstetric Medicine of Australia and New Zealand). The SOMANZ guidelines for the management of hypertensive disorders of pregnancy. Sydney: SOMANZ; 2014.
Department of Health. Clinical practice guidelines: pregnancy care. Canberra: Australian Government Department of Health; 2018.
National Institute for Health and Care Excellence. NICE guideline. Hypertension in pregnancy: diagnosis and management. NICE, 2019. Available from: https://www.nice.org.uk/guidance/ng133/resources/hypertension-in-pregnancy-diagnosis-and-management-pdf-66141717671365
LeFevre ML. Low–dose aspirin use for the prevention of morbidity and mortality from preeclampsia: U.S. preventive services task force recommendation statement. Ann Intern Med. 2014;161:819–26.
Al-Rubaie ZT, Askie LM, Hudson HM, Ray JG, Jenkins G, Lord SJ. Assessment of NICE and USPSTF guidelines for identifying women at high-risk of pre-eclampsia for tailoring aspirin prophylaxis in pregnancy: an individual participant data meta-analysis. Eur J Obstet Gynecol Reprod Biol. 2018;229:159–66.
Wright D, Syngelaki A, Akolekar R, Poon LC, Nicolaides KH. Competing risks model in screening for preeclampsia by maternal characteristics and medical history. Am J Obstet Gynecol. 2015;213:62.e1–10.
O'Gorman N, Wright D, Poon LC, Rolnik DL, Syngelaki A, de ALVARADO M, et al. Multicenter screening for pre-eclampsia by maternal factors and biomarkers at 11-13 weeks' gestation: comparison with NICE guidelines and ACOG recommendations. Ultrasound Obstet Gynecol. 2017;49(6):756–60.
Myers JE, Kenny LC, McCowan LM, Chan EH, Dekker GA, Poston L, et al. Angiogenic factors combined with clinical risk factors to predict preterm pre-eclampsia in nulliparous women: a predictive test accuracy study. BJOG Int J Obstet Gynaecol. 2013;120(10):1215–23.
Tan MY, Wright D, Syngelaki A, Akolekar R, Cicero S, Janga D, Singh M, Greco E, Wright A, Maclagan K, Poon LC. Comparison of diagnostic accuracy of early screening for pre-eclampsia by NICE guidelines and a method combining maternal factors and biomarkers: results of SPREE. Ultrasound Obstet Gynecol. 2018;51(6):743–50.
Poon LC, Rolnik DL, Tan MY, Delgado JL, Tsokaki T, Akolekar R, Singh M, Andrade W, Efeturk T, Jani JC, Plasencia W. ASPRE trial: incidence of preterm pre-eclampsia in patients fulfilling ACOG and NICE criteria according to risk by FMF algorithm. Ultrasound Obstet Gynecol. 2018;51(6):738–42.
Rocha RS, Alves JA, Júnior EA, Martins WP, Vasconcelos CT, Costa FD, Oriá MO. Comparison of three algorithms for prediction preeclampsia in the first trimester of pregnancy. Pregnancy Hypertens. 2017;10:113–7.
Al-Amin A, Rolnik DL, Black C, White A, Stolarek C, Brennecke S, da Silva Costa F. Accuracy of second trimester prediction of preterm preeclampsia by three different screening algorithms. Aust N Z J Obstet Gynaecol. 2018;58(2):192–6.
Al-Rubaie ZT, Askie LM, Ray JG, Hudson HM, Lord SJ. The performance of risk prediction models for pre-eclampsia using routinely collected maternal characteristics and comparison with models that include specialised tests and with clinical guideline decision rules: a systematic review. BJOG Int J Obstet Gynaecol. 2016;123(9):1441–52.
Khan KS, Wojdyla D, Say L, Gulmezoglu AM, Van Look PF. WHO analysis of causes of maternal death: a systematic review. Lancet. 2006;367(9516):1066–74.
Report of the National High Blood Pressure Education Program Working Group on High Blood Pressure in Pregnancy. Am J Obstet Gynecol. 2000;183(1):S1–s22.
Bilano VL, Ota E, Ganchimeg T, Mori R, Souza JP. Risk factors of pre-eclampsia/eclampsia and its adverse outcomes in low- and middle-income countries: a WHO secondary analysis. PLoS One. 2014;9(3):e91198.
Roberts CL, Ford JB, Algert CS, Antonsen S, Chalmers J, Cnattingius S, et al. Population-based trends in pregnancy hypertension and pre-eclampsia: an international comparative study. BMJ Open. 2011;1(1):e000101.
Thornton C, Dahlen H, Korda A, Hennessy A. The incidence of preeclampsia and eclampsia and associated maternal mortality in Australia from population-linked datasets: 2000–2008. Am J Obstet Gynecol. 2013;208(6):476.e1–5.
Al-Rubaie ZT, Hudson HM, Jenkins G, Mahmoud I, Ray JG, Askie LM, Lord SJ. The association between ethnicity and pre-eclampsia in Australia: A multicentre retrospective cohort study. Aust N Z J Obstet Gynaecol. 2019; Doi.org/https://doi.org/10.1111/ajo.13069.
Centre for Epidemiology and Evidence. New South Wales Mothers and Babies 2017. Sydney: NSW Ministry of Health; 2018.
Askie L, Duley L, Henderson-Smart D, Stewart L. On behalf of the PARIS collaborative group. Antiplatelet agents for prevention of pre-eclampsia: a meta-analysis of individual patient data. Lancet. 2007;369(9575):1791–8.
Brown MA, Lindheimer MD, de Swiet M, et al. The classification and diagnosis of the hypertensive disorders of pregnancy: statement from the International Society for the Study of Hypertension in Pregnancy (ISSHP). Hypertens Pregnancy. 2001;20:IXeXIV.
Census of Population and Housing: Socio-Economic Indexes for Areas (SEIFA), Australia. https://www.abs.gov.au/ausstats/abs@.nsf/Lookup/2033.0.55.001main+features100042011. Accessed 28 March 2013.
Steyerberg EW. Clinical Prediction Models: a practical approach to development, validation, and updating. New York: Springer; 2009.
Hosmer D, Lemeshow S, Sturdivant RX. Applied logistic regression. New York, NY: A Wiley-Interscience Publication.
Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1–73.
Bartsch E, Park AL, Kingdom JC, Ray JG. Risk threshold for starting low dose aspirin in pregnancy to prevent preeclampsia: an opportunity at a low cost. PLoS One. 2015;10(3):e0116296.
Rembold CM. Number needed to screen: development of a statistic for disease screening. Bmj. 1998;317(7154):307–12.
Bonnett LJ, Snell KI, Collins GS, Riley RD. Guide to presenting clinical prediction models for use in clinical settings. BMJ. 2019;365:l737.
Holmes VA, Young IS, Patterson CC, Pearson DW, Walker JD, Maresh MJ, McCance DR. Diabetes and pre-eclampsia intervention trial study group. Optimal glycemic control, pre-eclampsia, and gestational hypertension in women with type 1 diabetes in the diabetes and pre-eclampsia intervention trial. Diabetes Care. 2011;34(8):1683–8.
North RA, McCowan LM, Dekker GA, Poston L, Chan EH, Stewart AW, Black MA, Taylor RS, Walker JJ, Baker PN, Kenny LC. Clinical risk prediction for pre-eclampsia in nulliparous women: development of model in international prospective cohort. BMJ. 2011;342:d1875.
Poon LC, Kametas NA, Valencia C, Chelemen T, Nicolaides KH. Hypertensive disorders in pregnancy: screening by systolic diastolic and mean arterial pressure at 11–13 weeks. Hypertens Pregnancy. 2011;30(1):93–107.
Sovio U, Smith GC. Evaluation of a simple risk score to predict preterm pre-eclampsia using maternal characteristics: a prospective cohort study. BJOG: An International Journal of Obstetrics & Gynaecology. 2019.
Chen JS, Roberts CL, Simpson JM, Ford JB. Prevalence of pre-eclampsia, pregnancy hypertension and gestational diabetes in population-based data: impact of different ascertainment methods on outcomes. Aust N Z J Obstet Gynaecol. 2012;52(1):91–5.
Roberge S, Bujold E, Nicolaides KH. Meta-analysis on the effect of aspirin use for prevention of preeclampsia on placental abruption and antepartum hemorrhage. Am J Obstet Gynecol. 2018;3218:483.
We thank the data custodians of the WSLHD ObstetriX database for their valuable help in collecting and providing the data.
Ziad TA Al-Rubaie is supported by an Australian Research Training Program and Collaborative Research Network PhD Scholarship through The University of Notre Dame Australia. The funding source had no role in the design of the study, the collection, analysis, and interpretation of data and in writing the manuscript.
Ethics approval and consent to participate
Ethical approval was obtained from the Human Research Ethics Committees (HREC) of The University of Notre Dame Australia (HREC approval number 016071S) and the Western Sydney Local Health District (HREC approval number: LNR/16/WMEAD/239, LNR SSA/16/WMEAD/256). Study data were provided de-identified. A waiver for patient consent was approved for this study, so written consent was not required.
The study was undertaken as a part of a PhD project which was approved by The University of Notre Dame Australia Research Committee. No other administrative permissions and/or licenses were acquired by the investigator team for this research.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Additional file 1: Figure S1. Fitted trend between a continuous predictor and the logit pre-eclampsia using cubic spline function, Legend: (A) maternal age in years; (B) body mass index in kg/m2; all women (N = 12,395). Figure S2. Distribution of predicted probabilities for pre-eclampsia using the WS base model in the validation sample excluding women with high-risk factors (N = 6064). Figure S3. Distribution of predicted probabilities for pre-eclampsia using the WS full model in the validation sample (N = 6201). Figure S4 Receiver-operator characteristic curve for the WS final model for prediction of pre-eclampsia in the entire study sample (N = 12,395), area-under-curve 0.70 (95% CI 0.66–0.73).
Additional file 2: Table S1. Western Sydney (WS) model for prediction of pre-eclampsia in women without known high-risk factors, and all women, model development sample. Table S2. Internal validation of Western Sydney (WS) base and full model by bootstrapping, 1000 resamples. Table S3. Performance of the Western Sydney (WS) final model for prediction of early-onset pre-eclampsia and preterm pre-eclampsia at ≥8% risk threshold, N = 12,395. Table S4. Performance of NICE approach* to predict preterm, term, early- and late-onset pre-eclampsia for nulliparous women in WSLHD, 2011–2014 (n = 12,395).
About this article
Cite this article
Al-Rubaie, Z.T.A., Hudson, H.M., Jenkins, G. et al. Prediction of pre-eclampsia in nulliparous women using routinely collected maternal characteristics: a model development and validation study. BMC Pregnancy Childbirth 20, 23 (2020). https://doi.org/10.1186/s12884-019-2712-x
- Antenatal care
- Maternal health
- National Institute of health and care excellence
- Risk assessment
- Risk prediction model