Development and validation of an obstetric early warning system model for use in low resource settings

Abstract

Background

The use of obstetric early-warning-systems (EWS) has been recommended to improve timely recognition, management and early referral of women who have or are developing a critical illness. Development of such prediction models should involve a statistical combination of predictor clinical observations into a multivariable model which should be validated. No obstetric EWS has been developed and validated for low resource settings. We report on the development and validation of a simple prediction model for obstetric morbidity and mortality in resource-limited settings.

Methods

We performed a multivariate logistic regression analysis using a retrospective case-control analysis of secondary data with clinical indices predictive of severe maternal outcome (SMO). Cases for design and validation were randomly selected (n = 500) from 4360 women diagnosed with SMO in 42 Nigerian tertiary-hospitals between June 2012 and mid-August 2013. Controls were 1000 obstetric admissions without SMO diagnosis. We used clinical observations collected within 24 h of SMO occurrence for cases, and normal births for controls. We created a combined dataset with two controls per case, split randomly into development (n = 600) and validation (n = 900) datasets. We assessed the model’s validity using sensitivity and specificity measures and its overall performance in predicting SMO using receiver operator characteristic (ROC) curves. We then fitted the final developmental model on the validation dataset and assessed its performance. Using the reference range proposed in the United Kingdom Confidential-Enquiry-into-Maternal-and-Child-Health 2007-report, we converted the model into a simple score-based obstetric EWS algorithm.

Results

The final developmental model comprised abnormal systolic blood pressure-(SBP > 140 mmHg or < 90 mmHg), high diastolic blood pressure-(DBP > 90 mmHg), respiratory rate-(RR > 40/min), temperature-(> 38 °C), pulse rate-(PR > 120/min), caesarean-birth, and the number of previous caesarean-births. The model was 86% (95% CI 81–90) sensitive and 92%- (95% CI 89–94) specific in predicting SMO with area under ROC of 92% (95% CI 90–95%). All parameters were significant in the validation model except DBP. The model maintained good discriminatory power in the validation (n = 900) dataset (AUC 92, 95% CI 88–94%) and had good screening characteristics. Low urine output (300mls/24 h) and conscious level (prolonged unconsciousness-GCS < 8/15) were strong predictors of SMO in the univariate analysis.

Conclusion

We developed and validated statistical models that performed well in predicting SMO using data from a low resource settings. Based on these, we proposed a simple score based obstetric EWS algorithm with RR, temperature, systolic BP, pulse rate, consciousness level, urinary output and mode of birth that has a potential for clinical use in low-resource settings..

Background

The World Health Organization (WHO) estimated that 303,000 maternal deaths occurred globally at the end of the Millennium Development Goals (MDGs) in 2015. Over 99% of these deaths occurred in low and middle-income countries (LMICs), most of which made insufficient progress towards achieving the MDG maternal health targets [1]. It is also estimated that there are 27 million episodes of direct obstetric complications annually which contribute to long-term pregnancy and childbirth complications [2]. Following increasing access to facility-based births, partly because of Universal Health Coverage policies under the Sustainable Development Goals since 2016, opportunities to ensure good quality facility care are critical, if the new ambitious global and national Maternal Mortality Ratio (MMR) targets are to be achieved [3].

The increased burden of adverse outcomes, especially in LMICs is believed to be due primarily to delays in the recognition of pregnancy complications [4, 5]. Early warning systems (EWSs) are clinical diagnostic prediction models that involve serial clinical observations (“track”) with criteria (“trigger”) to identify patients at risk of complications [6]. A 2018 systematic review of EWS used in obstetrics found that they are effective in predicting adverse obstetric outcomes and reducing obstetric morbidity [7]. The United Kingdom Confidential Enquiry into Maternal and Child Health (CEMACH) (2003–2005 report) recommended “the use of obstetric EWS to improve timely recognition, treatment and referral of women who have or are developing a critical illness” [8]. Most of the available obstetric EWS versions used subsequently were designed based on clinical consensus rather than the application of recommended prediction model development methodology that should include statistical analysis of outcome measures [9,10,11,12,13,14,15,16,17].

Model development involves a statistical combination of predictor clinical observations into a multivariable model. The Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement recommends that new prediction models are tested on data used in its development (internal validation) and data from a different population (external validation) [10].

In 2013, the first statistically derived, obstetric EWS was developed in the UK. It was internally validated using clinical observations (physiological variables) collected from 4400 women during their first 24 h of critical care admission [18]. The EWS developed showed a good predictive ability to discriminate survivors from non-survivors in the derivation dataset, as well as on an external dataset [19]. However, since the database used in the development, internal [18] and external validation [19] of the EWS were only for women admitted to critical care, the EWS may not be suitable for obstetric populations without obvious need for critical care and in a non-UK setting.

No obstetric EWS has been developed and validated for a low resource setting. However, 14 of the 16 obstetric EWS identified in a recent systematic review, had five clinical observations of physiologic variables (pulse rate, systolic and diastolic blood pressure, respiratory rate, temperature and consciousness level) that can be easily collected even in low resource settings [7].

We report on the development and validation of a simple obstetric prediction model and EWS algorithm for use in resource-limited settings.

Methods

Study design

The study was a retrospective case-control analysis of secondary data on admissions to inpatient obstetric wards in Nigerian tertiary hospitals. Cases based on standardised definitions were derived from the Nigerian near-miss study, the largest prospective investigation of maternal deaths and near-misses in Africa [20].. Controls were women who were admitted for obstetric care at the same time as the cases.

Study population and data sources

The study protocol and findings of the Nigerian Near-miss study have been published elsewhere [20, 21]. All women admitted to 42 Nigerian tertiary hospitals for birth or within 42 days of birth or termination of pregnancy between June 2012 and August 2013 were eligible for enrolment into the near-miss study. The study reported severe maternal outcome (SMO) cases based on the WHO near-miss criteria (organ dysfunction, clinical and management-based) [20, 22].

For this analysis, we obtained data on 4360 women who had SMO during the 14-month surveillance, including 998 who died and 3362 who had a near-miss [22]. Overall, 2401 SMO cases occurred in 11 tertiary care hospitals across the 3 northern sub-regions of Nigeria. Cases (n = 500) were randomly selected from these, and split into the design (n = 200) and validation (n = 300) SMO cases dataset (Fig. 1). Three of the 11 tertiary care hospitals were coveniently selected across the three sub-regions (one tertiary hospital in each northern sub-region) for control data collection. One thousand women who were admitted for birth, between June 2012 and August 2013, when the near-miss study was conducted and discharged with no SMO were recruited as the controls.

Fig. 1

Creation of the development and validation data sets

The sample size estimate was based on a baseline SMO prevalence of 4·8% from the near-miss case dataset. At 5% level of significance, the analysis could detect an absolute difference in SMO prevalence of 6 and 5% at 90 and 80% powers, respectively.

All data were collected within 24 h of the occurrence of SMO (for cases) (Oladapo et al., 2013) or 24 h of birth or end of pregnancy (collected retrospectively for controls). Figure 1 illustrates allocation into, and composition of, the development and validation data sets.

Data abstraction

Individual-level data on all study variables were abstracted from the Near-miss study dataset (n = 4360) and from the case notes of 1000 controls. These included demographic characteristics (age, weight, height), obstetric variables (parity, the number of antenatal clinic visits, gestational age at the time of admission or birth, mode of birth, interval from last pregnancy, the number of previous caesarean section), diagnosis, length of stay in hospital, and the last haematocrit measured before occurrence of outcome. Abnormal clinical indices were extracted from the cases dataset based on their definitions codebook including high (> 140 mmHg) and low (< 90 mmHg) systolic blood pressure, high (> 90 mmHg) and low (< 60 mmHg) diastolic blood pressure, high (> 38 °C) and low (< 35 °C) temperature, marked tachycardia (PR > 120/min) and bradycardia (PR < 60/min), hypoxaemia (SpO₂ < 90%), severe tachypnoea (RR > 40/min) and bradypnea (RR < 6/min), severe oliguria (urinary output< 300 mL in 24 h), and coma (Glasgow Coma Score < 8/15). Among controls, data were collected as continuous variables and classified based on the cut-off values in the cases dataset codebook.

Outcome

The outcome measure for this analysis was SMO, computed as the sum of maternal deaths and near-misses and transformed into a binary variable: occurred or not. Maternal death was defined according to the International Classification of Diseases (ICD-10) [23]. Women were identified as maternal near-miss if they met any of the three near-miss criteria (clinical criteria related to specific disease entities, intervention-based criteria and organ dysfunction-based criteria) [22].

Statistical methods

Characteristics of the study population were summarised by means and standard deviations for continuous and percentages for categorical variables. Continuous variables were compared between cases and controls with independent sample t-tests or Mann Whitney U tests, depending on whether or not variables were normally distributed. Normality was assessed visually through distribution plots and with the Shapiro-Wilk test. Data for 1500 (cases = 500 and controls, n = 1000) study participants were randomly allocated into the development dataset (n = 600) or validation data set (n = 900) with two controls per case.

Model building

Univariable logistic regression models were fitted to assess the association between each predictor variable and outcome (SMO). A stricter inclusion criterion was applied and therefore variables were only selected from the univariable models for inclusion into the multivariable model if the model had a p-value< 0·05. This yielded 22 potential variables for inclusion.

Multivariable logistic regression models were fitted using a backward stepwise approach and factors were removed from the model, one at a time based on the highest p-value> 0·05 and their likelihood ratio. When the final model was achieved, a sensitivity test was performed by including each of the eliminated variables, in turn, into the final model to assess their significance. None of these was found significant in the final model. Variables were tested for collinearity, by a simple check of the correlation coefficients, and were dropped to improve parsimony of the model.

Model performance

Performance of the obstetric EWS clinical prediction model from the development data set was tested on the validation data set [24]. First, overall validity (sensitivity, specificity, negative predictive value-NPV and positive predictive value-PPV), as well as area under the curve (AUC) for Receiver Operating Characteristics (ROC) curves were assessed for the final model. Discrimination was assessed using the c statistic or AUC, an estimate of the probability of assigning higher risk to those who suffered SMO compared to those who did not. The final model was then applied to the validation dataset, and performance was similarly assessed. Finally, we estimated validity of the selected EWS predictors by applying the final (developmental) model to the entire (n = 5243 after excluding 117 missing data) dataset of cases and controls, bearing in mind that the ratio of cases to controls had been reversed – over 4 cases per control. All statistical analyses were performed using Statistical Package for the Social Sciences (SPSS) software version 25 and Stata version 15·1. We constructed 95% confidence intervals (CI) for all performance characteristics.

Obstetric EWS algorithm

Our statistically derived diagnostic prediction model was modified based on the clinical importance of rejected variables. Given that the cases data set provided categorised (binary) clinical variables, it was not feasible to validate reference ranges for the different model parameters. Hence, the final (validation) model was converted into simple score-based obstetric EWS algorithm using the reference range proposed in the MEOWS chart recommended in the 2007 CEMACH report [8] (Additional file 1).

Patient and public involvement statement

Patients were not involved in the development of the research question or design of this study. Secondary patient monitoring data was used for this analysis. Implications of this study will be disseminated through patient groups and blogs in the study setting.

We present our findings according to the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis statement [10].

Results

From the 97, 634 women admitted for pregnancy, childbirth or puerperal complications during the 14 months of the Nigerian near-miss study, 4360 developed SMO and their data were included in the cases data set. Of these, 998 women died and 3362 suffered maternal near-miss. A total of 117 (2·7%) women with SMO had missing data, with no records of the type of SMO they experienced. These women were excluded from the analyses as the missingness was assumed to be at random. Lack of data on other characteristics of these participants did not allow for assessment of bias in dropping them from the analysis. None of the control participants had missing parameters.

Characteristics of SMO cases and controls are given in Table 1. Those who experienced SMO tended to be older and more likely to be obese. They also stayed longer in hospital, with the mean number of days of admission four times greater than controls and were more likely to be anaemic. There was no significant difference in terms of parity, but SMO cases had more preterm births and had more antenatal visits than the controls (Table 1).

Table 1 Characteristics of women with severe maternal outcome (n = 4243) compared with controls who were discharged without SMO diagnosis (n = 1000)

Overall, women who had abnormal clinical observation measurements (either lower or higher than normal), based on the pre-defined cut-off points deciphered from the cases dataset, were more likely to develop SMO than controls (Table 2). Initially, bivariate analysis was performed considering all 22 potential variables for entry in the model. Of these, a total of 15 had significant p-values < 0·05 (Table 2) and so these were considered for potential inclusion in the multiple regression model.

Table 2 Statistically significant clinical variables from univariate analysis (dependent variable; SMO binary outcome variable) in the model design dataset (n = 600)

The significant variables were entered into a multiple logistic regression model. Both backward and forward stepwise selection methods were used to build the final parsimonious model, with the standard 5% significance level for entry and removal. Both techniques produced the same final (developmental) model, in five steps, with 8 parameters (high systolic blood pressure (> 140 mmHg), low systolic blood pressure (< 90 mmHg), high diastolic blood pressure (> 90 mmHg), severe tachypn0ea (RR > 40/min), fever (temperature > 38^o C), marked tachycardia (PR > 120/min), mode of birth (caesarean or vaginal birth), and the number of previous caesarean births).

Risk of severe maternal outcome

Table 3 gives odds ratios of developing SMO for the variables in the SMO models. From the developmental model, risk of developing SMO was five times greater among women with high systolic blood pressure (> 140 mmHg) and tachycardia (PR > 120/min), while low systolic BP (< 90 mmHg) increased SMO risk four times. Caesarean birth during index admission was found to increase risk of SMO significantly (odds ratio 6; 95% CI 2.8–12.5), but the number of previous caesarean sections did not increase risk. Most importantly, the variables with the highest risk of SMO in the developmental model were fever (OR 116.5; 95% CI 13.0–147.4) and tachypnoea, (OR 25.2; 95% CI 4.2–51.6).

Table 3 Odds ratios for SMO of the significant predictor variables

Although significant in the developmental model, high diastolic pressure (> 90 mmHg) was not as much of a risk factor as systolic BP, and thus ceased to have a significant effect on SMO risk in the validation model and the performance checking model with the whole data (Table 3). When interactions were considered, diastolic blood pressure was found to be strongly collinear (correlation coefficient, r = 0·95) with systolic blood pressure. When applied in the validation data set (n = 900) and the whole dataset (n = 5243, for sensitivity analysis), all other variables in the developmental model produced a consistent effect in the same direction with variations in effect sizes (Table 3). Similarly, the clinical variables with the highest risk of SMO in the two models were fever and tachypnoea, with odds ratios of 123.2 (95% CI, 62.8–137.0) and 18.2 (95% CI, 3.4–24.3) in the validation model, and 55.9 (95% CI, 22.6–138.0) and 23.2 (95% CI, 9.2–58.6) in the performance checking model with the whole data (Table 3).

Predictive accuracy for SMO

The developmental model explained 66% of the variability in SMO with AUROC of 92% (Table 4, Additional file 3). Given that the cases data set provides already categorised (binary) clinical variables, the diagnostic accuracy of the models was assessed based on the number of parameters required to predict SMO with the best screening properties. Using the presence of five or more triggers as the cut-off point to define SMO, the model predicted SMO with a sensitivity of 86%, a specificity of 92%, positive and negative predictive values of 84 and 93% (Table 4). The validation model produced very similar screening characteristic and discriminatory ability (AUROC 92%) (Table 4, Additional file 4). Not surprisingly, however, the sensitivity model with all cases and controls produced the highest positive predictive value (94%), but a significantly reduced negative predictive value (61%). This is expected given the high prevalence of the outcome measure in the data set (case: control = 4:1). The model explained 58% of the variability in SMO with 92% discriminatory ability (AUROC 82%) (Table 4, Additional file 5).

Table 4 Predictive accuracy of the SMO models (best performing cut-off 0.6)

Proposed EWS algorithm

Measurements of temperature, pulse rate, systolic blood pressure, respiratory rate and mode of birth in postpartum women (caesarean birth versus vaginal birth), constitute the primary early warning parameters from the three statistical models. Diastolic blood pressure was dropped as it was strongly collinear with systolic blood pressure (r = 0·95), and the latter was more clinically relevant and significant in all statistical models (Table 3). Consciousness level and low urinary output (anuria) were dropped in the statistical models due to the perfect prediction of outcome, not statistical significance; this implied that none of the controls (n = 1000) suffered prolong unconsciousness (GCS < 8/15) or experienced low urinary output. Although the two variables were strongly significant predictors of SMO at univariate level (Table 2), they were dropped in the statistical models. Therefore, both variables were forced into the proposed obstetric EWS algorithm, adopting the AVPU (alert, responds to voice or pain and unresponsive) for consciousness level from the MEOWS chart recommended in the 2003–2005 CEMACH report (Additional files 1 and 2). Defining trigger as a single markedly abnormal observation (red trigger) or the combination of two simultaneously mildly abnormal observations (two yellow triggers), the corresponding values from the CEMACH MEOWS chart (Additional file 1) were converted into scores of 0 (normal observation), 1 (yellow trigger) and 2 (red trigger) in the proposed algorithm (Table 5). Mode of birth was scored as 0 and 1 for vaginal and caesarean births.

Table 5 Scoring guideline for the proposed obstetric EWS algorithm

Discussion

This study, to our knowledge, reports for the first time the development and validation of an obstetric diagnostic prediction model for a general obstetric population in a low resource setting using recommended methodology [10]. The final developmental model comprised abnormal systolic blood pressure (SBP > 140 mmHg or < 90 mmHg), high diastolic blood pressure (> 90 mmHg), respiratory rate (RR > 40/min), temperature (> 38oC), pulse rate (PR > 120/min), caesarean birth, and the number of previous caesarean births. The model was 86% (95% CI 81–90) sensitive and 92% (95% CI 89–94) specific in predicting SMO with AUROC of 92% (95% CI 90–95%). We proposed a score-based obstetric EWS algorithm with seven clinical parameters (RR, temperature, systolic BP, pulse rate, consciousness level, urinary output and mode of birth) that has a potential for clinical use in low-resource settings.

A previous study in the United Kingdom using a similar methodology produced an obstetric EWS with similar discriminatory properties (AUROC) 96% (95% CI 92–99%) [18]. Although that study by Carle et al. (2013) had a larger sample size (model development n = 2240) and validation n = 2200), the dataset used was for women admitted to ICU and the EWS included the fraction of inspired oxygen and arterial blood gas, these may limit its use in a low resource setting [19, 25, 26], We did not have the fraction of inspired oxygen in our dataset but found that low SPO2 was not a significant predictor of SMO risk in our final model. It would have been useful to assess that performance on our dataset, but this was impossible because the clinical variables in the SMO cases of our dataset were collected as categorical (binary) variables.

We found that temperature > 38^o C was a predictor of SMO, similar to findings of Ryan et al. (2015) [27] and Singh et al. (2012) [15]. Our finding of high systolic BP as a predictor of SMO was consistent with findings of two inpatient obstetric ward-based validation studies which used obstetric morbidity [15], as defined by consensus of experts, and ICU admission as outcomes [26], and an EWS external validation study that had death as outcome [19]. These findings differed from the development and validation study by Carle and colleagues (2013), that used data within 24 h of admission into ICU, while the other studies used data from inpatients that had no obvious need for critical care [18, 28].

Our report of a significant association between SMO and tachypnoea, high pulse rate, diastolic blood pressure, and low consciousness level was consistent with the findings from the ICU-based external validation study by Paternina-Caicedo et al. [19]

We found a significantly increased risk of SMO among women who had caesarean compared to vaginal birth, and this remained an important predictor of SMO in all our models during development. This informed our inclusion of mode of birth in the proposed EWS tool (Table 5 and Additional file 2).

Our models have excellent predictive ability to discriminate women who developed SMO from those who did not (AUCs consistently above 90%). The model attained similar diagnostic predictive accuracy as one developed, internally [18] and externally [19] validated using data from obstetric ICU patients in the USA. Our model also performed similarly to non-obstetric cardiovascular, adult critical care and neonatal critical care score systems [27, 29, 30]. Our models have significantly better screening characteristics (PPV 94% CI 93–95) as compared to an average of 41% reported for 16 different EWS versions [7]. This is of particular significance, since an EWS that generates many false-positive findings may worsen clinical care, constituting a nuisance alarm by creating an excessive burden on health systems [31, 32]. Potentially, the proposed obstetric EWS algorithm presents an opportunity to institute life-saving interventions to improve clinical outcome. However, current evidence suggests that the use of EWS by itself is not enough to improve health outcomes and that for this tool to perform optimally, an EWS must be integrated with an outreach support system, such as a rapid response team [7, 33].

Of the seven parameters in the proposed EWS algorithm (Table 5 and Additional file 2), five (temperature, pulse rate, systolic blood pressure, respiratory rate and consciousness level) were included in the majority (> 80%) of EWSs published to date [7]. Delays in triage (identification of who is, or may become, severely ill and should be provided with a higher level of care) are believed to contribute immensely to an increased burden of adverse obstetric outcomes in this settings [5]. This is further confounded by the unavailability of patient monitoring devices and other diagnostic equipment, especially in primary healthcare settings [26]. Therefore, in addition to inpatient obstetric wards, we believe the proposed algorithm can present a potentially useful triaging tool to aid timely referral in primary healthcare centres in LMICs. However, prospective external validation is recommended to assess the effectiveness of this tool in primary, secondary and tertiary care in other low resource settings to substantiate these recommendations.

Main strengths of our analysis lie in the robust maternal deaths and near-misses data set which was prospectively collected primarily for research purposes with very few missing data (2·7% of participants), and strict adherence to a diagnostic predictive model development process and reporting as recommended by TRIPOD [10]. A limitation of EWS validation studies that were addressed in our analysis is lack of standardisation of outcome measures; this is especially common with studies using morbidity as outcome measures, as often, this was defined based on consensus rather than standardised definitions [7]. There are several limitations to this study. Firstly, data of SMO consisted of already categorised clinical variables. Although cut-offs were based on recommendations for defining specific disease conditions (such as high blood pressure) by policy-making organisations like WHO, it was not possible to validate different trigger thresholds for the model parameters. Secondly, lack of continuous data also made it impossible to externally validate other EWS versions like CEMACH’s MEOWS and the ICU based chart developed by Carle et al. (2013) [18]. Additionally, pulse oximetry was poorly recorded in our study data set, especially in controls where the parameter was assumed to be above 90%. Although the absence of oxygen saturation could make our proposed chart more feasible to use in low resource settings, evidence from other analyses has shown that it is a valuable predictor of death and serious obstetric complications [34,35,36,37]. This clinical variable would therefore probably still contribute to a statistically developed and validated EWS decision tool in our study population. This can be investigated in an appropriately designed study in the future. Finally, the performance of our model might have been overestimated by the case-control study design with a high proportion of SMO cases. External validation should preferably be performed in a cohort study with a representative incidence of SMO (2.7% as in the Nigerian near-miss study).

Conclusion

To the best of our knowledge, this study provides for the first time an internally validated statistically developed predictive model for SMO among all women admitted to obstetric wards in a low resource setting. This model was used to develop a simple score-based EWS algorithm that has easy to measure parameters with readily available patient monitoring tools, hence constituting a potentially useful triaging tool in low resource healthcare settings. Further work is, however, needed to validate this proposed chart externally in obstetric wards as well as primary healthcare settings.