Evaluation of the implementation fidelity of comprehensive emergency obstetric and newborn care at University of Gondar comprehensive specialized hospital, Northwest Ethiopia: a mixed-method evaluation

Abstract

Background

Approximately 15% of births worldwide result in life-threatening complications during pregnancy, delivery, or postpartum. Comprehensive Emergency Management of Obstetric and Newborn Care (CEmONC) is intended as one of the measures for maternal healthcare services to reduce the high burden with regard to childbirth complications. However, its state of implementation fidelity has not been well investigated. Therefore, this study aimed to evaluate the implementation fidelity of CEmONC services at University of Gondar Comprehensive Specialized Hospital, Ethiopia.

Method

A case-study design with an embedded mixed method was employed. Adherence, quality of delivery, and participant responsiveness dimensions from Carroll’s conceptual framework were used in this evaluation. Four hundred four exit interviews, 423 retrospective document reviews and 10 key informants were conducted. Moreover, a binary logistic regression model was fitted. The qualitative data were transcribed, translated, coded, and analysed using a thematic analysis approach. The overall implementation fidelity of the CEmONC was judged based on the pre-seated judgmental criteria.

Results

Overall the implementation fidelity of the CEmONC service was 75.5%. Quality of delivery, participant responsiveness and adherence were 72.7%, 76.6% and 77.2% respectively.

Signal functions like parenteral antibiotics and removal of retained products were insufficiently performed against the recommended protocols which was also evidenced by the key informant interviews. Healthcare providers’ respect for the clients was less. Age ≥ 35 years (AOR = 0.48, 95% CI: 0.24,0.98), educational status of college and above (AOR = 2.61, 95% CI: 1.46,4.66), being government employed (AOR = 1.85, 95% CI: 1.08,3.18), having ANC follow-up (AOR = 5.50, 95% CI: 1.83, 16.47) and grand multigravida (AOR = 2.17, 95% CI: 1.08, 4.38) were factors significantly associated with participant responsiveness towards the services.

Conclusions

The overall implementation fidelity of the CEmONC services was implemented in good fidelity. Moreover, the quality of delivery was judged as implemented in fair fidelity. Parenteral antibiotics and removal of retained products were not found to be sufficiently performed. Respect for the clients was insufficiently delivered. Therefore, it is recommended that parenteral antibiotics drugs be adequately provided and training for healthcare providers regarding compassionate and respectful care shall be facilitated. Moreover, healthcare providers are strongly recommended to adhere to the recommended guidelines.

Background

Comprehensive Emergency obstetric and newborn care (CEmONC) is defined as a set of life-saving interventions, that treat the major obstetric and newborn causes of morbidity and mortality [1]. Health facilities are classified as CEmONC facilities if they perform blood transfusion and Cesarean Section (CS) services in addition to the seven Basic Emergency Management of Obstetric and Newborn Care (BEmONC) signal functions [2]. These comprehensive services are the only proven approach to saving the lives of women who die while pregnant, giving birth, and after birth [3].

Maternal death is a woman’s death while pregnant or within 42 days after pregnancy termination, regardless of the duration and site of the pregnancy, from any cause related to or aggravated by the pregnancy or its management, but not from accidental or incidental causes [4]. Maternal mortality peaks during the first postpartum day and the intrapartum period around childbirth [5]. Worldwide 300,000 maternal deaths occur every year [6]. About 94% of all maternal deaths are in low and low-middle-income countries (LMICs) and Sub-Saharan African (SSA) countries account for 86% of global deaths [2]. The main reasons for maternal deaths are the lack of skilled birth attendants, remoteness of health facilities in catchment areas, delays in referral for emergency obstetric care, and poor implementation of interventions at the facility level [7].

In Ethiopia, facility-based cross-sectional investigations reveal variability in CEmONC signal function availability and performance [2]. Despite the government’s goal of 100% treatment of obstetric problems in healthcare institutions, the final report on Ethiopia’s EmONC assessment found that only a small portion of that need (18% in all facilities) was reached in 2016 [8].

Pregnancy and childbirth and the consequences of these events continue to be the leading causes of death, disease, and disability among women of reproductive age in developing countries, more than any other single health problem [9]. In low and middle-income countries (LMIC), where maternal and neonatal mortality is high, the availability, accessibility, and use of proven life-saving interventions for the treatment of obstetric complications are low [10, 11]. There is also little understanding of how to deliver obstetric and newborn care interventions effectively [12, 13]. Evaluating the implementation fidelity of the CEmONC ensures that the program is delivered as intended, identifies areas for improvement, holds stakeholders accountable, informs policy and decision-making and ultimately enhances maternal and neonatal health outcomes [13, 14].

Implementation fidelity, an important but underrepresented topic is part of a process evaluation and refers to the degree to which an intervention is delivered according to protocol [15]. Implementing CEmONC programs with high fidelity is crucial for adhering to the program’s design, procedures, and protocols as intended, which directly impacts the program’s effectiveness and the health outcomes of mothers and newborns [16]. However, implementing CEmONC programs with high fidelity can be challenging due to a range of problems. Resource constraints, poor competency level of health care providers, cultural and social barriers, lack of robust monitoring and evaluation systems, and inconsistent policies and guidelines are some of the barriers in implementation fidelity of CEmONC [17]. A major reason for program failure even among sound theory-based interventions is the failure to implement with fidelity [18].

Even though previous studies investigated the utilization of CEmONC services [19, 20], the implementation fidelity of CEmONC has not yet evaluated whether it needs reformulation of objectives, policies, and strategies in its service delivery. Therefore, this study aimed to evaluate the implementation fidelity of Comprehensive Emergency Obstetric and Newborn Care (CEmONC) and factors associated with participant responsiveness towards the CEmONC services at the University of Gondar Comprehensive Specialized Hospital, Ethiopia.

Methods

Evaluation design and settings

This evaluation used a case-study design with an embedded mixed-method from May 22, 2022 to June 22, 2022. The evaluation was conducted at the University of Gondar Comprehensive Specialized Hospital in Gondar City. According to the information obtained from the city administration’s health department office, there is one comprehensive specialized teaching hospital, eight health centers (HCs), and 14 health posts in Gondar city. The University of Gondar Comprehensive Specialized Hospital is one of the referral and teaching hospitals serving more than 7 million people in Godar City and the neighbouring regions. The hospital had 66 beds in obstetric wards, two operational theatres, three maternity wards, and 253 obstetric staff.

Evaluation approach and dimensions

A formative evaluation approach was used to evaluate the implementation fidelity of Comprehensive Emergency Obstetric and Newborn Care (CEmONC) services.

This evaluation used three dimensions: Quality of delivery, participant responsiveness and adherence from Carroll’s conceptual framework for implementation fidelity [21]. Carroll’s conceptual framework for implementation fidelity provides a structured approach to understanding and assessing the degree to which an intervention is implemented as intended [21] When applied to the implementation of CEmONC, this framework can help ensure that the key components of CEmONC are delivered consistently and effectively. The quantitative and qualitative data were collected simultaneously, analyzed separately, and mixed during interpretations of the findings. The evaluation focused on the program process theory of the implementation fidelity of CEmONC services, which included input, activity, and output of the service components as indicated in the program logic model (Fig. 1).

Fig. 1

A logic model of CEmONC services

Variables and measurements

Implementation fidelity is often defined as the degree to which a program follows the original program model intended to be used by the program developers [18]. Evaluation dimensions are measurable aspects of program performance that an evaluation intends to examine. Adherence refers to whether the program service or intervention is being delivered as designed, i.e., with all core components delivered to the appropriate population [22]. The adherence dimension was assessed using nine indicators to determine the extent to which all essential signal functions of CEmONC service were provided to the clients as the recommended protocol. Quality of delivery concerns whether an intervention was delivered appropriately to achieve what was intended, i.e., “the extent to which a provider approaches a theoretical ideal in delivering program content” [22]. Quality of delivery was assessed using five indicators to measure features of quality of service delivery [23]. Participant responsiveness refers to how participants are engaged by or respond to the intervention [22]. Moreover, the participant responsiveness dimension was measured using nine indicators to assess participants’ perceptions regarding the relevance and usefulness of the CEmONC services. It was measured with a five-point Likert scale ranging from (1 = strongly disagree to 5 = strongly agree). Then the participant responsiveness was dichotomized based on a threshold score. The threshold score was determined using the demarcation threshold formula [24].

$$\left(\frac{\text{Total highest score}-\text{Total lowest score}}{2}\right)+\text{Total lowest score}$$

Accordingly, participants who scored above 65 on the participant responsiveness questions were considered to having a positive attitude otherwise a negative attitude toward the services.

Stakeholders determined the indicators’ weight during the evaluability assessment (Table 1). The weight was calculated using the formula $(\frac{\text{Observed number X Indicator weight}}{\textrm{Expected number}})$ [24]. The achievement was calculated by $(\frac{\text{Score}}{\textrm{Weight}}\text{x}100)$ [25]. Then judgment parameter for the indicators was rated as poor, fair, good, and very good if the score was < 50, 50–74.9, 75–90, and > 90, respectively.

Table 1 Stakeholders’ identification and analysis matrix for CEmONC service at the University of Gondar comprehensive specialized hospital, northwest Ethiopia, 2022

Sample size and sampling procedures

The sample size for quality of delivery and participant responsiveness was calculated using a single population proportion formula: $n=\frac{\text{z}{\left(\frac{{\alpha}}{2}\right)}^{2}\text{x p }(1-\text{p})}{{\textrm{d}}^{2}}$ with the assumption of a 5% margin of error (d), 95% confidence level, p = 50% and 10% non-response rate. Then, the final sample size was 404. To measure adherence, six-month document reviews (client charts documented for six months prior to the data collection period) were reviewed retrospectively. The documents were selected through a systematic sampling technique. Moreover, ten key informants were purposively selected and interviewed for qualitative data to get adequate and in-depth information. These key informants were heads in the delivery, neonatology and postnatal wards.

A systematic sampling technique was used to select the study participants at the maternity unit. All mothers attending CEmONC services at the University of Gondar Comprehensive Specialized Hospital during the data collection period and the maternal and child health (MCH) coordinator working in the maternity ward were involved in this evaluation. Those key informants and CEmONC clients who were severely ill and unable to have an interview were excluded from the interviews.

Data collection tools and procedures

This evaluation used a structured questionnaire for the exit interview, document review checklists, and guiding questions for key informant interviews. The questionnaires were adapted from the Averted Maternal Death and Disability (AMDD) data collection module, WHO CEmONC guidelines, and related literature [8, 26]. These tools were first developed in English, translated to the local language (Amharic), and back to English to ensure clarity and consistency. Three BSC midwives for the data collector and two MSC midwife nurses as supervisors were recruited from the nearby primary Hospital. Before engaging in the data collection process, the data collectors and supervisors were trained for two days on the evaluation objectives, data collection tools and techniques, and ethical issues. The whole evaluation process was supervised daily and closely. A pre-test was conducted on 21 (5%) participants at Felegehiwot Comprehensive Specialized Hospital, and the necessary modification was made.

Exit interviews were done in the waiting room to ensure the clients’ auditory privacy. Moreover, the Cronbach alpha value of 0.72 ensured the reliability of tools for quantitative data. The qualitative data was collected using open-ended questions from ten key informants. An interview with each key informant was audio-recorded, transcribed and translated into text format. Both the supervisor and the principal evaluator checked the accuracy of the data that the data collectors collected daily.

Data management and analysis

Data were cleaned and entered into Epi data version 4.6 and exported to STATA version 14 statistical software for analysis. A binary logistic regression model was fitted to determine the association between dependent and independent variables. The Hosmer and Lemeshow goodness of fit test checked the model fitness. Variables with a p-value of less than 0.2 during bivariable regression were fitted into multivariable logistic regression. Adjusted Odd Ratio (AOR) with a 95% confidence interval (CI) and p-value < 0.05 were used to declare the significant factors and strength of association. Audio-recorded qualitative data were transcribed and translated into text format and then coded and categorized through a thematic analysis approach. Then qualitative and quantitative data were mixed and integrated during the interpretation phase and supplemented the quantitative findings.

The stakeholders determined the weight of each evaluation dimension and the respective indicators used to decide the degree of the implementation fidelity of CEmONC based on the degree of relevance for the program. Accordingly, each dimension was judged as poor, fair, good, and very good if the score was < 50, 50–74.9, 75–90, and > 90, respectively. Then the score of each dimension was aggregated to decide the level of performance of the implementation fidelity of the CEmONC services.

Results

In our evaluation, we conducted a survey among a total of 404 study participants, 423 retrospective document reviews, and ten key informant interviews.

Quality of delivery of CEmNOC services

Socio-demographic characteristics of participants

A total of 404 clients were enrolled in this study. The majority (45.5%) of the clients were aged 25–34, and their mean age was 27.83 (SD ± 5.86). More than three-fourths (76.5%) of the clients were rural dwellers, and 81.9% were married. About 46.5% were housewives, and the religious preference of 80.4% was Orthodox (Table 2).

Table 2 Socio-demographics characteristics of study participants at the university Gondar comprehensive specialized hospital, northwest Ethiopia, 2022 (n = 404)

Obstetric characteristics of participants

Nearly 15% of participants’ recent pregnancy was unplanned. Three-fourths (75.5%) of participants had ANC follow-up 4 + , and nearly two-thirds (65.8%) were multigravida. About 60.3% of deliveries were spontaneous vaginal delivery and 97.6% were alive (Table 3).

Table 3 Obstetrics characteristics of study participants at the University Gondar Comprehensive Specialized Hospital, northwest Ethiopia, 2022 (n = 404)

Quality of delivery of CEmNOC services

This evaluation founds the overall score of fidelity of quality of delivery was 72.7%. Moreover, 68% of clients perceived that healthcare providers respected them during CEmONC service provision. About three-fourths of participants perceived good interaction with healthcare providers, and 78% of clients perceived they got important information about the services. Likewise, about 68% of clients perceived that their confidentiality was kept by healthcare providers (Table 4). Qualitative results supported this finding.

“Since information regarding the accessibility of the CEmONC service in the neighboring region is not well provided adequately, the client flow in the hospital is very large, and clients are served in an insufficient space as a whole. Hence, it has made providing a service which ensured their confidentiality difficult” [Senior Obstetrician working at UOGCSH].

Table 4 Judgment analysis matrix of quality of delivery dimension indicators for CEmONC service at the University of Gondar Comprehensive Specialized Hospital, northwest Ethiopia, 2022 (n = 404)

Participant responsiveness towards the CEmONC services

This evaluation revealed that 68% of the participants had a positive attitude about the relevance of CEmONC services. The key informant interviews supported this finding.

“We would have expected to work more on awareness creation of the community. However, we didn’t do it as sufficiently as possible since there is work overload in the hospital” [ A 26-year-old male mid-wife nurse].

Moreover, three-fourths of participants had a positive attitude about the benefit of breastfeeding initiation within one hour of delivery. Moreover, 82% of participants perceived that ANC follow-up is important, and 85% had a positive attitude toward the importance of institutional delivery. Furthermore, 81% had a positive attitude towards the importance of ultrasound examination and administration of oxytocin. In this evaluation, the overall fidelity of participant responsiveness dimension was 77.2% (Table 5).

Table 5 Judgment analysis matrix of participant responsiveness dimension indicators for CEmONC service at the University of Gondar comprehensive specialized hospital, northwest Ethiopia, 2022 (n = 404)

Factors associated with participant responsiveness towards the CEmONC services

In the multivariable logistic regression analysis, age, maternal education status, maternal occupation status, number of ANC follow-ups, and gravidity were factors associated with participant responsiveness regarding the CEmONC services.

Clients aged 35 years and above were 52% (AOR = 0.48, 95% CI: 0.24,0.98) less likely to have a positive attitude towards the CEmONC services than those aged ≤ 25 years^. In this evaluation, the likelihood of having a positive attitude towards the CEmONC services among clients with the educational status of college and above was 2.61 (AOR = 2.61, 95% CI: 1.46,4.66) times higher as compared to clients with no formal education. Moreover, clients who were government employees were 1.85 (AOR = 1.85, 95% CI: 1.08,3.18) times more likely to have positive attitudes than those housewives. The odds of clients’ attitude towards the CEmONC services among clients who had ANC follow-up was 5.5 (AOR = 5.50, 95% CI: 1.83, 16.47) times higher to be positive than clients who had no ANC follow-up. Furthermore, the odds of clients’ attitude towards the CEmONC services among clients with grand multigravida was 2.17 (AOR = 2.17, 95% CI: 1.08, 4.38) times higher to be positive than clients of primi gravida (Table 6).

Table 6 Bivariable and multivariable logistic regression analysis to assess factors associated with participant responsiveness of the CEmONC services at the University of Gondar Comprehensive Specialized Hospital, 2022 (n = 404)

Adherence of the CEmONC program to the guideline

The retrospective document review showed that parenteral antibiotics, parenteral anticonvulsants, and uterotonic drugs were administered based on the indication for 60%, 85%, and 89% of clients, respectively. Moreover, healthcare providers manually removed placenta, retained products and cesarean delivery per the guideline for 86%, 69%, and 75% of clients, respectively. In this fidelity evaluation, the overall fidelity of adherence dimension was 76.6% (Table 7). The qualitative findings supported this result.

“Since the University of Gondar comprehensive specialized hospital is the only specialized Hospital serving the population of Gondar city and the neighboring zones, the maternity client flow ranges from 160 to 200 clients per day. Implementing comprehensive emergency maternal and child care services fully in line with the recommended protocol is difficult. However, we tried our best to provide the service by the recommended CEmONC guideline” [Female midwife-nurse maternal and child health coordinator].

Table 7 Judgment analysis matrix of adherence dimension for the evaluation of the implementation fidelity of the CEmONC program at the University of Gondar Comprehensive Specialized Hospital, northwest Ethiopia, 2022

Discussion

The overall implementation of the CEmONC program was found to be 75.5% which was judged as implemented in good fidelity. This finding was low compared to the WHO-recommended implementation standard of CEmONC service delivery [27]. This implies that further coordinated activity is needed from the health care providers and government to improve the fidelity status of the services.

Moreover, about three-fourths (75%) of participants perceived that they had good interaction with healthcare providers. This finding was lower than the study conducted in Tanzania, which revealed that 87.4% of clients interacted positively with healthcare providers [28]. This might be due to variations in compassionate and respectful care (CRC) service delivery for the clients.

In this evaluation, about 78% of clients perceived they got important information about their health issues from healthcare providers. An enormous difference had been seen when compared with the study conducted in Kenya, where 41% of clients got sufficient information about their health issues from healthcare providers [29]. The discrepancy may be due to the work overload as a result of high number of clients in the health facilities which limits healthcare providers from providing holistic healthcare information to the clients. Likewise, about 68% of clients perceived that healthcare providers kept their privacy. Our finding was lower than the study conducted in Gondar, which showed that 92.8% of participants believed their privacy was kept [30]. This inconsistency might be due to healthcare providers’ negligence towards ensuring clients’ privacy. Moreover, about two-thirds( 68%) of clients perceived that healthcare providers respected them during CEmONC service delivery. This implies that a substantial number of clients were not delivered respectful care which is a crucial component of quality healthcare.

This evaluation finding briefed that the overall score of fidelity of quality of delivery was 72.7% which was found to be implemented in fair fidelity. This suggests that the program is not consistently delivered according to established standards or protocols.

Results of this evaluation also showed that 68% of participants had a positive attitude about the relevance of CEmONC services. In comparison with a cross-sectional study conducted in Nigeria [31], this finding was lower. This implies that awareness creation both at the individual and the community level regarding the worth of the CEmONC services was not adequately performed. This evaluation revealed that the overall fidelity of participant responsiveness was implemented in good fidelity.

In this evaluation, it is shown that clients aged 35 years and above were 52% (AOR = 0.48, 95% CI: 0.24,0.98) less likely to have a positive attitude towards the CEmONC services than those aged ≤ 25 years^.

It is also observed that education and having a positive attitude towards the CEmONC services had a direct relationship. There was an increase in the likelihood of having a positive attitude towards the services as the education level of respondents increased. The likelihood of having a positive attitude towards the CEmONC services among clients with the educational status of college and above was 2.61 (AOR = 2.61, 95% CI: 1.46,4.66) times higher as compared to clients with no formal education. This might be because a higher level of education enhances the understanding and attitude of clients regarding CEmONC services.

Moreover, clients who were government employees were 1.85 (AOR = 1.85, 95% CI: 1.08,3.18) times more likely to have positive attitude than those housewives. This might be because government employees spend most of their time at work and might get more information from their colleagues about the services.

This evaluation also revealed that having ANC follow-up had a positive role in shaping the clients’ attitudes regarding the BEmONC services. The odds of clients’ attitude towards the CEmONC services among clients who had ANC follow-up was 5.5 (AOR = 5.50, 95% CI: 1.83, 16.47) times higher to be positive than clients who had no ANC follow-up. This might be due to mothers who had ANC follow-up having the opportunity to get more information about CEmONC services from healthcare providers.

Furthermore, the odds of clients’ attitude towards the CEmONC services among clients with grand multigravida was 2.17 (AOR = 2.17, 95% CI: 1.08, 4.38) times higher to be positive than clients of primi gravida. This might be because mothers who had more gravidity might have the opportunity to contact healthcare providers frequently and be more familiar with CEmONC services.

This finding also revealed that the adherence dimensions were implemented in good fidelity. This finding was higher than the study conducted in Mekele public teaching hospitals in Ethiopia, which showed that the overall providers’ adherence to the guideline was 22.8% [32]. This difference might be due to differences in the provision of refreshment training and the utilization of the CEmONC guidelines.

In this fidelity evaluation, the retrospective document review showed that the proportion of clients with parenteral antibiotics administered based on the recommended protocols was 60%, which was judged as fair. This finding was lower than an evaluation conducted on the process of the CEmONC program [33] and a systematic review conducted on antibiotic prophylaxis versus no prophylaxis for preventing infection after cesarean section [34]. This discrepancy might be due to the working setup variation and stocked-out of antibiotic drugs. This finding implies that sufficient measure is not taken to prevent obstetric infections in the evaluation area.

Moreover, parenteral anticonvulsants and uterotonic drugs were administered properly for 85% and 89% of clients and judged as a very good fidelity. As per the preset judgment criteria, these findings were inline with the recommendations of WHO standard treatment guidelines for obstetrics management [35]. This implies that obstetrics-related complications like epilepsy and postpartum haemorrhage were managed appropriately.

Likewise, cesarean delivery and retained products were managed as per the guideline for 75% and 69% of clients. This finding contrasts the requirements of WHO standard treatment for obstetrics management guideline, which recommends that the management of all obstetrics complications should adhere to the protocol [36]. This has a serious implication that the crucial components of obstetrics care which expose clients to further obstetrics complications were not managed as per the protocol.

Strength and limitations

The strength of this evaluation was that it used three dimensions, making the evaluation more valid than assessing its implementation fidelity by a single dimension. In addition, the evaluation employed mixed methods (both quantitative and qualitative methods).

As a limitation, the measure of fidelity of implementation of the CEmONC program in this study was from the clients’ perspectives, which means the study did not consider the providers’ perspectives.

Conclusion

In this evaluation, the overall implementation level of the CEmONC service was found to be implemented in good fidelity. Likewise, the adherence of health care providers to the recommended protocol was implemented in good fidelity. Parenteral antibiotics were not found to be sufficiently administered based on the recommended protocols. Moreover, the quality of delivery was judged as implemented in fair fidelity. The respect for clients from health care providers during CEmONC service delivery was minimally delivered.

Furthermore, participant responsiveness was found to be implemented in good fidelity. Moreover, the client’s attitude towards the relevance of CEmONC services was poor. Age ≥ 35 years (AOR = 0.48, 95% CI: 0.24,0.98), educational status of college and above (AOR = 2.61, 95% CI: 1.46,4.66), being government employed (AOR = 1.85, 95% CI: 1.08,3.18), having ANC follow-up (AOR = 5.50, 95% CI: 1.83, 16.47) and grand multigravida (AOR = 2.17, 95% CI: 1.08, 4.38) were factors significantly associated with participant responsiveness towards the services.

It is recommended that the University of Gondar Comprehensive Specialized Hospital is shall provide parenteral antibiotics drugs sufficiently. Healthcare providers are advised to make strong efforts towards adherence to the recommended guidelines of CEmONC services.

The University of Gondar Comprehensive Specialized Hospital with other stakeholders shall facilitate on-the-job training for healthcare providers regarding compassionate and respectful care (CRC). Awareness creation for women of reproductive age regarding the CEmONC services shall also be emphasized. The government, partners, programmers, and implementers shall give primer attention to improving the implementation fidelity of CEmONC services.