Description of SMI
SMI was planned from the beginning to have two to three interconnected phases. A first phase of SMI (2013–2014) supported activities to improve the availability of staff, equipment, and supplies in MOH health facilities in all participating countries —such as improving supply chains and inventory management systems, designing and beginning implementation of the EONC strategy, reviewing staffing policies, and creating capacities for emergency obstetric and neonatal care. In a second phase (2015–2017), SMI incorporated supply-side and demand-side interventions aiming to improve the coverage and quality of care. In addition to previous interventions, interventions in this phase included the implementation of community and individual birth plans, behavior change communication strategies with cultural adaptations (specifically targeted media, speakers, messages, images, and languages), establishing or improving community platforms, strategies for early catchment of pregnant women in communities and facilities, funding systems for the transportation of pregnant women, operation of maternity homes, strengthening referral systems, establishing user management systems, training traditional birth attendants to detect risks and refer cases, implementing continuous quality improvement strategies and health care network management initiatives (for example, managerial electronic dashboards, local implementation plans, improved supervision tools, etc.), among others [19,20,21]. The third phase is currently in progress in Nicaragua and Honduras.
Design
The SMI impact evaluation has a quasi-experimental design, with three partial-panel measurements in health facilities and two repeated cross-sectional measurements in households. This analysis uses information collected in health facilities and households in intervention and comparison areas of the Initiative at baseline (2011–2013) and following implementation of two phases of interventions (2017–2018).
Population and sample
This analysis is restricted to Honduras, Nicaragua, and Guatemala, countries for which we have the required information for this analysis. Further details on SMI measurement methodology have been published elsewhere [22]. In summary, in each country, SMI administrators identified municipalities based on two criteria: first, those with the highest concentration of population in the poorest income quintiles according to national poverty estimates, and second, considering their proximity to enable interventions in the entire healthcare network. In Nicaragua, intervention municipalities were selected according to the concentration of the population in extreme poverty based on unmet basic needs; in Honduras, municipalities were selected according to the national poverty index and healthcare network manager (healthcare network managers with the most municipalities with high poverty were randomized in intervention and comparison groups); and in Guatemala, intervention municipalities were selected according to the concentration of extreme poverty, from departments with the largest concentration of the population in extreme poverty. Next, they identified municipalities with similar levels of poverty in Nicaragua and Guatemala to serve as comparison areas.
To get information from households, we conducted our own census in a set of communities selected with probability proportional to size to identify eligible households with women aged 15 to 49 years and children under 5. Among eligible households, we selected a random sample of 30 households per community to conduct the full SMI survey. To get information from health facilities, at the baseline, among the facilities that provide services to communities selected for the household survey, facilities that provide ambulatory EONC were selected at random from the MOH roster of facilities. Given the small number of facilities providing basic or comprehensive EONC in the study areas, we included in our sample all facilities of those levels in the intervention and comparison areas. We used a similar procedure for the selection of health facilities in the follow-up measurement, with the only difference that half of selected ambulatory-level facilities included in the follow-up sample were selected from facilities visited in the baseline measurement. Due to safety issues, data collection in Nicaragua during the baseline measurement had to be interrupted, allowing us to get information from 31% of the facilities providing basic or comprehensive EONC in the proposed sample.
Data collection
The baseline surveys were conducted in Honduras between January 17 and June 1, in Nicaragua between March 1 and August 29, and in Guatemala between April 15 and August 11, all in 2013. Follow-up data were collected in Honduras between May 29 and October 25, 2017, in Nicaragua between June 14 and December 20, 2017, and in Guatemala between May 7 and August 29, 2018.
SMI used a computer-assisted personal interviewing (CAPI) system for data collection. The household survey was conducted at each household by trained interviewers, who read the questions to the interviewees and registered the answers in the computer, and assessed utilization of health services, barriers to care, and population health outcomes alongside health system infrastructure and delivery care components. Interviews with indigenous-language speakers were conducted in the corresponding indigenous languages by interviewers fluent in them, using a standard translation. Respondents were asked to indicate which health facilities were visited for different types of care, allowing us to link household information with facility conditions and services. The household survey included an interview of the head of household or person best informed about the household to collect information on household services and materials, ownership of assets (durable goods, land, livestock), household expenses, and sources of health care financing. All women of reproductive age (15 to 49 years) in the household were interviewed about their demographic characteristics, access to health care, current health status, recent history of illness and associated medical expenses, birth history, knowledge and use of family planning methods, exposure to health interventions, and satisfaction with community health workers. Women with children 0 to 59 months old were asked questions about health conditions and health services utilization of each child.
The health facility survey was conducted by trained medical personnel using CAPI and collected data on facility conditions, service provision and utilization, and quality of care. Health facilities were grouped according to three levels of EONC – ambulatory, basic, and comprehensive – following the official classification of each country. Mainly, ambulatory facilities provide outpatient care; basic facilities are able to attend uncomplicated vaginal deliveries and provide immediate emergency obstetric and neonatal care; and comprehensive facilities have a surgery room and are able to attend most obstetric and neonatal complications (not including intensive care). The facility director or person in charge of the health facility was interviewed by a trained interviewer in the facility to capture information on general facility characteristics, infrastructure, human resources, supply logistics, infection control, child health care, vaccine availability, family planning service provision, availability of contraceptives, and antenatal, delivery, and postpartum care services. Surveyors used an observation checklist to record direct observations of the availability and functionality, as applicable, of essential equipment and supplies required for maternal and child health care, including pharmaceuticals.
The health facility survey also included a medical record review (MRR) where information was extracted from a random sample of medical charts of women who had given birth in the facility in the 2 years prior to data collection. The methods for selection and review of records have been described elsewhere [22,23,24].
The study received institutional review board (IRB) approval from the leading institution and the MOH in each country. Methods were carried out in accordance with the national guidelines and regulations of participant countries and the Declaration of Helsinki. All women and personnel responsible for health facilities responding to the interviews signed informed consent forms prior to data collection. Identifying personal information was not collected in any component of the study.
Study variables
Institutional delivery
We define as institutional delivery any delivery that took place in a health facility regardless of facility type (public/private) or level (ambulatory, basic, or comprehensive EONC). For births in health facilities, our sample is restricted to births for which the nearest facility could be identified and was included in the study (see Fig. 1). The household survey collected information on the place of delivery for all births of women living in the household in the last 5 years. The analysis includes each woman’s most recent birth during the last 5 years in the baseline measurement. At the follow-up, only births from the last 2 years are included in the analysis in order to exclude births that took place before the SMI interventions were implemented.
Place of delivery and distance from home to delivery place
In addition to recording municipality and locality of residence, the household survey asked women about the name of the closest health facility, the health facility she usually attends, and the most recently visited health facility; the name and type of the facility she attended for her delivery (if any), and the self-reported distance (km), travel time (hours or minutes), and mode(s) of transport to the delivery facility. Because women typically reported ambulatory care facilities, which are limited to outpatient care and do not routinely provide delivery services, as their nearest and usual health facilities, we determine as the “closest” health facility for delivery a public establishment where the woman would be expected to deliver based on municipality and locality of residence, the referral network of the public health system, and a locality-by-locality review of the facilities women reported attending for delivery and for other health care needs. In this manner, we matched each selected community to its expected delivery facility. This matched facility is hereafter referred to as the “closest” facility for delivery.
The health system referral networks are based on municipal boundaries, and most municipalities have at least one basic or comprehensive EONC level facility, which in the majority of cases was also the nearest facility with capacity to attend delivery by travel time. Women in two municipalities (one in Nicaragua at the baseline, and one in Honduras in both rounds) matched to facilities that were not included in the health facility survey; we excluded these deliveries from the analysis of delivery location since facility-level variables were not measured at their closest facility.
Health facility characteristics and delivery capacity
We use the information from the health facility survey to construct a 6-point score of capacity to attend uncomplicated vaginal deliveries that included round-the-clock availability of skilled birth attendants (doctor or nurse, 2 points), availability of all basic equipment for antenatal and postpartum care (gynecological or exam table, lamp, obstetric tape measure, sphygmomanometer, and stethoscope, 1 point), and availability of basic inputs for delivery care such as oxytocin (to start labor, increase speed of labor, or stop bleeding after delivery, 1 point), methylergometrine or ergometrine maleate (to prevent excessive bleeding following childbirth, 1 point), and Ringer’s lactate/Hartmann’s solution or saline solution (for fluid resuscitation after blood loss, 1 point).
Additionally, we use information extracted from the charts of patients with uncomplicated vaginal deliveries at these health facilities to construct two indicators for quality delivery care as defined through SMI. First, the survey recorded whether the patient was administered oxytocin or another uterotonic after delivery according to standards for active management of the third stage of labor. Second, it collected information on immediate postpartum care, including checks of the mother’s temperature and blood pressure during the first 2 h after delivery and at discharge from the health facility.
Characteristics of delivery
The household questionnaire collected information on which health personnel were present during delivery, reasons for not delivering in a health facility (when the delivery occurred outside a health facility), accompaniment by a traditional birth attendant, type of delivery (planned C-section, emergency C-section, or vaginal delivery), seizures experienced during delivery, receipt of antenatal care, advice to have the delivery in a health facility, and advice to create a transportation plan.
Women and household characteristics
We collected information on the woman’s age, marital status, literacy, education (no school attendance, primary, secondary, high school, or university), occupation (housewife versus other [employed, student]), and number of previous pregnancies. In order to measure household conditions, we calculated a household asset index based on household assets (including piped water, improved toilets, and having a designated kitchen area, electricity, radio, stereo, television, telephone [mobile and fixed line], refrigerator, laundry machine, computer, bicycle, guitar, scooter, car, truck, land, cattle, mules, goats, chickens, or pigs). We collected information on the itemized monthly household expenditure as reported by the survey respondent and calculated per capita expenditure quintiles in each country and round. We use the countries’ census data [25,26,27] to define urban or rural status using 2500 inhabitants as a threshold.
Statistical analysis
We analyze the impact of SMI on three outcome variables: institutional delivery (1 = delivery in a health unit vs. 0 = delivery outside a health unit); choosing a more distant facility for delivery (1 = delivery at a more distant facility vs. 0 = delivery at the closest facility) and the capacity score of facilities to attend deliveries. We restrict our sample to women who gave birth in a health facility, if the facility was included in the study, for the analysis of choice of delivery facility. Additionally, we exclude deliveries via C-section (to account only for uncomplicated deliveries) in the analysis of choice of delivery facility (see Fig. 1). We use a difference-in-difference approach fitting linear models to assess the impact of the intervention on the facility capacity score, institutional delivery, and delivering in a more distant facility. We include as covariates in our models the intervention or comparison group, measurement time (baseline or follow-up), and an interaction term between intervention and time, which is our impact estimate. In order to control for baseline differences between intervention and control groups, we adjust the models by baseline women’s characteristics (age, parity, education, household conditions, exposure to health interventions), health facility level, and country. In sensitivity analyses, we conduct the analysis substituting per capita itemized monthly household expenditure for asset indices in each model and find no meaningful changes in results. We fit models using Poisson and conditional logit specifications as robustness checks. Since results yield similar conclusions, only results from the linear models are presented.
We calculate the probability of women mentioning different reasons for not attending the delivery in a health institution from a series of logistic regression models. Since the choice of the place to attend delivery is not entirely an individual one, but it is influenced by other social and contextual factors, we group the reasons provided by women as cultural beliefs, finances and logistics, health facility limitations, and other, adjusted for age, education, parity, urban residence, asset index, and maternal literacy. Cultural beliefs include individual factors but also others related to the role of the family and community, like preferring labor under the care of a traditional birth attendant; preferring to give birth in the family home or another house; religious or cultural beliefs; wanting a traditional birth attendant to accompany; or being prevented from going by husband, partner, or another member of the family. Finances and logistics refers to reasons related to transportation problems, travel times, facility too distant, problems finding transportation, lacking someone to travel with, having no place to stay, not knowing where to go, and health facility charges for delivery. Health facility limitations included problems with health facilities (not having sufficient drugs or ill-equipped), problems with staff (not staffed, staff not trusted, not well informed or difficult to deal with, being previously treated poorly by the health facility), and having care denied when they have tried to go to a health facility. Other obstacles groups not being advised to deliver in a health facility and other reasons.
We used Stata version 15 [28] and R version 3 [29] for analyses. All analyses are conducted for the pooled sample and individually by country. Analyses using data from the household survey are weighted and adjusted for clustering and stratification in the sample design.