eRegistries: indicators for the WHO Essential Interventions for reproductive, maternal, newborn and child health
BMC Pregnancy and Childbirth volume 16, Article number: 293 (2016)
Electronic health registries – eRegistries - can systematically collect relevant information at the point of care for reproductive, maternal, newborn and child health (RMNCH). However, a suite of process and outcome indicators is needed for RMNCH to monitor care and to ensure comparability between settings. Here we report on the assessment of current global indicators and the development of a suite of indicators for the WHO Essential Interventions for use at various levels of health care systems nationally and globally.
Currently available indicators from both household and facility surveys were collated through publicly available global databases and respective survey instruments. We then developed a suite of potential indicators and associated data points for the 45 WHO Essential Interventions spanning preconception to newborn care. Four types of performance indicators were identified (where applicable): process (i.e. coverage) and outcome (i.e. impact) indicators for both screening and treatment/prevention. Indicators were evaluated by an international expert panel against the eRegistries indicator evaluation criteria and further refined based on feedback by the eRegistries technical team.
Of the 45 WHO Essential Interventions, only 16 were addressed in any of the household survey data available. A set of 216 potential indicators was developed. These indicators were generally evaluated favourably by the panel, but difficulties in data ascertainment, including for outcome measures of cause-specific morbidity and mortality, were frequently reported as barriers to the feasibility of indicators. Indicators were refined based on feedback, culminating in the final list of 193 total unique indicators: 93 for preconception and antenatal care; 53 for childbirth and postpartum care; and 47 for newborn and small and ill baby care.
Large gaps exist in the availability of information currently collected to support the implementation of the WHO Essential Interventions. The development of this suite of indicators can be used to support the implementation of eRegistries and other data platforms, to ensure that data are utilised to support evidence-based practice, facilitate measurement and accountability, and improve maternal and child health outcomes.
Lack of quality data on the health status of individuals is a major contributor to poor health outcomes at a population level . Progress towards the United Nations recently proposed Sustainable Development Goals (SDGs)  can only be monitored with efficient and effective health information systems. Despite this obvious need, most low- and middle-income countries (LMIC) have insufficient systems in place for the collection, analysis and reporting of health data, severely hampering both health system and policy-level decision-making .
Improvement of data collection is ongoing in many countries with the advent of electronic methods of data collection, including electronic medical records to replace paper-based systems. The eRegistries Initiative aims to support a safe and efficient transition to integrated electronic health information systems in LMIC [4, 5] (Frost M, Hodne Titlestad O, Lewis J, Mehl G, Frøen JF: eRegistries: Architecture and Free Open Source Software for maternal and child health Registries, submitted). eRegistries should collect and manage information that is adapted both to the health system’s information needs, as well as the data collection and real-time analysis methodology. Many existing indicators for monitoring health have been designed and constructed to be reliably measured from household and facility surveys [6, 7]; for example, breastfeeding rates for children <6 months of age. While these surveys are critical for national and global monitoring needs , the historically weak data-collection capabilities of LMIC [9, 10] render the utility of such indicators limited in the context of the current global transition to more robust health information systems.
Further, while these indicators are designed to guide national policy level decisions and are collected through relatively infrequent population surveys, improvement in service delivery is not their primary aim. Similarly, many facility-based surveys only include indicators to assess readiness to deliver healthcare, leaving little attention paid to quality of care. There are gaps in the specific indicators needed to monitor and improve quality of care, which are needed to supplement the existing facility readiness and the population health indicators.
The Donabedian framework  describes three types of indicators typically needed for monitoring the provision of healthcare: structure, process and outcome. In this framework, ‘structure’ refers to the assets needed to support health care, also known as ‘inputs’ and ‘outputs’ in monitoring and evaluation frameworks , and include access, supplies and personnel. Process indicators refer to actions/activities performed during the delivery of health care and may be known as coverage (or, as in monitoring and evaluation frameworks, outcome) indicators. The Donabedian framework uses ‘outcomes’ indicators to refer to the endpoint of the individual health status, community or population; these are often termed impact indicators by monitoring and evaluation frameworks. ‘Bottleneck analyses’ consider six factors to assess how to change impact: three structural (access, availability, training) and three process (initial coverage, continued coverage, effective coverage) indicators [13, 14]. Effective coverage indicators attempt to include all aspects of quality within a single final indicator that summarises the proportion of patients receiving high-quality of care.
In this second paper of eRegistries series, we report on indicators for use in monitoring the WHO Essential Interventions Commodities and Guidelines for RMNCH , which aim to improve quality of care and health outcomes in reproductive, maternal and newborn care.
This component of the eRegistries Initiative had two main aims:
to assess the status of global indicators by describing the data that are currently available to address the WHO Essential Interventions, and
to develop a robust suite of indicators and respective data points to monitor WHO Essential Interventions – both screening and management components – for use in eRegistries.
We considered the 45 WHO Essential Interventions delivered up to 6 weeks post-partum, including those for: (1) preconception and periconceptual care; (2) antenatal care; (3) childbirth care; (4) postpartum care (of the mother); (5) immediate care of the newborn; (6) neonatal infection management; and (7) care for small and ill babies.
Assessment of the current status of global indicators
We first assessed the extent to which the 45 WHO Essential Interventions  were being addressed in either household or facility surveys, to better understand the landscape of current data needs and gaps that our indicator development project should address. Our search focussed on the most recently reported process and outcome indicators identified from globally recognised sources. First, we examined the common international databases which compile indicators from various sources. These included the Countdown to 2015 databases, UNICEF databases, WHO observatory, and UNAIDS. These sources typically reported household surveys, primarily the large scale multi-country initiatives of the Demographic and Health Surveys (DHS) and Multiple Indicator Cluster Survey (MICS). The DHS and MICS websites were then assessed for additional information which was not yet available in the large compilations already mentioned. The average value for the most recent estimate of each indicator was calculated in Microsoft Excel (Redmond, WA).
As household surveys are not designed for, nor capable of, assessing process indicators relevant to care delivered at facilities, we also assessed the availability of indicators of key interventions directly from facility-based sources. Eight survey instruments were identified which collected data through either direct observations of patients or via medical records review; we did not consider routine health information systems as each country has their own unique set of indicators that may or may not be available in the public domain. Two published reviews [16, 17] which identified available survey tools or instruments were sourced. Surveys mentioned in these two sources that did not focus on health, or were not used in multiple LMIC, were excluded. In addition, to ensure accuracy, two facility survey programs were directly contacted for supplemental information. These facility surveys have several purposes, one of which is to assess the readiness of a facility to implement quality emergency obstetric care. The survey instruments ultimately included were: Service Provision Assessment (SPA) (2012 version), Averting Maternal Death and Disability (AMDD), the Maternal and Child Health Integrated Program Quality of Care survey (MCHIP-QoC), the Postpartum Hemorrhage Prevention and Treatment questionnaire on Active Management of the third stage of Labor (POPPHI), the World Bank’s Service Delivery Indicators tool (SDI) (2012 Kenya version), WHOs Service Availability and Readiness Assessment (SARA) (version 2.1), as well as the WHO’s surveys on Perinatal Health and on Maternal and newborn health from 2007 and 2010, respectively. All documents reviewed were the most recent versions available in November 2014, unless otherwise stated. One instrument included information on management of post-partum haemorrhage and was only included in our results as a denominator for relevant interventions. Data were extracted from website materials or survey instruments. Where information was unavailable, survey support staff were interviewed to identify the number of national facility surveys performed for the specified interventions. For each intervention, three domains related to process indicators were assessed: training and knowledge, availability of supplies (inventory), and whether the intervention was actually performed. The number of survey instruments collecting any information relevant to each of the 45 Essential Interventions in each of these three domains was measured through careful reading of each survey module.
Defining indicators and data points
For each of the 45 Essential Interventions, we conducted a comprehensive search of existing WHO indicators, followed by indicators from other professional bodies (Table 1). Existing indicators were adapted or developed as required by the eRegistries technical team with reference to the practice guidelines and training manuals cited within each WHO Essential Intervention, and other resources where appropriate.
Four types of indicators were defined for each WHO intervention:
process indicator/s for screening/risk identification (the proportion of patients for whom screening tests/risk identification measures were performed);
outcome indicator/s for screening/risk identification (the proportion of patients screening positive/identified as ‘at-risk’);
process indicator/s for treatment/management (the proportion of patients treated); and
outcome indicator/s for treatment/management (the proportion of patients with adverse outcomes in the population).
For some interventions, screening/risk assessment indicators were not applicable. We considered screening/risk indicators not applicable where the given treatment/management was recommended to all women or babies of a given clearly defined population (e.g. Antenatal care essential package for all pregnant women; Provision of thermal care for all newborns to prevent hypothermia; Early initiation and exclusive breastfeeding).
By data points we refer to the primary data being captured at the point of care, which is the source of information for numerator or denominator. Data points that could be readily collected using an electronic form addressing each of the process and outcome indicators were included. Data items specifically measuring each indicator (numerator and denominator) were developed balancing specificity and feasibility by country resource level. To maximise the feasibility of data collection, for defined conditions such as preeclampsia, simple data points referring to the diagnosis (yes/no) of the condition were accepted, rather than delineated by the individual components of the clinical and laboratory diagnosis.
Evaluation and refinement of indicators
Evaluation and refinement of the indicators occurred via two stages: 1) Expert panel evaluation and; 2) Response to feedback and refinement within the eRegistries technical team.
Expert panel evaluation
An international group of 47 experts in maternal and child health was assembled via the network of the International Stillbirth Alliance. Thirty-four panel members were invited to participate in consultation round 1; 35 in consultation round 2; and 44 in consultation round 3. Invited evaluators included researchers, senior clinicians and academics, obstetricians, neonatologists, maternal-fetal medicine specialists, epidemiologists, consumer advocates and others.
The eRegistries indicator evaluation tool assessing 10 domains was developed. The domains were derived by the eRegistries technical team after reviewing several existing indicator evaluation frameworks, including the New Economics Foundation AIMS criteria for indicators [18, 19], the Agency for Healthcare Research and Quality standards by which to judge quality indicator performance , Indicators to Monitor Maternal Health Goals , and the SMART criteria . The evaluation tool (Additional file 1) was simplified based on pilot testing with a subsample of the expert panel, culminating in the final evaluation tool assessing the below five domains.
eRegistries indicator evaluation criteria:
Action focused: “It is clear what needs to be done to improve outcomes associated with this indicator (e.g., immunised with tetanus toxoid to reduce neonatal tetanus)”
Important: “The indicator and the data generated will make a relevant and significant contribution to determining how to effectively respond to the problem”
Operational: "The indicator is quantifiable; definitions are precise and reference standards are developed and tested or it is feasible to do so"
Feasible: “It is feasible to collect data required for indicator in the relevant setting”
Simple and valued: “The people involved in the service can understand and value indicator”
Panel members were asked to indicate via categorical response their agreement with each of five statements addressing the above domains (Yes/Probably/Unsure/Possibly/No/Do not wish to respond). A comment box was provided for each indicator to enable detailed feedback. Panel members were invited to suggest other indicators or adjustments to the existing indicators. Data were analysed descriptively in Microsoft Excel by tallying the number of responses for each response category.
Indicators were evaluated in three consultation rounds: (1) Preconception/periconceptual care and antenatal care; (2) Childbirth care and postpartum care (of the mother) and; (3) Immediate newborn care, neonatal infection management, and care for small and ill babies. Additional file 2 presents a list of interventions addressed, including the number of indicators within each intervention, and where the indicators were sourced. Panel members were assigned 3 to 4 interventions in each round each and asked to evaluate all indicators within the given intervention. Interventions were assigned to evaluators randomly, unless the panel member indicated a preference based on their area of expertise.
For each intervention the panel members were given a detailed breakdown of the indicators which included definitions, numerators and denominators, data points, and references. An evidence summary for the interventions was provided based on the evidence cited in the Essential Interventions (predominantly Cochrane systematic reviews). Panel members were given an evaluation sheet along with a separate document containing background information and evaluation instructions, including further detail on the eRegistries indicator evaluation tool development. Evaluation materials were sent to and returned by panel members by email.
We adopted a quasi-anonymous approach for indicator evaluation; that is, while individual panel members may have known the names of other members in the group, individual responses were not identifiable to the group and panel members were not aware which interventions had been assigned for evaluation to whom.
Response to feedback and refinement within the eRegistries technical team
Following descriptive analyses, indicators that consistently failed to meet (or ‘probably’ meet) the defined criteria were amended based on the evaluators’ comments, or else removed if deemed unnecessary for effective monitoring and evaluation of the given intervention. A series of meetings of the eRegistries technical team was held to review the updated indicators to ensure consistency in nomenclature across indicators and their data points, numerators and denominators.
Graphical display of potential utilisation of the eRegistries indicators
To inform the plausible utilisation of these indicators, a power graph was created reflecting different use cases. The power to detect a significant change in a given indicator was graphed in association with the indicator prevalence and a given sample size. Three scenarios of different sample sizes were assumed: 200 births annually, 10,000 births annually and 500,000 births annually, to reflect a typical rural clinic, a typical district and a typical LMIC. The graphed indicator prevalence ranged from 75 % to 0.01 %. The most likely value for each of the indicators was calculated (Additional file 3) and placed alongside the graph.
Assessment of the current status of global indicators
Of the 45 WHO Essential Interventions, only 16 were addressed in any of the household survey data available. Of these 16 interventions, only 7 had more than one indicator for screening and management available. In addition to the indicators themselves, data were often available on contact between a woman and the health system during either antenatal care (one or four antenatal care visits) or during care at birth (either facility delivery or skilled birth attendance) (Table 2, population data). For example, data were available for caesarean section, ANC visits, skilled birth attendance, and facility delivery from 70, 73, 75, and 75 countries, respectively, with at least 75 % of these data from 2010–2015.
The facility-based instruments did collect data on process indicators for many of the Essential Interventions, but were more focused on readiness to deliver antenatal care and emergency obstetric care; outcome indicators were rarely included. The availability of supplies was universally available from the survey instruments for only four interventions, while the supplies available for seven interventions were not tracked in any of the instruments. Similar results were observed for training relative to the Essential Interventions and for whether the intervention was actually performed. In addition, eight of the WHO Essential Interventions were not able to be tracked at facilities by any of the identified facility survey instruments (Table 2, facility data).
Compilation and expert assessment
A total of 216 indicators were assembled across the following areas: 107 for preconception and antenatal care; 53 for childbirth and postpartum care; and 56 for care for newborns and small and ill babies. Of these, 122 were sourced or modified from existing indicators identified in instruments from available data sources and 94 were developed by the eRegistries technical team. Indicators were subsequently independently reviewed by evaluators. Response rates across the evaluation rounds were 23, 21, and 25 (68 %, 60 %, and 57 % respectively), with 31 evaluators in total participating (see Fig. 1 for distribution of countries represented). Some evaluators agreed to score a second set of indicators within the same round.
Indicators for preconception/periconceptual care and antenatal care generally fulfilled the criteria ‘Action focused’, Important’, and ‘Simple and valued’. Indicators were less often deemed ‘Operational’ and ‘Feasible’, particularly for cause-specific mortality where capacity in some regions to accurately attribute death to specific conditions was considered lacking. The lack of availability of resources and skilled health personnel in LMIC settings was also seen as a barrier to the utility of many indicators. Figure 2 presents evaluation data for the treatment outcome indicator ‘Malaria-specific stillbirth rate (per 100,000 births)’, which represents the typical pattern of results received across evaluation rounds.
Indicators for childbirth care and postpartum care (of the mother) interventions were deemed to meet (or probably meet) all criteria by the majority of evaluators. Indicators consistently scored lowest on the criteria ‘Feasible’ and (less often) ‘Operational’, with variable evaluations on the criteria ‘Action focused’, ‘Important’, and ‘Simple and valued’. Panel members raised concerns around the feasibility of data collection in LMIC where many births occur at home and where follow-up is difficult. Difficulty accurately attributing cause of death for specific conditions/complications was again raised for all settings, but particularly in LMIC. The feasibility and appropriateness of the maternal near-miss indicators introduced in this round was questioned. A number of panel members thought that health workers may lack understanding and/or appreciation of the concept of near-miss, and some anticipated the indicators may lead to data manipulation in order to conceal what may be perceived as suboptimal care. Some of the indicators associated with the HIV interventions appeared too complex and needed clarification and simplification.
Indicators around newborn care received more mixed evaluations across the criteria, again, with feasibility a clear impediment. Evaluations varied on the criteria ‘Action focused’, ‘Important’, and ‘Simple and valued’.
Response to feedback and refinement within the eRegistries technical team
Amendments were made to a number of indicator titles, data points and definitions based on feedback from the expert panel (see Additional file 4 for examples). Amendments to indicators and definitions involved better operationalisation of key terms (e.g. breastfeeding counselling, ‘successful’ ECV, continuous support during labour). Indicators were reworded and redefined to improve direct measurability (e.g. Antenatal detection of breech presentation was revised to ‘Proportion of pregnant women who have presentation of baby checked by skilled birth attendant at or after 37 weeks of gestation’). Indicators and data points were also altered in some cases to ensure applicability in different settings; for example, the indicator measuring the proportion of women with postpartum haemorrhage who received therapeutic oxytocin was deemed too restrictive for settings where other uterotonics such as misoprostol are used.
Following refinements, there were a total of 193 unique indicators: 93 addressing preconception and antenatal care; 53 addressing childbirth and postpartum care; and 47 addressing care for newborns and small and ill babies Additional file 5 presents the final list of indicators including definitions, data points and sources (full indicator reports are available from the authors on request).
A series of meetings of the eRegistries technical team was held to review the updated indicators to ensure consistency in nomenclature across indicators and their data points, numerators and denominators. We established a set of definitions to guide the use of denominators and data points to ensure consistency across the indicator suite (Additional file 6).
Display of utilisation of the indicators
Figure 3 demonstrates the association between statistical power, sample size and the prevalence of an indicator. As shown, there is likely to be insufficient power when measured at a clinic or district levels to determine differences in rare outcomes such as mortality, while adequate power would exist at national levels. In small clinics, there is likely to be sufficient power to measure differences in common management indicators, such as skilled birth attendance and iron supplementation.
We used the Donabedian framework  to determine the indicators required to successfully monitor the WHO Essential Interventions. This framework encompasses the intuitive relationship between three related concepts: first, the structures of health care are defined as the physical and organisational aspects of care settings (e.g., facilities, equipment, personnel); second, the processes of patient care in order to improve patient health; and, third, the well-known concepts of outcomes of medical care. Four groups of indicators were needed to clearly identify areas for quality improvement: process indicator/s for screening/risk identification (i.e. the proportion of screening tests/risk identification measures that were performed); outcome indicator/s for screening/risk identification (i.e. the proportion of women screening positive/identified as at risk); process indicator/s for treatment/management (i.e. the proportion of women treated); and outcome indicator/s for treatment/management (i.e. the proportion of adverse outcomes in the population). This work also drew upon health service ‘bottleneck analyses’ [13, 14], which consider structural and process factors that influence service delivery.
Our review of existing household and facility surveys which aredesigned to support national and global decision making, revealed a large gap in the availability of information to support the implementation of the WHO Essential Interventions, showing a critical need for improvements in data collection to monitor these interventions consistently across countries. Many of the needed indicators cannot be assessed retrospectively at the population level due to recall issues and lack of medical knowledge. On the other hand, many of the facility surveys focus narrowly on the structures of health care delivery. This leaves a large gap specifically in the arena of process indicators. In addition, the survey results and instruments evaluated demonstrate that several key interventions are not being monitored at all, even within the domains of structure or outcome. For example, there is no population level information to monitor antibiotics for pPRoM, while family planning activities and prevention and treatment of HIV are monitored relatively comprehensively. Although smoking cessation is recommended, none of the facility survey instruments included questions to assess this intervention and only one survey included questions on structural components to support the availability and utilization of calcium supplementation. These large gaps indicate that there is insufficient information available to guide countries and programs as they attempt to ensure the availability of the WHO Essential Interventions in their facilities and for their populations. There is a minimum of information to support the decision making at any of facility, national or global levels. It is this gap which the current indicator selection attempts to address.
We assembled a set of clearly defined process and outcome indicators, with a comprehensive suite of synergistic and consistent data points for effective measurement, for the 45 WHO Essential Interventions spanning preconception to newborn care, which can be collected from routine sources or facility surveys. Further, the eRegistries indicator evaluation tool makes a unique contribution to the metrics field, as it enables a priori assessment of the ‘likely’ or ‘potential’ utility of novel indicators. In our literature review, we found a paucity of information on indicator evaluation approaches to that did not rely on post-hoc review of the data generated by the indicator.
Feasibility in LMIC (informed by the direct experiences of many expert panel members) was a clear concern throughout the indicator evaluation process. Indeed, household surveys would, for example, not be able to assess all indicators proposed. Not all denominators would be measureable, making the results uninterpretable, and unattended home births would skew the numbers dramatically. However, the purpose of the proposed indicators is to facilitate uniform data collection in the presence of health workers documenting their actual work in a setting that accommodates an eRegistry or similarly-structured system for prospective data collection. Therefore, that a service (e.g. induction of labour for prolonged pregnancy) is not offered in a particular setting does not necessarily pose a problem for feasibility of data collection inherent in the indicator itself.
This study has several limitations. Although the expert panel consisted of members from low, middle and high-income settings, it did not include health systems specialists or programme managers, who may offer additional expertise. A legitimate feasibility problem may be posed by the use of cause-specific morbidity and mortality indicators, due to difficulty ascertaining cause of death, especially in some low-resource settings. This underscores the critical need for a quality, international, standardised cause of death classification system for maternal and perinatal mortality. Consistent application of the WHO’s International Classification of Diseases for Maternal Mortality and Perinatal Mortality (ICD-MM and ICD-PM)  may therefore enhance the utility of some of the indicators.
The indicators presented here are important for understanding the interplay between the patient and the health system, and how health system improvement can be achieved. The indicators presented do not represent an exhaustive list of the necessary indicators for all population-based or facility-based assessments, and should not be used in isolation. While these indicators are designed for use in settings with a functional health system collecting information on care-provision provision across the continuum of care, they are developed to minimise erroneous estimates in settings where a significant proportion of individuals do not attend a health system, and thus may not be included in denominators of the population. For example, where pregnancies in need of ANC are used in the denominator for ANC indicators, we define an eligible pregnancy as “a woman having one documented ANC visit (unless the ANC visit is only for termination of pregnancy), OR any data documenting a pregnancy outcome or infant at any point of care” (Additional file 6). This means that in an eRegistry setting tracking individuals, only individuals who are not in contact with any professional care provider from the health system through pregnancy, childbirth, post-partum care, newborn and infant care including vaccinations, will be missed as a population denominator.
In addition, these indicators are not a comprehensive set of indicators required to monitor a health system. These indicators are those that can be collected through an eRegistry and are necessary to address the 45 WHO Essential Interventions. Additional indicators for other components of a health system are clearly needed, as the WHO list does not include all interventions delivered by a health system (e.g. the management of diabetes in pregnancy). Other data sources, and potentially other indicators, should be utilized to understand the full continuum of availability, coverage and quality of community based activities, not only as stand-alone activities but also linked with facility-based activities.
The development of the eRegistries indicators is part of a global push to increase quality of care through an increased focus on measurement. The WHO recently published their vision for quality of care for maternal and newborn health , drawing partially upon the frameworks discussed in this paper, and have begun developing metrics to address quality of care. Two global action agendas, (‘Strategies toward Ending Preventable Maternal Mortality’ (EPMM) and the ‘Every Newborn Action plan’ (ENAP)) are also in the process of finalizing consensus metrics to increase the global ability to measure quality of care. These strategies contributed to the development of UN Secretary General’s Global Strategy for Women’s Children’s and Adolescents’ Health (2016–2030) which addresses relevant issues across the health in order to attain the related SDGs.
The eRegistries indicators contribute to the Indicator and Monitoring Framework  developed to support monitoring progress in implementation of the Global Strategy for Women’s Children’s and Adolescents’ Health (2016–2030) . ‘Quality indicators’ are needed for monitoring healthcare, whether for internal quality improvement or for external accountability . The eRegistries technical team developed this set of indicators to provide guidance on core indicators that should be part of routine information systems to ensure that a comparable, consistent and comprehensive set of indicators is available to countries as they attempt to link their new electronic data collection systems with the older household survey-based data sources. Importantly, any implementation of an eRegistry must adhere to national guidelines, and include careful development of customised indicators and associated data points. This set of indicators should serve as a warehouse or library of indicators for use whenever possible to ensure comparability with WHO standards for care. These indicators should not be used en masse to monitor all levels of a health system equally. Different indicators will have different uses as well as different characteristics, dependent on the size of the population which is being monitored and the type of decisions needed at various levels of the health system. As evidenced from the graphic display of the association between power, sample size and indicator prevalence, the selected indicators can vary dramatically in usefulness for the different levels of the health system. Indicators relevant for individual clinics will typically relate to those activities being undertaken at the clinic for all women. Small sample sizes related to rare complications and outcomes will make some indicators less useful for understanding practices within specific individual clinics, but collection of these data is nonetheless highly relevant for monitoring of public health at the national-level.
With many countries in transition from paper to electronic data collection, more effort needs to be made to collect useable data at the point of data creation, and to minimise issues with recall, transcription and bias. The development of this suite of indicators can be used to support the implementation of eRegistries and other data platforms, to ensure that data are utilised to support evidence-based practice, facilitate measurement and accountability, and improve maternal and child health outcomes.
Demographic and health surveys
External cephalic version
- ICD PM:
International classification of diseases perinatal mortality
International stillbirth alliance
Low- and middle-income countries
Multiple indicator cluster survey
Prelabour rupture of membranes
Reproductive, maternal, newborn and child health
Sustainable development goals
Termination of pregnancy
Universal health care
Joint United Nations, Programme on HIV/AIDS
The United Nations Children's Fund
World Health Organisation
World health statistics
Haazen DS, Slote A, Shorbaji A, D’Adamo M. eHealth Technical Paper: Measurement and accountability for results in health: A common agenda for the post-2015 era; 2015. http://www.healthdatacollaborative.org/resources/.
United Nations. Transforming our world: The 2030 agenda for sustainable development. Geneva: United Nations, Department of Economic and Social Affairs; 2015.
World Bank, WHO, USAID. Health measurement and accountability post 2015: Five-point Call to Action: Measurement and accountability for results in health: A common agenda for the post-2015 era; 2015. http://www.healthdatacollaborative.org/resources/.
Frøen JF, Myhre SL, Frost MJ, Chou D, Mehl G, Say L, et al. eRegistries: Electronic registries for maternal and child health. BMC Pregnancy Childbirth. 2016;16(1):1–15.
Myhre SL, Kaye J, Bygrave LA, Aanestad M, Ghanem B, Mechael P, et al. eRegistries: governance for electronic maternal and child health registries. BMC Preg Childbirth. 2016;16(1):1–11.
Bryce J, Arnold F, Blanc A, Hancioglu A, Newby H, Requejo J, et al. Measuring coverage in MNCH: new findings, new strategies, and recommendations for action. PLoS Med. 2013;10(5):e1001423.
Hancioglu A, Arnold F. Measuring coverage in MNCH: Tracking progress in health for women and children using DHS and MICS household surveys. PLoS Med. 2013;10(5):e1001391.
Requejo JH, Newby H, Bryce J. Measuring coverage in MNCH: Challenges and opportunities in the selection of coverage indicators for global monitoring. PLoS Med. 2013;10(5):e1001416.
Ndabarora E, Chipps JA, Uys L. Systematic review of health data quality management and best practices at community and district levels in LMIC. Inf Dev. 2014;30(2):103–20.
World Health Organisation. Monitoring maternal, newborn and child health: understanding key progress indicators. Geneva: Health Metrics Network and World Health Organisation; 2011.
Donabedian A. The quality of care: How can it be assessed? JAMA. 1988;260(12):1743–8.
World Health Organization. Monitoring, evaluation and review of national health strategies: a country-led platform for information and accountability. Geneva: World Health Organization; 2011.
Tanahashi T. Health service coverage and its evaluation. Bull World Health Organ. 1978;56(2):295–303.
UNICEF. Reaching Universal Health Coverage through District Health System Strengthening: Using a modified Tanahashi model sub-nationally to attain equitable and effective coverage. New York: United Nations Children’s Fund Knowledge Management and Implementation Research Unit, Health Section, Program Division; 2013.
PMNCH. A global review of key interventions related to reproductive, maternal, newborn and child health (RMNCH). Geneva: The partnership for maternal, newborn and child health; 2011.
Measure Evaluation USAID. Profiles of health facility assessment methods. Arlington: Health Facility Assessment Technical Working Group (HFA TWG); 2005.
Nickerson JW, Adams O, Attaran A, Hatcher-Roberts J, Tugwell P. Monitoring the ability to deliver care in low- and middle-income countries: a systematic review of health facility assessment tools. Health Policy Plan. 2015;30(5):675–86.
The New Economics Foundation. Prove it! Measuring the effect of neighbourhood renewal on local people. Birmingham: Groundwork and The New Economics Foundation; 2000.
UNAIDS. Monitoring and Evaluation Fundamentals: An Introduction to Indicators. Geneva: UNAIDS; 2010.
Agency for Healthcare Research and Quality. Refinement of the HCUP Quality Indicators. Rockville: Agency for Healthcare Research and Quality; 2001.
World Health Organisation. Indicators to monitor maternal health goals: report of a technical working group Geneva, 8–12 November 1993. Geneva: World Health Organisation; 1994.
Performance Indicators in Community Health Project Working Group. Performance Indicators in Community Health: Development of a Process. Adelaide: Department of Human Services and Community Health Services South Australia; 2002.
Allanson ER, Tunçalp Ӧ, Gardosi J, Pattinson RC, Vogel JP, Erwich JJHM, Flenady VJ, Frøen JF, Neilson J, Quach A, Francis A, Chou D, Mathai M, Say L, Gülmezoglu AM. Giving a voice to millions: developing the WHO application of ICD-10 to deaths during the perinatal period: ICD-PM. BJOG. 2016. doi:10.1111/1471-0528.14243.
Tunçalp Ӧ, Were WM, MacLennan C, Oladapo OT, Gülmezoglu AM, Bahl R, et al. Quality of care for pregnant women and newborns—the WHO vision. BJOG. 2015;122(8):1045–9.
World Health Organization. Indicator and monitoring framework for the global strategy for women’s, children’s and adolescents’ health (2016–2030). Geneva: World Health Organization; 2016.
Every Woman Every Child. The global strategy for women's, children's, and adolescents' health (2016–2030). New York: Every Woman Every Child; 2015.
Freeman T. Using performance indicators to improve health care quality in the public sector: a review of the literature. Health Serv Manage Res. 2002;15(2):126–37.
World Health Organisation. Indicator Code Book: World Health Statistics - World Health Statistics indicators. Geneva: World Health Organisation; 2011.
World Health Organization. Evaluating the quality of care for severe pregnancy complications: The WHO near-miss approach for maternal health. Geneva: World Health Organization; 2011.
World Health Organization, United Nations Population Fund, UNICEF, Mailman School of Public Health. Averting Maternal Death and Disability. Monitoring emergency obstetric care: a handbook. Geneva: World Health Organization; 2009.
Inter-agency Working Group on Reproductive Health in Crises: Inter-agency Field Manual on Reproductive Health in Humanitarian Settings - 2010 Revision for Field Review. Geneva: Inter-agency Working Group on Reproductive Health in Crises; 2010.
World Health Organization and UNICEF. Countdown to 2015 - Maternal, Newborn and Child Survival: Building a Future for Women and Children. Geneva: World Health Organization and UNICEF; 2012.
We thank the eRegistries indicators expert panel for participating in indicators scoring and feedback: Zulfiqar Bhutta; Eckhart Buchmann; Sheila Clow; Ruth Davidge; Declan Devane; David Ellwood; Jan Jaap Erwich; Adrienne Gordon; Gill Gyte; Jane Hirst; Alexander Heazell; Jacqueline Ho; Dell Horey; Rachel Koshy; Carin Maree; Jiji Matthews; Elizabeth McClure; Philippa Middleton; Coceka Mnyani; Shadi Mphahlele; Stephen Munjanja; Linda Murray; Victoria Nankabirwa; Harriet Nabudere; Thanh Ha Nguyen; Robert Pattinson; Ranjan Kumar Pejaver; Haroon Saloojee; Thach Tran; David Tudehope; Mariet van Diem; and Xiaoying Zheng.
We thank Liam Flenady for technical support.
The Norwegian Agency for Development Cooperation (Norad) funded the harmonized Reproductive Health Registries project (GLO-4279 QZA 12/0355 harmonized Reproductive Health Registries) that led to the eRegistries Initiative (QZA-14/0022 Every Mother and Child Counts). This was led in partnership by the Norwegian Institute of Public Health and the WHO Department for Reproductive Health and the Mater Research Institute, University of Queensland (Australia), the University of Oxford (UK), and the Health Information Systems Program (Vietnam).
The contribution by JFF and JVJ was supported in part by the Centre for Intervention Science in Maternal and Child Health (CISMAC; project number 223269), which is funded by the Research Council of Norway through its Centers of Excellence scheme and the University of Bergen, Norway.
Availability of data and materials
VF, AW, VN, IF, DC and JFF collaboratively developed the project plan. IKF, IF, JVJ and JFF undertook the review of current use of indicators. DC, LS and GM provided advice on the design and conduct of the project from the global and WHO perspective. PM, RP, CC, JM, DE, DT, JH and ABO provided technical and clinical advice on indicators with the technical team that refined the final indicators (VF, AW, IF, JVJ and JFF). VF, AW, PM, CC, and JFF developed the eRegistries indicator evaluation tool. AW managed the compilation and development of indicators and scoring by the expert panel. IKF assisted in literature reviews. MV developed the power calculations relating to indicator use. All authors assisted with preparation of the manuscript and read and approved the final version for publication.
The authors declare that they have no conflicts of interest. The views expressed in this article do not necessarily represent the decisions, policy, or views of the authors’ affiliations.
Consent for publication
Ethics approval and consent to participate
The Mater Health Services Human Research Ethics Committee assessed this research as meeting the conditions for exemption from HREC review and approval in accordance with section 5.1.22 of the National Statement on Ethical Conduct in Human Research (2007).
Example score sheet with indicator criteria. (DOCX 329 kb)
Evaluation rounds, interventions addressed with number of indicators and indicator sources. (DOCX 44.4 kb)
Response to feedback – examples of indicator amendments. (DOCX 41.5 kb)
List of final indicators. (DOCX 45 kb)
Glossary of denominator units and definitions. (DOCX 37.9 kb)
About this article
Cite this article
Flenady, V., Wojcieszek, A.M., Fjeldheim, I. et al. eRegistries: indicators for the WHO Essential Interventions for reproductive, maternal, newborn and child health. BMC Pregnancy Childbirth 16, 293 (2016). https://doi.org/10.1186/s12884-016-1049-y