An evaluation of the quality, suitability and impact on equity of clinical practice guidelines relevant to preterm birth for use in Aotearoa New Zealand

Abstract

Background

Preterm birth is a leading cause of perinatal morbidity and mortality and a defining event for pregnant people, infants, and whānau (extended families). Recommendations have been made for a national preterm birth prevention initiative focusing on equity in Aotearoa New Zealand, including the development of a national best practice guide. An understanding of the number and quality of guidelines, and consideration of their suitability and impact on equity is required.

Methods

Guidelines were identified through a systematic literature search, search of professional bodies websites, and invitation to regional health services in Aotearoa New Zealand. Obstetric and midwifery clinical directors were invited to report on guideline use. Identified guidelines were appraised by a 23-member trans-disciplinary Review Panel; quantitatively using the AGREE-II instrument and qualitatively using modified ADAPTE questions. The quality of guidelines available but not in use was compared against those in current use, and by health services by level of maternity and neonatal care. Major themes affecting implementation and impact on equity were identified using Braun and Clarke methodology.

Results

A total of 235 guidelines were included for appraisal. Guidelines available but not in use by regional health services scored higher in quality than guidelines in current use (median domain score Rigour and Development 47.5 versus 18.8, p < 0.001, median domain score Overall Assessment 62.5 versus 44.4, p < 0.001). Guidelines in use by regional health services with tertiary maternity and neonatal services had higher median AGREE II scores in several domains, than those with secondary level services (median domain score Overall Assessment 50.0 versus 37.5, p < 0.001). Groups identified by the Review Panel as experiencing the greatest constraints and limitations to guideline implementation were rural, provincial, low socioeconomic, Māori, and Pacific populations. Identified themes to improve equity included a targeted approach to groups experiencing the least advantage; a culturally considered approach; nationally consistent guidance; and improved funding to support implementation of guideline recommendations.

Conclusions

We have systematically identified and assessed guidelines on preterm birth. High-quality guidelines will inform a national best practice guide for use in Taonga Tuku Iho, a knowledge translation project for equity in preterm birth care and outcomes in Aotearoa.

Background

Preterm birth is a leading cause of perinatal morbidity and mortality and may lead to lifelong disability and poor health [1]. The risk of preterm birth varies both between and within populations, often due to differences in individuals’ risk status, but also by factors such as ethnic and cultural background, place of residence, and socioeconomic status [2,3,4,5,6]. This is clearly seen in Aotearoa New Zealand with a lower rate of preterm birth, as well as higher rates of survival following extreme preterm birth, for European and Asian babies than for their Māori (Aotearoa New Zealand Indigenous/First Nations), Pacific and Indian counterparts [3, 7]. Improving quality and consistency of care with equity as a focus should result in less variation in practice and lead to improved outcomes, especially for those currently experiencing the least advantage.

Clinical practice guidelines have the potential to improve practice and outcomes by summarising and synthesising complexities of evidence, thereby increasing speed and efficiency of clinical decision-making [8]. Furthermore, when applied consistently in practice, clinical practice guidelines should support improved equity in care and outcomes. In order to achieve this, they should contain and/or consider a systematic review of all relevant evidence and be developed in partnership with key stakeholders including the populations most affected by the clinical issue [9, 10]. If equity is to be effectively addressed it must be considered from the outset of a clinical practice guideline’s development, and factored into every step including through implementation and evaluation. Without this consideration, there is a risk that there will be no reduction in differences seen by equity factors, or even worsening of equity issues by only improving care and outcomes for those already experiencing system and other advantage [11].

Improving care and outcomes for preterm birth poses a challenge due to the diverse number and type of conditions contributing to both spontaneous and provider-initiated preterm birth. Clinical practice guidance must consider this broad range of conditions and include prediction and prevention of preterm birth, as well as, preparation and management when preterm birth is inevitable or expected.

In Aotearoa New Zealand, preterm birth and equity of access to care have been identified as priority areas for maternity care practice improvement [3, 12,13,14,15]. A stakeholder-led initiative, the Carosika Collaborative, has been funded through the New Zealand Health Research Council to undertake a knowledge translation project for equity in preterm birth care and outcomes in Aotearoa. This project, named Taonga Tuku Iho (Te Reo Māori referring to the legacy of preterm babies) includes the development of a national best practice guide to support consistent and equitable practice. This best practice guide will be developed using the ADAPTE framework; adopting, contextualising and adapting recommendations across all aspects of preterm birth care, using existing high-quality clinical practice guidelines relevant to Aotearoa New Zealand, where available, and formulating new recommendations where no high quality guidelines exist, to provide a single point of reference for national use. This requires an understanding of the quality, suitability and impact on equity of the guidelines that are currently in use or are appropriate for use in Aotearoa New Zealand.

Methods

This study aims to systematically appraise the quality of clinical practice guidelines relating to preterm birth and to assess their perceived impact on equity and suitability for use in the Aotearoa New Zealand context.

Guideline selection

Clinical practice guidelines were identified and considered for inclusion from the published literature, relevant professional bodies, and the previously named Aotearoa New Zealand District Health Boards. District health boards were the name given to the regional health services in Aotearoa New Zealand up until July 2022. Before they were disestablished, there were 20 district health boards across the country.

Systematic literature search

A search strategy was developed and refined (Appendix A). Searches were undertaken of EMBASE, MEDLINE and CINAHL databases (Appendix B). Titles and abstracts were screened by two independent reviewers using the Covidence systematic review reference management tool [16], then full-text articles of references thought to be potentially relevant were retrieved and reviewed. A third reviewer was available if consensus was not able to be reached after initial discussion of any discrepancies.

Professional bodies search

Relevant professional bodies, colleges and societies were identified by the investigator team alongside an internet search. Professional bodies were included if they were involved with any aspect of preterm birth care or the management of its major risk factors. Included professional bodies were Aotearoa New Zealand specific, Australasian or international with relevance to countries with similar resources to Aotearoa New Zealand. A manual search of the websites of each professional body was undertaken by a single investigator to retrieve potentially eligible guidelines. Two reviewers then independently assessed the full text guidelines for inclusion, with a third reviewer available if consensus was not reached after initial discussion of any discrepancies.

District health board search

Invitations were sent to the obstetric and midwifery clinical directors at each of Aotearoa New Zealand’s 20 district health boards (assigned regional health services until 2022). Clinical directors were contacted by email and invited to complete a questionnaire about guidelines in use at their institution and requested to contribute these guidelines to the study. If a local guideline was explicitly reported as being based on a national or international guideline then the national/international guideline was also considered to be in use. Written informed consent was obtained from all participants. Non-responders were contacted via email two weeks later and any ongoing non-responders were contacted once by telephone a further two weeks later. Ethical approval was obtained.

Inclusion criteria

Guidelines could be regional, national, bi-national (Australasian), or international (defined as those developed for use in more than one country and with similar resources and health context to Aotearoa New Zealand). Guidelines were included if they contained recommendations on prediction or prevention of preterm birth, preparation for preterm birth, or the acute management of preterm labour. Guidelines on both spontaneous and provider-initiated preterm birth were included, as well as the conditions and major risk factors that contribute to preterm birth. Guidelines were in English and must have been published, reviewed or updated within the last ten years. If multiple versions of a guideline existed, only the latest version was included. Where further guideline documents were referenced, these were retrieved and assessed alongside their relevant guideline.

Appraisal

Descriptive data were collected for each guideline by a single investigator. Guideline appraisals were undertaken by 23 members of the transdisciplinary Taonga Tuku Iho Review Panel including midwives, obstetricians, neonatologists, radiologists, consumers, Māori, Pacific and Indian representatives, and individuals with experience in clinical practice guideline development. Review Panel members were invited as representatives of a professional college, society or group including consumer groups and national guideline groups or due to expertise in specific areas such as preterm birth and Māori health. All members provided a disclosure of interest declaration with information on their professional affiliations and previous involvement in guideline development. An on-line meeting provided background information and a briefing on the appraisal process, including provision of the Review Panel Terms of Reference, ADAPTE Toolkit [17], AGREE Trust training exercises [18], AGREE II Instrument User’s Manual [19] and an AGREE II tutorial.

Each guideline was allocated to four reviewers representing a variety of stakeholder groups, with two reviewers (BH and MW) undertaking reviews on every guideline. Guidelines were not allocated to reviewers employed or associated with the organisation responsible for the development of a guideline or involved in its development.

Assessment of quality was made using the AGREE II tool [19] via the online My AGREE Plus appraisal platform [20]. This tool was chosen as it is suggested to be the most commonly applied and comprehensively validated guideline appraisal tool worldwide, and the accepted standard for evaluating clinical practice guidelines [21,22,23,24]. The AGREE II checklist includes 23 key items organised into six domains. Each guideline was scored by domain, with an overall assessment score and recommendation for use in clinical practice.

Assessment of applicability and/or acceptability for use in Aotearoa New Zealand was undertaken using a modification of Tool 15 from the ADAPTE Resource Toolkit [17] answered on a secure google form. This tool was chosen as it is specifically designed for evaluating guidelines, as well as being in keeping with use of the ADAPTE framework for the overall development of the national best practice guide. It was modified to allow a broader exploration of perceived impact on equity.

Outcomes

The primary outcome was the overall number and quality of guidelines relevant to preterm birth in Aotearoa New Zealand; firstly those in current use and and secondly those available but not in use.

Secondary outcomes

• The proportion of guidelines in use in Aotearoa New Zealand, that are considered to be high-quality.

• The proportion of guidelines that are available, but not currently in use in Aotearoa New Zealand, that are considered to be high quality.

• The quality of guidelines used by district health boards with secondary compared with tertiary (highest level, subspecialist) maternity and neonatal services. (In Aotearoa New Zealand there are three levels of maternity and neonatal care; primary: >37 weeks, no identified pregnancy complications, secondary: caring for babies born from 32 weeks, tertiary: highest level sub-specialist care, caring for babies born from peri-viable gestations.)

• The perceived impact on equity and suitability of guidelines for the Aotearoa New Zealand context.

Predefined criteria for guidelines to be considered ‘high quality’ were any of:

• Score of > 60% in AGREE II Domain 3 Rigour of Development

• Score of > 60% in AGREE II overall assessment score

• > 50% of appraisers recommend them for use in clinical practice or use with modifications.

Analysis

Descriptive data were reported for all guidelines. AGREE II scores were compared by district health board use and level of maternity and neonatal services, using the Mann-Whitney U test. The frequency of high-quality guidelines was compared using the Chi-squared test. Descriptive statistics were calculated using Microsoft Excel (version 16.59) and SPSS (version 28.0.1.1) [14]. A p value less than 0.05 was considered significant. Variance between appraisers was calculated by determining the intraclass correlation coefficient for consistency. Thematic analysis of the ADAPTE questions was undertaken using Braun and Clarke methodology [25] by a single investigator, with a group conference involving two other investigators to review and refine themes.

Ethics statement

Written informed consent was obtained from all participants. The study was approved by the University of Auckland Health Research Ethics Committee on 27/09/2021, reference number AH23265.

Results

A total of 235 eligible clinical practice guidelines were evaluated (Fig. 1). Guidelines were shared by 14/20 (70.0%) of Aotearoa New Zealand district health boards, with 13/20 (65.0%) completing the questionnaire on guideline use. Guideline descriptions and their method of identification are reported in Table 1. There were low rates of reporting of funding source (78/260, 30.0%), consumer involvement (63/260, 24.2%), Indigenous involvement in guideline development (11/260, 4.2%) and use of an equity tool (11/260, 4.2%).

Fig. 1

Identified clinical practice guidelines. CINAHL, Cumulative Index to Nursing and Allied Health Literature; RANZCOG, Royal Australian and New Zealand College of Obstetricians and Gynaecologists; NZCOM, New Zealand College of Midwives; NZMFMN, New Zealand Maternal Fetal Medicine Network; MOH, Ministry of Health; WHO, World Health Organization

Table 1 Description of included guidelines

Of the 235 identified guidelines, 151 were reported to be currently in use by district health boards. Another 84 were available but not currently in use in Aotearoa New Zealand. The majority of these were national or international guidelines not identified as being utilised by district health board clinical directors.

The AGREE II quality scores are reported in Table 2. A greater proportion of guidelines that were reported to be available but not in use met the pre-defined criteria for high-quality across every category, than those currently in use (Table 2).

Table 2 Quality of guidelines by district health board use

Guidelines reported to be in use by district health boards with tertiary maternity and neonatal services, had higher median scores in the domains Rigour of Development, Clarity of Presentation and Applicability, as well as in Overall Assessment Score, compared with guidelines reported to be in use by district health boards with secondary services (Table 3). There was a greater proportion of guidelines in use by district health boards providing tertiary services that met the high-quality criteria for Overall Assessment Score and recommendation for use in clinical practice or use with modifications compared with district health boards providing secondary services (Table 3). Consistency in AGREE II scores between appraisers was moderate or good for (153/235) 65.1% of guidelines and poor for the remainder.

Table 3 Quality of guidelines by type of district health board

Responses to questions regarding equity and implementation constraints are shown in Table 4. Identified constraints and limitation themes included; cultural considerations in care; service capacity, access to tests, interventions and specialist services; funding to support implementation of recommendations, and staff shortages across the maternity sector (Table 5). Groups identified as experiencing the greatest constraints and limitations to guideline implementation were rural, provincial, low socioeconomic, Māori, and Pacific populations. Major themes of factors that had negative or positive impact on equity were; nationally consistent guidance; culturally considered approach and feasibility of guideline implementation (Table 5). Identified themes to improve equity included a targeted approach to groups experiencing the least advantage; a culturally considered approach; nationally consistent guidance; and improved funding to support implementation of guideline recommendations (Table 5).

Table 4 Responses to modified ADAPTE questions

Table 5 Themes identified from modified ADAPTE questions

Discussion

Discussion of important findings

Guidelines that were available but not currently in use by district health boards scored higher in quality than guidelines reported to be in current use. This is a potential major concern but may be justified in some instances as guidelines that were available but not in use did not always cover the same topics as the guidelines in use by district health boards. Furthermore, some guidelines with higher quality scores were from international organisations, such as the World Health Organization, and may have been judged by clinicians as less applicable to the local Aotearoa New Zealand clinical environment. However, similar results were found in an appraisal of clinical practice guidelines on reduced fetal movements in the United Kingdom; with consistently lower AGREE II scores in local maternity unit guidelines compared with established national guidelines [26]. Overall this makes a compelling argument for a high-quality national best practice guide that is prepared for country-specific context allowing easy adoption and implementation by all local health services.

Guidelines reported to be in use by district health boards with tertiary maternity and neonatal services had significantly higher median AGREE II scores in several domains, than those with secondary level services. Again, this is a potential major concern and may contribute to differences seen in care and outcomes by equity factors, in particular by region of residence [27]. Similar findings have been reported in an appraisal of Australasian antenatal corticosteroid guidelines, where institutions providing secondary neonatal services had lower scoring guidelines, than institutions providing tertiary services [28]. Our Review Panel identified that rural and provincial areas have greater constraints and limitations to implementing guideline recommendations. Again this strongly supports the development of a high-quality national best practice guide that is suitable for use for all.

Despite 61.5% of clinical practice guidelines being developed in Aotearoa New Zealand, and 80.0% being developed in either Aotearoa New Zealand or Australasia, Indigenous involvement in guideline development was extremely low (reported in 4.2%). This low rate of inclusion of Māori as tangata whenua (people of the land), combined with a low rate of reporting equity tool use during guideline development (4.2%), is in contrast with the established roles of district health boards and Aotearoa New Zealand professional bodies to uphold the principles of Te Tiriti o Waitangi, a founding document of Aotearoa New Zealand [29]. Having a culturally considered approach to guideline development and implementation was identified by our Review Panel as a key factor to support the ability for guideline recommendations to reduce differences by equity factors. This key factor has been noted in previous studies incorporating equity in guideline implementation [30]. Inclusion of members of the most impacted populations and use of an equity tool from the outset of guideline development have been identified as essential for guideline success in addressing equity issues [30], and have been studied specifically in the Aotearoa New Zealand context [31]. It is possible that broader Māori representation occurred than was reported and that some institutions may detail guideline development processes in supplementary documents that were not provided to our investigator team, however full disclosure of the guideline development processes and stakeholder involvement is a long-established recommendation for clinical practice guideline reporting [9, 19].

Further themes identified in the analysis of qualitative comments on enablers to improve equity were nationally consistent guidance, and improved funding to support implementation of guideline recommendations. The latter theme has been identified elsewhere as a major equity consideration, likely to be relevant worldwide [32].

Although this study was carried out in the Aotearoa New Zealand context, findings are likely to be relevant and of interest to other countries with similar health resources. This is particularly important given the apparent exponential growth in clinical guidelines in recent years and increasing awareness of consideration of equity in all areas of clinical care including the development and use of guidelines.

Strengths and limitations

This study included a systematic search covering prediction, prevention, preparation and management of preterm birth. The use of a multi-pronged search strategy allowed for identification of a broad range of guidelines, not limited to those formally published in peer-reviewed journals. Inclusion of regional clinical practice guidelines and a questionnaire on guideline utilisation by maternity care providers, has not been captured in previous appraisals of New Zealand’s maternity sector guidelines [33]. District health board engagement was good, increasing the generalisability of the findings.

The AGREE II instrument was chosen as a systematic, validated, and globally-recognised clinical practice guideline appraisal tool, which is appropriate for use across a wide range of guideline subject matters [21,22,23,24]. The use of the online My AGREE Plus platform made for convenient coordination of appraisals and straightforward collaboration of a large panel of reviewers. While the platform and built-in user’s manual made the appraisals easy to complete, the platform was limited in its ability to report results; generating individual data files for each guideline. For a project of this size, this resulted in a large amount of manual processing of results, introducing the potential for human error in transcribing data.

A limitation of AGREE II is the lack of established threshold criteria for quality, which is left to individual research groups to determine based on their research question. Domain 3 ‘Rigour of Development’ was chosen as a priority area for this study, as we were interested in identifying guidelines that had methodological strength and a firm evidence basis that could be trusted for use in developing a national best practice guide. ‘Overall Assessment Score’ was chosen as another quality criteria to provide a global assessment score of the guideline. Guideline recommendation for clinical use provided a pragmatic indicator of value in clinical practice, for guidelines that may have practical value that is not necessarily reflected in their AGREE II domain ratings.

A further major limitation of the AGREE II tool is the lack of comment or consideration of equity in guideline implementation, for which it has received criticism previously [10, 34]. For this reason, we included additional questions modified from the ADAPTE Handbook, specifically to look at guideline applicability, acceptability, and impact on equity [17]. This enabled identification of the major themes of constraints and limitations to guideline implementation and identification of guideline factors that impact on equity. Including a qualitative appraisal component allowed the wealth and breadth of experience of our Review Panel to be appreciated and added a depth to the assessment that would not have been possible from quantitative appraisal alone. These insights were invaluable in identifying enablers and barriers for developing a national best practice guide on this topic.

All appraisers underwent training on the AGREE II tool, however despite this, there was significant variation seen between the scores given by different appraisers, with 35% of guidelines having low consistency in the scores given by different appraisers. This is in part a reflection of the subjective nature of AGREE II scoring, with no definitive guidance provided informing the score that should be given when different criteria are met. Variation likely also arose due to the wide range of backgrounds of the Review Panel and the large number of appraisers, although this broad representation is also a strength of the appraisal.

Conclusions

This was a comprehensive review of clinical practice guidelines on preterm birth available for use and in use in the Aotearoa New Zealand public health system. The mixed methods appraisal had strengths in identifying guidelines of high methodological rigour, as well as potential constraints and limitations to guideline implementation. Major factors contributing to impact on equity were identified; providing invaluable insight for developing a national best practice guide on this topic to be used across Aotearoa New Zealand.

Development of a national best practice guide for equity in preterm birth care is now underway, in partnership with the groups most impacted in Aotearoa New Zealand; with a culturally considered approach and nationally consistent, feasible recommendations. Future research is planned to assess the impact of this best practice guide, through national reporting of preterm birth core outcomes informed by perinatal healthcare professionals and consumers.