Article Text

Download PDFPDF

Review
A systematic review and appraisal of the quality of practice guidelines for the management of Neisseria gonorrhoeae infections
  1. Catherine Dickson1,
  2. Trevor Arnason1,
  3. Dara Spatz Friedman2,
  4. Gila Metz2,3,
  5. Jeremy M Grimshaw4,5
  1. 1 School of Epidemiology, Public Health and Preventive Medicine, University of Ottawa, Ottawa, Ontario, Canada
  2. 2 Ottawa Public Health, Ottawa, Canada
  3. 3 Department of Family Medicine, University of Ottawa, Ottawa, Canada
  4. 4 Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, Canada
  5. 5 Department of Medicine, University of Ottawa, Ottawa, Canada
  1. Correspondence to Dr Catherine Dickson, School of Epidemiology, Public Health and Preventive Medicine, University of Ottawa, Alta Vista Campus, Room 101, 600 Peter Morand Crescent, Ottawa, Ontario, Canada K1G 5Z3; Catherinedickson01{at}gmail.com

Abstract

Background Clinical guidelines help ensure consistent care informed by current evidence. As shifts in antimicrobial resistance continue to influence first-line treatment, up-to-date guidelines are important for preventing treatment failure. A guideline's development process will influence its recommendations and users' trust.

Objective To assess the quality of current gonorrhoea guidelines' development processes.

Data sources Multiple databases.

Study eligibility criteria Original and current English-language guidelines targeting health professionals and containing treatment recommendations for uncomplicated gonorrhoea in the general adult population.

Study appraisal and synthesis methods Two appraisers assessed the guidelines independently using the Appraisal of Guidelines for Research and Evaluation II (AGREE II) tool. Scores were combined as per the AGREE II users' manual.

Results We identified 10 guidelines meeting the inclusion criteria. The quality of the gonorrhoea treatment guidelines varied. Most scored poorly on Rigour of Development; information on the evidence review process and methods for formulating recommendations was often missing. The WHO Guidelines for the Treatment of Neisseria gonorrhoeae and UK National Guideline for the Management of Gonorrhoea in Adults scored the highest on Rigour of Development. Methods to address conflicts of interest were often not described in the materials reviewed. Implementation of recommendations was often not addressed.

Limitations By limiting our study to English-language guidelines, a small number of guidelines we identified were excluded. Our analysis was limited to either published or online materials that were readily available to users. We could not differentiate between items addressed in the development process but not documented from items that were not addressed.

Conclusions and implications of key findings Gonorrhoea treatment guidelines may slow antimicrobial resistance. Many current guidelines are not in line with the current guideline development best practices; this might undermine the perceived trustworthiness of guidelines. By identifying current limitations, this study can help improve the quality of future guidelines.

  • GONORRHOEA
  • ANTIBIOTIC RESISTANCE
  • ANTIBIOTICS
  • SYSTEMATIC REVIEWS
  • GUIDELINE DEVELOPMENT

Statistics from Altmetric.com

Background

Clinical guidelines are a tool for ensuring that medical care is consistent between providers and in line with current evidence.1 However, guidelines are only as good as their development process. Guideline quality, or ‘the confidence that the biases linked to the rigour of development, presentation and applicability of a guideline had been minimized during the development process’1 is important to ensure that optimal recommendations are made.

Neisseria gonorrhoeae, the bacteria responsible for gonorrhoea infections, has developed and maintained resistance to nearly every antibiotic that has been used to treat it,2–4 presenting a particular challenge for guideline developers. Clinicians need up-to-date guidelines based on current evidence in combination with knowledge of local resistance pattern epidemiology. This will delay development of resistance to current treatment regimens and prevent sequelae of undertreated infections.

Numerous gonorrhoea guidelines exist across different jurisdictions to guide appropriate antibiotic and dosage selection. In addition to improving individual patient outcomes, high-quality gonorrhoea treatment guidelines would improve public health outcomes.5–7 Appropriate treatment of gonorrhoea can prevent treatment failure and subsequent reinfection of partners and potentially help to delay further progression of antimicrobial drug resistance.8

We conducted a systematic review of current clinical guidelines for the treatment of gonorrhoea and reviewed their quality using the Appraisal of Guidelines for Research and Evaluation II (AGREE II), a critical appraisal tool designed to assess the quality of guidelines.7

Methods

Guideline selection

We developed a search strategy with the assistance of a medical librarian. Relevant guidelines were identified using multiple databases: Medline, PubMed, Embase, National Guideline Clearinghouse, Guideline International Network, Canadian Medical Association's CPG Infobase: Clinical Practice Guidelines Database, Trip Database, National Institute of Health and Care Excellence, Scottish Intercollegiate Guidelines Network, Australian Government National Health and Medical Research Council Clinical Practice Guidelines Portal, Google (limited to the first five pages) and Google Scholar (limited to the first five pages). In addition, individual guideline documents were reviewed for references to other relevant guidelines. Search terms included: gonorrhoea, Neisseria gonorrhoeae, gonorrh*, gonococc*, guidelines, guideline, practice guidelines, practice guideline, health planning guidelines, clinical protocols, clinical pathways, healthcare planning, gonorrhoea treatment guidelines. A detailed description of each search is found in table S1 of the online supplementary appendix.

One author (CD) conducted the original searches between 11 and 18 August 2014. The same searches were repeated (with date limits of 1 August 2014 to present) on 6 and 7 February 2017 to identify the new guidelines released since the previous search. Guidelines were selected for inclusion using predetermined inclusion criteria:

  • English language

  • Guidelines for the treatment of gonorrhoea in the general adult population

  • Guidelines including recommendations on the treatment of uncomplicated infections (ie, urogenital, rectal and pharyngeal infections without complications such as pelvic inflammatory disease or epididymitis)—including antibiotic name and dose

  • Guidelines targeting an audience of health professionals

  • Original guideline document (ie, not based on another guideline)

  • Most recent version of the guideline available at the time.

We identified accompanying documents, such as methods papers, that were either published or available online during our searches and through scanning the guidelines for references to such documents. Accompanying documents, if available, were reviewed along with each guideline.

When the guidelines covered multiple infections, the appraisers reviewed the full documents but focused on the sections and recommendations for gonorrhoea.

In January 2017, CD verified each of the previously identified guidelines to ensure that the most recent version of the guideline was appraised.

Data extraction

One appraiser (CD) reviewed each guideline to document: the guideline development agency, publication year, country or jurisdiction to which the guideline applied and the recommended treatment for uncomplicated gonorrhoea.

Guideline appraisal

Two appraisers (CD and TA) assessed each guideline independently using the AGREE II critical appraisal instrument.7 AGREE II appraises a guideline's development process and the reporting of the process. The instrument assesses 23 items in six domains (Scope and Purpose, Stakeholder Involvement, Rigour of Development, Clarity of Presentation, Applicability and Editorial Independence). Each item is scored using a scale of 1 (strongly disagree) to 7 (strongly agree). In addition, the appraisers rated the overall quality of the guideline from 1 (lowest possible quality) to 7 (highest possible quality).7

Prior to reviewing the guidelines, both appraisers reviewed the AGREE II User's Manual.7 In addition, CD completed the online training module on the AGREE II instrument available on the AGREE Enterprise website.9 The appraisers used the instrument to review an unrelated guideline (Guidelines for the Prevention and Control of Measles Outbreaks in Canada)10 together to verify that they had a common understanding of the items and domains.

The appraisers discussed any scores that had a discrepancy of 3 points or more out of 7 as we felt that such a discrepancy could be a sign that an appraiser might have missed an important piece of information. During these discussions, the appraisers revisited their common understanding of the items within each domain. Appraisers were not required to reach consensus; however, each was allowed to alter his or her score if desired.

Calculating domain scores

We combined the appraisers' scores using the instructions in the AGREE II Users' Manual. A scaled domain score was calculated for each domain by summing the appraisers' scores for all items in the domain and expressing it as a percentage of the maximum possible score as follows:7 Embedded Image

Inter-rater reliability

Inter-rater reliability refers to the level of agreement between appraisers and is used to assess the internal validity of measures.11 We measured inter-rater reliability using Cohen's weighted κ12 We calculated inter-rater reliability for each domain score as well as overall guideline quality. We used Landis and Koch's guidelines to interpret κ values: 0.0–0.2 indicate slight agreement, 0.21–0.40 indicate fair agreement, 0.41–0.60 indicate moderate agreement, 0.61–0.80 indicate substantial agreement and 0.81–1.0 indicate almost perfect or perfect agreement.11 Inter-rater reliabilities were assessed both before and after the appraisers reviewed their scoring.

Statistical software

Domain scores were calculated using Microsoft Excel. Inter-rater reliability was determined using STATA V.13.

Results

Ten guidelines published between 2011 and 2016 met our inclusion criteria (table 1). Two guidelines that had been withdrawn by their developers between August 2014 and January 2017 were excluded from our review.13 ,14 Most guidelines recommend intramuscular ceftriaxone (the North American and WHO guidelines recommend 250 mg,15–18 and the UK, European, New Zealand and Australian guidelines recommend 500 mg19–24) in combination with oral azithromycin. The European guidelines19 recommend 2 g of azithromycin rather than the 1 g recommended in the other guidelines. The chapter on gonorrhoea from the Canadian and WHO Guidelines, 2013 allows oral cefixime as a first-line alternative to ceftriaxone.17 ,18

Table 1

Summary of selected guidelines

The domain and overall scores for each guideline are summarised in table 2. Individual scores by item for each guideline are summarised in online supplementary table S2 of the appendix. Inter-rater reliability scores are summarised in table 3. Following appraisers' discussions, inter-rater reliability increased for each domain and overall from 0.49 (moderate agreement) to 0.69 (substantial agreement).11

Table 2

Summary table of scaled domain scores by guideline based on materials published or available online

Table 3

Inter-rater reliability before and following appraisers' discussions of scores with discrepancies

Domain 1: scope and purpose

This domain considers the clarity of the guideline's overall objectives, the specific clinical questions addressed by the guideline and the population to whom the guideline is meant to apply. The appraisers were looking for overall statements regarding the guideline's objectives and the intended population. Clinical questions could either be directly stated (eg, in the introduction) or identified throughout the guideline (eg, in section headings). Domain scores ranged from 50.0% (Western Australia Silver Book) to 94.4% (WHO). Most guidelines scored high in this domain.

Domain 2: stakeholder involvement

This domain considers whether all relevant professional groups were represented on the guideline development group, whether patients in the target group were consulted regarding options or preferences, and whether the target users of the guideline are clearly identified. The appraisers were looking for sufficient information to identify the different organisations and specialties represented. Professional groups would ideally include a range of organisations, multiple medical specialties involved in STI management and laboratory experts. Additionally, the appraisers were looking for any mention of patient consultation as well as sufficient information to understand who was consulted, how they were consulted and how representative they were of the target population. The appraisers were looking for the target guideline users to be specified. Scores ranged from 22.2% (Melbourne) to 86.1% (UK). Seven guidelines scored 50% or higher. Many guidelines scored poorly on the item ‘The views and preferences of the target population (patients, public, etc.) have been sought’.

Domain 3: rigour of development

This domain relates to the processes used for evidence collection, review and synthesis, as well as for developing recommendations and updating the guideline. The appraisers were looking for sufficient detail about the methodology to understand how the evidence was gathered, appraised and used, as well as a description of the process for future updates. Scores ranged from 5.2% (Melbourne and Western Australia) to 94.8% (UK). Six out of ten guidelines scored below 50%. Information was frequently lacking for: ‘A procedure for updating the guideline is provided’, ‘Systematic methods were used to search for evidence’, ‘The criteria for selecting the evidence are clearly described’, ‘The guideline has been externally reviewed by experts prior to its publication’, ‘The methods for formulating the recommendations are clearly described’ and ‘The strengths and limitations of the body of evidence are clearly described’.

Domain 4: clarity of presentation

This domain considers the clarity of recommendations' wording, the clarity of presentation of different management options and the ease of identifying key recommendations. The appraisers were looking for recommendations that were easy to understand, presented treatment options and made key information easy to find for a clinician who might be quickly scanning for guidance at point of care. Domain scores ranged from 52.7% (Western Australia) to 88.9% (UK).

Domain 5: applicability

This domain includes the items: ‘The guideline describes facilitators and barriers to its application’; ‘ The guideline provides advice and/or tools on how the recommendations can be put into practice’; ‘The potential resource implications of applying the recommendations have been considered’; and ‘The guideline presents monitoring and/or auditing criteria’. The appraisers were looking for discussions of facilitators, barriers and resource implications as well as ways to address them. The appraisers were also looking for materials to assist clinicians in adopting the guidelines such as wall charts, summary documents or patient information sheets, as well as the identification of criteria that could be used to assess guideline implementation or adherence to recommendations. Scores ranged from 4.2% (Melbourne) to 64.6% (UK and WHO). All but two guidelines scored below 50%. Most guidelines did not provide auditing or monitoring criteria, and did not address potential resource implications such as the need to administer antibiotics intramuscularly.

Domain 6: editorial independence

This domain considers the funding body's influence on the guideline's content and whether the potential conflicts of interest of individuals involved in the development of the guidelines have been recorded and addressed. Appraisers were looking for a statement that the funding body had not influenced the content of the guidelines. Appraisers were also looking for both the declaration of potential conflicts of interest and a description of how the potential conflicts of interest were addressed. Scores for this domain ranged from 0% (Melbourne, New Zealand and Western Australia) to 83.3% (UK). Only four guidelines scored above 50%. Several guidelines did not describe whether and how conflicts of interest of the guideline developers were recorded and addressed. Also, guidelines often did not state whether or not the funding body's views influenced the guideline's content.

Overall quality

Overall quality scores from the appraisers ranged from 2.5 (Melbourne) to 6 (UK and WHO) out of 7. Over the 5-year time frame during which the reviewed guidelines were released, we did not see a trend towards improvement in quality of guidelines over time. However, the most recently released WHO Guidelines for the Treatment of Neisseria gonorrhoeae was one of the highest scoring guidelines in our review; it scored over 50% for all domains and was ranked first in four domains (scope and purpose, stakeholder involvement, rigour of development, applicability), second in one domain (clarity of presentation) and third in the final domain (editorial independence).

Discussion

Using the AGREE II tool to assess the quality of current gonorrhoea treatment guidelines, we determined that quality varied considerably between guidelines. The WHO guidelines scored the highest overall, ranking first in four domains and either second or third in all others. The guidelines generally scored high on Scope and Purpose, and Clarity of Presentation. Stakeholder Involvement, Rigour of Development, Applicability and Editorial Independence scores varied. As Rigour of Development ultimately affects the scientific basis of the recommendations made, it can be considered as the most important domain when assessing the quality of a guideline. Key guideline development information, such as the evidence review process, methods used for formulating recommendations and potential conflicts of interest were often missing from the materials available. The 2012 European Guideline on the Diagnosis and Treatment of Gonorrhoea in Adults, the US Centers for Disease Prevention and Control's Sexually Transmitted Diseases Guidelines, the UK National Guideline for the Management of Gonorrhoea in Adults and the WHO Guidelines for the Treatment of Neisseria gonorrhoeae deserve recognition as the guidelines that scored above 50% in Rigour of Development. Most guidelines did not address potential practice or resource implications of the recent change in treatment recommendations to include an intramuscular as well as an oral antibiotic. This change in recommendations could potentially be a barrier for patients who may prefer an oral treatment as well as for care providers who would require additional time to reconstitute, administer and sometimes obtain the ceftriaxone.

Many guidelines did not address the item ‘The views and preferences of the target population (patients, public, etc.) have been sought’, potentially due to the challenges of identifying and reaching people who would be representative of the diverse target population. When this topic was addressed, it was done in a variety of ways. The UK guideline indicated that this was done, but gave no indication of who was consulted or how they were involved. The WHO guideline included studies on patient preferences as part of their review. The New Zealand guideline included a consultation with a women's health collective for a consumer's perspective; this perspective may have been quite different from that of other segments of the target population such as men who have sex with men.

In our review of the literature, we could not find any prior systematic reviews of guideline development quality for STI treatment guidelines. The strength of this review was that we used a large number of databases in order to identify guidelines that were published in peer-review journals as well as in the grey literature. The use of the AGREE II tool is an additional strength as it was developed by a large group of experts in the area of clinical guidelines and has been validated to show that the items and domains included are useful and that the tool can differentiate between high and low quality guidelines;7 ,25 ,26 it has also been applied to clinical guidelines in a range of medical areas.27–30

Limitations of our study include the restriction of our analysis to English language guidelines, which excluded a small number of additional guidelines that we identified in the searches but excluded due to language. In addition, we only assessed information that was either published or available online as this is the information that a guideline user would access. Although we searched multiple databases and searched within each guideline to identify all available information, it is possible that we missed some material. By limiting our review to published and online material, we could not differentiate between items that were addressed by guideline developers but not reported from items that were not addressed (eg, if an internal process is in place to address conflicts of interest, but is not described in the guideline document).

Our results were similar to other guideline reviews in other clinical areas.27–30 In general, other reviews found that guidelines often did well in Scope and Purpose, and Clarity of Presentation but had mixed results in Rigour of Development, Applicability and Editorial Independence.

In light of ongoing concerns regarding the impact of antibiotic selection on resistance patterns as well as treatment failure, high-quality up-to-date clinical treatment recommendations for gonorrhoea are important to delay further drug resistance. Some of the guidelines that we reviewed are ‘parent’ guidelines that have been used by other jurisdictions to create ‘daughter’ guidelines based on the original guideline. This broad use of existing guidelines to guide clinical care in other jurisdictions underlines the importance of well-developed and transparent guidelines. The recommendations in the original guidelines inform the practice of clinicians well beyond the originally intended audience. The results of our systematic review can help guideline developers understand limitations in current clinical guidelines and identify ways to improve on guideline quality in future versions. Most of the reviewed guidelines could be improved by focusing on Rigour of Development and reporting the details of their development process, addressing and recommending solutions for potential issues with implementation (such as the introduction of intramuscular antibiotics), and addressing editorial independence and potential conflicts of interest. In addition, guideline developers may wish to use the AGREE II tool to guide their guideline development and reporting process and identify information to report on in guideline documents.

Future research could look at clinical guidelines from a user's perspective, considering users' knowledge of guidelines, opinions of their recommendations, as well as adherence to guideline recommendations.

Conclusion

Clinical guidelines for the management of gonorrhoea may help to slow further spread of antimicrobial resistance as well as prevent transmission of infection and disease sequelae due to treatment failure. By considering the AGREE II criteria during guideline development and disclosing further details on the guideline development process, guideline developers can improve users' confidence that the recommendations are evidence-informed.

Key messages

  • The guideline development process is important to consider when assessing the quality of a clinical guideline.

  • The quality of gonorrhoea practice guidelines' development process varied.

  • There is room for improvement for many guidelines around their use of current evidence to develop their recommendations, and on the reporting of these processes.

  • Many guidelines did not provide information on how potential conflicts of interest were addressed and whether the guideline's funder influenced the recommendations.

Acknowledgments

The authors acknowledge Sydney Bubis for assistance with data entry and reviewing drafts, Lynda Dickson for reviewing drafts, Stefanie Linklater for her assistance with manuscript preparation, Katrina Sullivan for her assistance with the literature searches and Dr Tom Wong for his feedback on the manuscript.

References

Footnotes

  • Handling editor Jackie A Cassell

  • Contributors CD developed the research protocol, developed the search strategy, conducted the original searches, acted as one of the AGREE II appraisers, conducted the data analysis and wrote the manuscript. TA acted as one of the AGREE II appraisers and reviewed and provided feedback on the manuscript. DSF and GM provided guidance on the planning of the systematic review, and reviewed and provided feedback on the manuscript. JMG provided guidance on the planning of the systematic review, recommended the use of the AGREE II tool and provided input on the write-up of the manuscript.

  • Funding JMG holds a Canada Research Chair for Health Knowledge Transfer and Uptake.

  • Ethical approval Ottawa Health Science Network Research Ethics Board (OHSN-REB) – IRB00002616. REB ethics approval number: OHSN-REB 20140074-01H.

  • Competing interests None declared.

  • Provenance and peer review Not commissioned; externally peer reviewed.

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.