Appraisal of International Guidelines on Smoking Cessation using the AGREE II Assessment Tool
K.I. Quintyne1, P. Kavanagh1
1. Department of Public Health, Health Service Executive (HSE) North-East, Meath, Ireland
On behalf of the Guideline Development Group on Smoking Cessation
Several international professional bodies have produced and disseminated clinical practice guidelines (CPGs) for smoking cessation. However, to date, the quality of guidelines for use in the Irish context has never been appraised and explored. The aim of this study was to identify and evaluate the quality of methodological rigours and transparency used in guidelines for smoking cessation (for specific groups including: general adult population; persons with mental illness; and pregnant women).
The research for the relevant smoking cessation guidelines was conducted using a systematic search strategy of scientific databases (including guideline websites; PubMed; and Google ®) from January 2006 to June 2017. The quality of the CPGs was independently assessed by at least two assessors using the Appraisal of Guidelines for Research & Evaluation II (AGREE II) instrument, and specific recommendations in guidelines were evaluated. Domain scores were considered of sufficient quality when ≥ 60% and of good quality when ≥ 80%.
Nine guidelines were retrieved. Five guidelines scored ≥ 60% in at least four domains. The median score for the scope and purpose domain was 80% (Range: 61 to 94%). The median score for the stakeholder involvement domain was 63% (Range: 26 to 85%), and six guidelines scored ≥ 60%. The median score for the rigour of development domain was 39% (Range: 23 to 77%), and four guidelines scored ≥ 60%. The median score for clarity of presentation domain was 89% (Range: 56 to 96%), and eight guidelines scored ≥ 60%. The median score the applicability domain was 39% (Range: 21 to 57%). None of guidelines scored ≥ 60%. The median score for the editorial independence domain was 78% (Range: 0 to 90%); only seven guidelines scored ≥ 60%.
Smoking cessation guideline quality assessment varied across all six AGREE II domains, demonstrating the importance of using a formal appraisal tool prior to guideline adaptation and implementation into clinical settings. Our findings have demonstrated higher scores among the most recent guidelines, reflecting improvement in the quality of guideline development over time. Methodology and editorial independence were particular concerns and this assessment also highlighted a need for contextualisation to the Irish healthcare system. In conclusion, the plan for Ireland is to adapt rather than simply adopt existing guidelines.
Tobacco smoking is a major preventable risk fact for the development of non-communicable diseases, including cancers, cardiovascular and respiratory diseases.1 Accordingly, 12% of all adult deaths worldwide can be attributable to tobacco use.2 In Ireland, among persons 15 years and older, the prevalence of tobacco use is 22%. Smoking prevalence is, however, higher: among males (25%) than females (20%); and among those aged 25 to 34 years (34%) as compared to other age groups.3
Long-term smoking cessation strategies have been shown to reduce health risks, and consequently reduce the impact on both morbidity and mortality associated with tobacco use.1 4 5 From the international evidence, it has been well documented that nationally implemented services for smoking cessation support (i.e. face-to-face support, and telephone-/internet- based facilities), have been found to be effective in successful smoking cessation.6-8 Additionally, easy and timely access to both pharmacological and behavioural intervention can contribute to improving smoking cessation rates.9
Published reports have highlighted that smokers are more commonly identified, advised and prescribed supports for smoking cessation in a primary care setting (i.e. in general practice).4 Healthcare systems with established infrastructure and smoking cessation programmes in place, general practice is the favoured environment for the identification and referral of smokers to take place.4 10
Clinical practice guidelines (CPG) in which this evidence is communicated to healthcare professionals can be effective in improving clinical practice, CPG for smoking cessation help to translate research into clinical practice.11-13 Therefore, it is critical that guidelines reflect the best practice, and are kept uptodate and cover areas of greatest need. In Ireland, the National Clinical Effectiveness Committee (NCEC) provides a framework for national endorsement of clinical guidelines and audits; whereby it systematically reviews statements, based on a thorough evaluation of the evidence, to assist practitioner and service users’ decisions about appropriate healthcare for specific clinical circumstances across the entire clinical system.14 A Guideline Development Group on Smoking Cessation was established to develop national guidelines. This was in fulfilment of Ireland’s obligation as a signatory to the World Health Organization’s (WHO) Framework Convention on Tobacco Control (FCTC).
We assessed the quality of smoking cessation CPG which included recommendations for smoking cessation among: adult population; persons with mental illness; and pregnant women. We used the Appraisal of Guidelines for Research and Evaluation (AGREE II) tool to evaluate each guideline.15 The AGREE II tool is a widely-used instrument to assess methodological rigour and transparency of guideline development and has been tested for its validity and reliability. It uses a details framework to assess guideline development and content.
The primary outcomes of this study was to inform the development of national CPG for the Health Service Executive (HSE), though a robust and comprehensive review of the quality of published smoking cessation guidelines so as to determine if any guidelines were suitable for adoption or adaptation and use in Ireland.
The search for guidelines for smoking cessation was carried out using CPG issued by national scientific societies or by government organisations between January 2006 and June 2017 in every language, through the use of appropriate keywords and the following search engines:
• PubMed/PubMed Central (Available URL: https://www.ncbi.nlm.nih.gov/pubmed/);
• National Guideline Clearinghouse (Available URL: www.guideline.gov);
• NICE: National Institute for Health and Care Excellence (Available URL: www.nice.org.uk);
• Canadian CPG InfoBase: clinical Practice Guidelines Database (Available URL: www.cma.ca/En/Pages/clinical-pracitce-guidelines.aspx);
• SIGN: Scottish Intercollegiate Guidelines Network (Available URL: www.sign.ac.uk);
• Australian Clinical Practice Guidelines (Available URL: http://www.clinicalguidelines.gov.au/);
• Guidelines International Network (Available URL: http://www.g-i-n.net/);
• Cochrane Library (Available URL: http://www.cochranelibrary.com);
• FDA: Food and Drug Administration (Available URL: https://www.fda.gov/regulatoryinformation/guidances/).
Additional research was conducted on Google ®/Google Scholar ®. In this research, we used the following keywords: ‘smoking/nicotine-addiction/tobacco-use cessation’, ‘smoking/nicotine-addiction/tobacco-use interventions’, and ‘treatment of smoking/nicotine-addiction/tobacco-use’. The search strategy was limited to January 2006 and not earlier as the latest pharmaceutical agent for smoking cessation, Varenicline (trade name Chantix ® and Champix ®) was licensed and introduced into clinical practice from September 2006. The guidelines for smoking cessation from NICE were not included, as they were being updated at the time of generation of candidate CPG. As they are our closest neighbours with similar healthcare infrastructure, they will reviewed when available an added to final list of candidate CPG.
Guidelines that did not focus on the management of smoking cessation were excluded, as well as any documents that were not guidelines (such as position papers and reviews).
Each candidate guideline was reviewed by two to four assessors. At least one assessor for each guideline had experience in developing and evaluating guidelines. All the assessors used the online training tools recommended by the AGREE collaboration before conducting the appraisals. They independently evaluated the included guidelines using the AGREE II instrument, which consists of a total of 23 items in six domains: ‘Scope and purpose’, ‘Stakeholder involvement’, ‘Rigour of development’, ‘Clarity of presentation’, ‘Applicability’, and ‘Editorial independence’. Each item was rated on a seven-point scale from 1 (strongly disagree) to 7 (strongly agree). A scaled domain percentage score was calculated, according to the AGREE II methodology, as follows: (obtained score-minimum possible score) / (maximum possible score-minimum possible). The ‘obtained score’ is the sum of the appraisers’ scores per each item, making it possible to consider the natural discrepancies between the appraisers evaluating a candidate guideline.
Although the domain scores are useful for comparing guidelines and attest whether a guideline should be recommended for use, the AGREE II instrument does not set minimum domain scores or patterns of scores across domains to differentiate between high quality and poor quality guidelines. These decisions should be made by the users and guided by the context in which AGREE II is being used. Then, as reported in studies in the literature, we considered a value > 60% as sufficient quality score and a value > 80% as a good quality score.
On completing the 23 items, the appraisers provided the overall assessment for each guideline, and decided which guideline was recommendable, with or without modifications, which was not recommendable. This choice was the result of the six domains’ score and of the personal judgement of the appraisers. The Clinical Guideline Development Group decided to recommend without modification the guidelines with an overall score equal to 7, to recommend with modifications the guidelines score ≥ 5 but < 7, and not to recommend the guidelines with an overall score ≤ 4.
Comparison of recommendations
Recommendations regarding interventions for smoking cessation, reported in the selected guidelines, have been extracted and summarised in comparative tables focussing on common messages and highlighting potential gaps.
In total, 871 guidelines were identified as possibilities from the above keyword search strategy. Sixty-nine potentially relevant articles met the inclusion criteria and were entered for further screening. The screening strategy is outlined in Figure 1. At the first screening, titles were reviewed and all duplicates were removed. At the second screening, all non-English guidelines were removed. At the third screening, all executive summaries were reviewed and all that did not meet the inclusion criteria were removed. And on the fourth screening, full documents were reviewed, and if they were not available they were removed, resulting in nine candidate guidelines for consideration.
The characteristics of the included candidate CPG are presented in Table 1.
The domain-standardised scores for the selected guidelines and overall recommendations are presented in Table 2 and Figure 2.
The Australian guidelines had good scores only in clarity of presentation domain; sufficient scores in scope and purpose, stakeholder involvement, and editorial independence domain; and low scores in rigour of development and applicability domains.
The Canadian guidelines for general population had good scores in scope and purpose clarity of presentation, and editorial independence; sufficient scores in stakeholder involvement, and rigour of development domains; and low scores in the applicability domain.
The Canadian guidelines for pregnancy had good scores in clarity of presentation domain; sufficient scores in scope and purpose, stakeholder involvement, rigour of development, and editorial independence domains; and low scores in the applicability domain.
The Indian guidelines had no good scores in any domain; sufficient scores in the scope and purpose domain; and low scores all other domains, with a zero for editorial independence.
The Japanese guidelines had no good scores in any domain; sufficient scores in scope and purpose, stakeholder involvement, and clarity of presentation domains; and low scores in rigour of development, applicability, and editorial independence domains.
The guidelines from New Zealand had good scores in scope and purpose, and clarity of presentation domains; sufficient scores in stakeholder involvement, rigour of development, and editorial independence domains; and low scores in the applicability domain.
The South African guidelines had good scores in scope and purpose, and editorial independence domains; sufficient scores in only the clarity of presentation domain; and low scores in stakeholder involvement, rigour of development, and applicability domains.
The guidelines from the USA had good scores in scope and purpose, stakeholder involvement, clarity of presentation, and editorial independence domains; sufficient scores in only the rigour of development domain; and low scores in the applicability domain.
The European guidelines had good scores in scope and purposes, and clarity of presentation; sufficient scores in only the editorial independence domain; and low scores in stakeholder involvement, rigour of development, and applicability domains.
Scope and purpose
The median score for the scope and purpose domain was 80% (Range: 61% to 94%). Most guidelines clearly described their overall objectives, health questions and target populations. The guideline from India had the lowest score.
The median score for the stakeholder involvement domain was 63% (Range: 26% to 85%). Only the Australian, Canadian (general and pregnancy populations), Japanese, New Zealand, and USA guidelines scored above 60% for this domain. However the Indian and South African guidelines did not openly reveal the views and preferences of the target populations under review. No guideline clearly described their members’ roles in the guideline development process. Furthermore, none of the guidelines included health economists among the guideline authors.
Rigour of development
The median score for rigour of development domain was 39% (Range: 23% to 77%). Only the Canadian (general and pregnant populations), New Zealand, and USA guidelines scored above 60% because they used systematic methods of searching for evidence and for formulating recommendations; only the Canadian (general population) and USA guidelines clearly described methods for conducting external reviews; and only the guidelines from USA described their procedures for updating guidelines.
Clarity of presentation
Most of the guidelines provided specific, unambiguous and easily identifiable recommendations. The median score for the clarity of presentation domain was 89% (Range: 56% to 96%). Only the Indian guideline scored less than 60%.
The median score for the applicability domain was 39% (Range: 21% to 57%). None of the guidelines scored above 60%. Most of the guidelines did not describe the facilitators and barriers of their applications and did not adequately consider the cost of applying their recommendations.
The median score for the editorial independence domain was 78% (Range: 0% to 90%); only the Indian guidelines did not score above 60%. All of the guidelines did not clearly provide financial support information; and additionally did not report any possible funding influence on the guideline content.
Based on the six domain scores, the Australian, Canadian (general and pregnant populations), New Zealand, USA, and European guidelines were recommended with modifications, and Indian, Japanese and South African were not recommended. None of the candidate guidelines were recommended without modifications.
Specific recommendations have been identified and summarised in Table 3. They primarily focussed on pharmacological and non-pharmacological interventions for smoking cessation.
This study aimed to assess: (1) the quality of national guidelines for smoking cessation; (2) the recommendations that were made within these guidelines; and (3) how these recommendations correlated among the guidelines reviewed. It involved a systematic search of the available national guidelines regarding the management of smoking cessation. The key recommendations were compiled and reviewed.
Overall, only nine guidelines were retrieved, which was less than expected is considering that smoking and its health-related impacts represents a considerable part of the healthcare provision worldwide. All candidate guidelines considered focussed on pharmacological and non-pharmacological interventions for smoking cessation.16-24
In this review, nine guidelines from seven countries and the European Union were included in the analyses. Eight of the guidelines focussed on the treatment of tobacco dependence in the general population, and three (including one focussing only on pregnant women only) including specific populations: pregnant smokers (Canada, USA and EU).16-24 The guidelines were produced/commissioned by varying entities, which included: governmental organisations, medical societies, research networks, and research centres.
Appraisal of the candidate guidelines for smoking cessation using the quality scores from the AGREE II instrument was employed to objectively review the impact of their quality relative to the ease of their application.15
Interestingly, lower quality scores were generally observed in the domains of rigour of development, applicability, and editorial independence, whereas all the guidelines had moderate to high scores in the scope and purpose, and clarity of presentation of recommendations. The methodology analysis showed acceptable results only for the Australian, Canadian (general and pregnant populations), New Zealand, USA and Europe, whereas the Indian, Japanese, and South African guidelines had insufficient scores because they were not openly robust with the evidence to support their recommendations, and they also did not involve all of the relevant stakeholder in their guideline development group.16-24
Regarding the ‘applicability’ domain, most of the guidelines did not describe the necessary facilitators and barriers to their application and did not provide relevant audit criteria.17-20 22 24 Furthermore, some of the candidate guidelines lacked a summary document, and educational tools. This finding is in contrasts with the need for clarity and user-friendliness suggested in the published literature.15 This needs to be a major consideration when developing any new guidelines. Only the guidelines from USA and EU reported detailed information on potential conflicts of interest.23 24 This is highly relevant when considering that conflicts of interest are the most common sources of bias in guideline development.15 A noted strength of this work on guideline development for smoking cessation in Ireland, is that it used the AGREE II instrument to objectively assess the methodological quality of guidelines related to smoking cessation interventions.14 This is in keeping with national recommendations as outlined by NCEC.
The AGREE II instrument is a tool that assesses the methodological rigour and transparency with which a guideline is developed.15 A potential limitation of this method is that there is no threshold for distinguishing between high quality and low quality candidate guidelines. Thus, the guideline quality would be left to the appraisers to identify and the scores of an AGREE II evaluation have to be interpreted with caution and confined to a particular situation. Furthermore, AGREE II does not consider the relative importance of the six domains of quality: rigour of development is considered of equal importance to the other five domains.15 This suggests that the domains of AGREE II should not be weighed equally. If the guideline has a low score on the domain of rigour of development, the corresponding recommendations have a high risk of bias, and the other domains are of little relevance in quality assessment. Another possible limitation of this study is that some of the guidelines may have been missed by our research.
Overall, the recommendations made from the assembled guidelines focussed on the treatment of tobacco dependence in the primary care setting. In particular, all guidelines agree that combined intervention strategy employing pharmacological and non-pharmacological means was associated with the greatest success for sustained smoking cessation.16-24 The majority of the guidelines recommended that smokers should be promptly identified, be offered a brief advice to quit, be assessed for motivation to quit, and be offered support to quit with either pharmacological and/or non-pharmacological intervention.17 18 20 21 23
Some inconsistencies were noted among the guidelines. The first was related to the specific content (i.e. practical counselling techniques vs. no details on content) and delivery format (i.e. via telephone, web-based platform or face-to-face vs. multiple formats) of non-pharmacological intervention varied greatly throughout the guidelines.17 18 20-22 24 These can be partly explained by the differences within the healthcare infrastructure at the country level.
Other inconsistencies among the recommendations were related to the provision of pharmacotherapy for smoking cessation among the general population. Excluding the Canadian guidelines, which do not make recommendations relating to the use of pharmacotherapy, NRT was recommended by all the guidelines.16 19-24 Both bupropion and varenicline were recommended by the majority of guidelines (each reporting 6/9; 67%), and a combination of NRT was recommended by only five of nine guidelines. Recommendations for the use of nortriptyline (4/9; 44%), a combined NRT and bupropion (3/9; 33%), and clonidine (2/9; 22%) were less common. These variations between recommendations may be due to differences in pharmaceutical licensing across countries, or to a difference in access or interpretation of the current available scientific evidence. Furthermore, the costs of pharmacological interventions could be a barrier in certain countries for uptake of certain recommendations within their guidelines.
Finally, recommendations for specific subpopulations were also less consistent throughout guidelines. The major highlighted issue was whether or not to use NRT for pregnant women; the Canadian guidelines only recommended non-pharmacological intervention, while other guidelines supported NRT use.18 23 24 Cultural influences (i.e. the place of pharmacotherapy within pregnant population) may play a role as well.
In conclusion, most guidelines issued at national or international levels were of good quality and could be adopted into clinical practice within the Irish healthcare system. Nevertheless, the quality of the candidate guidelines could be improved, in particular the methodological, applicability, and editorial independence domains. This undertaking further highlights the need to adapt the best features from the gathered candidate guidelines to facilitate more sustainable Irish CPG for effective smoking cessation.
We would like to thank the Guideline Development Group on Smoking Cessation for its expertise and input into the generation of the candidate guidelines. We would also like to thank the work of the many appraisers from the Faculty of Public Health, RCPI.
Conflict of Interest
The authors declare no conflict of interest.
Dr. Keith Ian Quintyne
Department of Public Health,
Ph: 046 907 6412
1. The Health Consequences of Smoking-50 Years of Progress: A Report of the Surgeon General. Atlanta GA2014.
2. World Health Organization (WHO) | Publications - Global Tobacco Epidemic: World Health Organization; 2013 [updated 04-11-2013. Available from: http://www.who.int/tobacco/publications/en/index.html.
3. Department of Health (DoH). Healthy Ireland Survey 2017: Summary of Findings 2017 [Available from: http://health.gov.ie/wp-content/uploads/2017/10/16-048825-Healthy-Ireland-Survey-18-October_for-printing.pdf accessed 14th December 2017.
4. Pirie K, Peto R, Reeves GK, Green J, Beral V, Million Women study Collaborators. The 21st century hazards of smoking and benefits of stopping: a prospective study of one million women in the UK. Lancet 2013;381(9861):133-41.
5. Kenfield SA, Stampfer MJ, Rosner BA, Colditz GA. Smoking and smoking cessation in relation to mortality in women. JAMA 2008;299(17):2037-47.
6. Bauld L, Bell K, McCullough L, Richardson L. The effectiveness of NHS smoking cessation services: a systematic review. J Public Health (Oxf). England2010:71-82.
7. Miller CL, Wakefield M, Roberts L. Uptake and effectiveness of the Australian telephone Quitline service in the context of a mass media campaign. Tobacco control 2003;12 Suppl 2:ii53-8.
8. Durkin S, Brennan E, Wakefield M. Mass media campaigns to promote smoking cessation among adults: an integrative review. Tobacco control. England2012:127-38.
9. Levy DT, Chaloupka F, Gitchell J. The effects of tobacco control policies on smoking rates: a tobacco control scorecard. J Public Health Manag Pract. United States2004:338-53.
10. Rimer BK, Gierisch JM. Public education and cancer control. Seminars in oncology nursing 2005;21(4):286-95. doi: 10.1016/j.soncn.2005.06.003 [published Online First: 2005/11/19]
11. Davis DA, Taylor-Vaisey A. Translating guidelines into practice. A systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines. CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne 1997;157(4):408-16.
12. Greenhalgh T, Howick J, Maskrey N. Evidence based medicine: a movement in crisis? BMJ 2014;348:g3725.
13. Pinnock H, Ostrem A, Rodriguez MR, Ryan D, Stallberg B, Thomas M, Tsiligianni I, Williams S, Yusuf O. Prioritising the respiratory research needs of primary care: the International Primary Care Respiratory Group (IPCRG) e-Delphi exercise. Prim Care Respir J. England2012:19-27.
14. National Clinical Effectiveness Committee (NCEC). Clinical Effectiveness 2018 [Available from: http://health.gov.ie/national-patient-safety-office/ncec/ accessed 26th February 2018.
15. Brouwers MC, Kho ME, Browman GP, Burgers JS. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne 2010;182(18):E839-42. doi: 10.1503/cmaj.090449 [published Online First: 2010/07/07]
16. Royal Australian College of General Practitioners (RACGP). Supporting smoking cessation: a guide for health professionals 2011 [Available from: https://www.racgp.org.au/your-practice/guidelines/smoking-cessation/ accessed 15th February 2018.
17. Canadian Action Network for the Advancement Dissemination and adoption of Practice-informed Tobacco Treatment (CAN-ADAPPT). Canadian Smoking Cessation Clinical Practice Guideline 2011 [Available from: https://www.nicotinedependenceclinic.com/English/CANADAPTT/Documents/CAN-ADAPTT Canadian Smoking Cessation Guideline_website.pdf accessed 15th February 2018.
18. Canadian Action Network for the Advancement Dissemination and adoption of Practice-informed Tobacco Treatment (CAN-ADAPPT). Canadian Smoking Cessation Guideline: Specific Populations: Pregnant and Breastfeeding women. 2011 [Available from: https://www.nicotinedependenceclinic.com/English/CANADAPTT/Documents/Guideline/Pregnant and Breastfeeding Women.pdf accessed 15th February 2018.
19. National Tobacco Control Programme (NTCP). Tobacco Dependence Treatment Guidelines 2011 [Available from: http://www.tabaccologia.it/PDF/India treatment guidelines in English 2011.pdf accessed 15th February 2018.
20. Japanese Circulation Society (JCS). Guidelines for Smoking Cessation 2010 [Available from: https://www.jstage.jst.go.jp/article/circj/76/4/76_CJ-88-0021/_pdf accessed 15th February 2018.
21. National Institute for Health Innovation (NIHI). The New Zealand Guidelines for Helping People to Stop Smoking 2014 [Available from: https://www.health.govt.nz/publication/new-zealand-guidelines-helping-people-stop-smoking accessed 15th February 2018.
22. South African Thoracic Society (SATS). South African tobacco smoking cessation clinical practice guideline 2013 [Available from: http://www.cansa.org.za/files/2014/05/SA-Tobacco-Smoking-Cessation-Clinical-Practice-Guideline-2014.pdf accessed 15th February 2018.
23. Agency for Healthcare Reseaarch and Quality (AHQR). Behavioural and pharmacotherapy interventions for tobacco smoking cessation in adults, including pregnancy women: U.S. Preventative Services Task Force recommendation statement 2015 [Available from: https://www.uspreventiveservicestaskforce.org/Page/Document/draft-recommendation-statement147/tobacco-use-in-adults-and-pregnant-women-counseling-and-interventions1 accessed 15th February 2018.
24. European Network for Smoking and Tobacco Prevention (ENSP). Guidelines for treating tobacco dependence 2016 [Available from: http://ensp.org/wp-content/uploads/2016/12/ENSP-ESCG_FINAL.pdf accessed 15th February 2018.