INTRODUCTION

The evidence-based medicine (EBM) movement in the late twentieth century, the Institute of Medicine (now National Academy of Medicine [NAM]) standards for systematic reviews and guideline development, and the development of checklists and tools for the critical appraisal of clinical practice guidelines (CPGs) have highlighted opportunities to improve the rigor, transparency, and trustworthiness of the guideline development process and reinforced the importance of following the appropriate requirements and procedures to produce credible guidelines.1,2,3,4 For instance, guidelines should be based upon a systematic review of the evidence and use a standardized framework for assessing the certainty of the body of evidence and formulating recommendations. One such framework is the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach, a system of guideline development that is widely used by health organizations globally.5 To meet these standards, guideline-producing organizations may seek additional opportunities for training and education or funding to support individuals or groups with necessary methodological expertise.

Collaborations between guideline producers provide a unique opportunity for organizations to pool resources, reduce redundancies, and save time and effort. However, collaborations present their own host of unique challenges, such as ensuring there is alignment in methodological approaches, an appropriate division of labor and authorship, and agreement about ownership and publication rights to guidelines. Despite the challenges, collaboration between guideline producers provides unique opportunities for allowing broader inclusion of content experts and stakeholders, dissemination of guidelines among a wider membership base, and increased uptake by healthcare providers and policymakers.

The present study aimed to (1) better understand the needs of guideline producers and describe the current challenges in developing rigorous and trustworthy guidelines, (2) explore the extent to which organizations collaborate with other entities to produce jointly developed guidelines, and (3) elicit barriers to collaborations between organizations. We conducted a needs assessment survey among guideline producers to explore the perceived value of guidelines, better characterize available resources, and identify current needs, with a specific focus on barriers related to collaboration between organizations.

METHODS

Survey Development and Data Collection

A preliminary version of the survey was developed by the Collaborations Working Group of the Guidelines International Network (G-I-N), then refined and piloted by the US GRADE Network (USGN). The survey was informed by established approaches for conducting systematic needs assessment surveys.6 The survey was then piloted in September 2019 using a convenience sample of attendees (n=24) at a USGN Guideline Development Workshop. Following completion of the survey, a focus group session was conducted using semi-structured and open-ended questions regarding the content, wording, and format of the survey (face validity), redundancy of questions, missing questions, and time for completion. Comments from this focus group were collated and used to inform refinements to the wording of the survey and to develop the next iteration of the questionnaire. Members of the G-I-N Collaborations Working Group and USGN then evaluated the revised survey for content validity and ease of completion.

The final iteration of the survey was disseminated at the annual G-I-N conference in Australia in October 2019 using the online platform SurveyMonkey, Inc. (San Mateo, CA, USA). Additionally, the survey was e-mailed to various listservs for individuals involved in guideline development and evidence synthesis work. Using a snowball sampling method, participants were asked to forward the survey to other interested parties, prohibiting the quantification of total recipients. A minimum of two follow-up reminders were sent between October 2019 and April 2020 to maximize survey responses. These communications were sent periodically in relation to annual systematic review and guideline meetings and based on response rate. No incentives, monetary or otherwise, were offered for participation. Unique site visitors were identified via IP address and personal information was collected on a voluntary basis from participants who wished to be contacted about the survey’s results or to participate in a follow-up in-depth interview. The study was approved by the Institutional Review Board of McMaster University.

Survey Content

The final survey included 46 total questions and was presented in a fixed order and used adaptive questioning (i.e., skip logic) to tailor the survey based upon previous responses to gateway questions. The survey included questions about the following domains: (1) general characteristics and guideline development processes of the respondent’s organization, (2) perceived value of guidelines, (3) perceived resource availability and organizational needs, and (4) collaboration practices among organizations (Appendix 1). Question format included free-text responses, multiple-choice questions, and responses using a seven-point Likert scale. For ten questions about perceived resource availability, answer options ranged from 1 (“Strongly disagree”) to 7 (“Strongly agree”). The survey included thirteen statements related to perceived needs and four questions related to the challenges of collaborations between organizations to develop guidelines. Answer options in this matrix ranged from 1 (“Not relevant [to my organization]”) to 7 (“Extremely relevant”). Finally, respondents rated the potential utility of three services that may facilitate guideline development within their organization (perceived resource and organizational needs). Answer options in this matrix ranged from 1 (“Not helpful at all”) to 7 (“Extremely helpful”).

For the majority of the survey items, respondents could skip individual questions; therefore, denominators are reported for each specific question. For instance, 111 of the 171 total respondents answered the question regarding collaboration; of these, 84 (75.7%) answered that their organization collaborated with other organizations. For questions that were part of a matrix, the lowest number of responses provided for that matrix is reported. The overall distribution of responses to Likert-scale questions was observed to be non-parametric via visual assessment of bar plots. Thus, the median response was presented along with the interquartile range (IQR). For the survey items related to organizational needs, participants who responded with a 5 or greater out of 7 are collapsed.

Statistical Analysis

Subgroup analyses were conducted using the Kruskal-Wallis or Mann-Whitney U test for non-parametric data to identify the potential relationships between organization size, type of organization, collaborative activity of the respondent’s organization and respondents' perceived availability of resources for guideline development, based on a priori hypotheses and following published guidance for reliable subgroup effects.7 Statistical significance was defined as p < 0.05 (two-tailed) for all tests. Post hoc tests were not conducted among multiple subgroups.

RESULTS

General Characteristics and Guideline Development Processes of Respondents’ Organizations

Table 1 summarizes the characteristics of survey respondents. One hundred seventy-one individuals responded to the survey from 30 countries and more than 112 unique organizations, Of these respondents, 59.5% were from North America and 25% were from Europe, with the remaining from Australia and New Zealand; Mexico and Central and South America; Asia; the Middle East; and Africa, in order of representation. The largest portion of respondents (43.5%) listed their primary affiliation as a professional society, whereas 24.1% answered on behalf of an academic institution and 18.2% on behalf of a governmental agency or entity with up to seven respondents representing a single organization or agency.

Table 1 Characteristics of Survey Respondents (n=171)
Full size table

Guideline Development Methodologies and Tools

Three-quarters (75%) of respondents who reported using a specific methodology for guideline development (n=116) reported using the GRADE approach in the development of guidelines, while about one-quarter reported using a proprietary (i.e., internally developed) approach (24.1%), a Delphi/consensus-based approach (25.9%), or an approach that varied depending on the guideline topic (22.4%). Approximately three-quarters (73.8%) of 107 respondents reported using at least one tool to aid the development, reporting, or appraisal of systematic reviews or guidelines within their organization. Over half (53.2%) reported using the Appraisal of Guidelines for Research & Evaluation (AGREE) I/II2 to appraise the quality of guidelines, while 19.0% reported using A Measurement Tool to Assess Systematic Reviews (AMSTAR) 1 or 28 to assess systematic review quality. In terms of reporting checklists, 46.8% reported using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses9 (PRISMA) tool. About 11.4% of respondents reported using various systematic review management software platforms, including Covidence (Veritas Health Innovation, Melbourne, Australia), DistillerSR (Evidence Partners, Ottawa, Canada), and Rayyan (Qatar Computing Research Institute, Hamad Bin Khalifa University, Doha, Qatar).

Less than two-thirds of 110 respondents reported that their organization had a conflict of interest (COI) policy that defined potential disclosures (63.6%), described the management of such disclosures (61.8%), and/or the assessment of potential conflicts (50.9%). The most common forms of COI included in an organization’s policy were financial interests such as stocks and bonds (92.7%), consultancies and speaker’s fees (87.2%), and research funding sources (81.7%). The majority of these respondents (75.5%) reported that COI was collected at the beginning or panel onboarding stage of guideline development, while less than half reported the collection of COI at any other time in the process: at each call or meeting (32.7%), annually (27.3%), before publication (40.9%), or when the guideline was considered for updating (29.1%). See Figure 1.

Figure 1
figure 1

A–C Conflict of interest disclosure and management policies of participating organizations.

Full size image

Fifty-nine percent of respondents (n=110) reported that external review of guidelines was open to select individuals or organizations (by invitation), with about one-third reporting the use of any other approach, such as opening comments to the general public (30%) and/or to the organization’s members (36.4%). Eighty-one percent of respondents (n=110) reported that review by the organization’s leadership or governing body was required for guideline approval, while 70% reported the requirement of scientific review. Fifty-one percent of respondents (n=109) reported that guidelines were updated by their organization at variable intervals of time depending on the topic. Less than one-third (range: 25–29%) reported the use of fixed intervals of time for updating, and 13.8% reported the use of a living guideline format. See Figure 2.

Figure 2
figure 2

Guideline update processes of participating organizations.

Full size image

Perceived Value of Guidelines

The majority of respondents (n=106) responded with at least a 6 on the Likert scale that the development of systematic and rigorous guidelines currently improves the reputation of an organization (85.3%), is preferred over the development of expert review or consensus documents (70.4%), improves the level of satisfaction of an organization’s members (60.7%), leads to benefits that outweigh the costs (58.9%), and improves the impact factor of the journal publishing the guidelines (56.6%). Agreement was the lowest for the statement that systematic and rigorous guidelines improved member retention (30.8%).

Perceived Resources and Organizational Needs

Respondents (n=104) were most likely to agree that their organization’s leadership prioritized the development of systematic and rigorous guidelines (median=6; IQR=4–7), allowed for guideline authors and committees to independently and autonomously complete their work (median=6; IQR=5–7) and that their organization was able to coordinate with journals and publishers to effectively publish and disseminate guidelines (median=6; IQR=4–7). Respondents were least likely to agree that their organization had adequate in-house resources (e.g., librarians) to conduct literature searches and complete evidence reports (median=4; IQR=3–6) or that their organization had adequate resources to develop rigorous guidelines rapidly (median=4; IQR=2–5).

The perceived needs of organizations were broadly categorized into the following subdomains: (1) organization-level needs, (2) guideline development process-specific needs, and (3) methodological challenges. Relevant (≥5 on Likert scale) organizational-level needs included managing the time required for high-quality systematic reviews and guideline development (92.5%) and educating about and communicating the value of rigorous methodology to members, committees, and leadership (86.9%). Common process-specific needs included updating out-of-date guidelines or specific recommendations (87.9%) and training new guideline developers and setting standards for skills and knowledge (86.9%). Methodological challenges that were identified included dealing with low-quality evidence and expert opinion (82.4%) and dealing with lack of collaboration and discordant recommendations within specialties (68.2%). See Figure 3.

Figure 3
figure 3

Challenges to collaboration efforts among organizations that collaborate. The relevance of each challenge was rated on a Likert scale from 1 (“Not relevant “) to 7 (“Extremely relevant”). See Appendix 1 for details about survey questions.

Full size image

Collaboration Practices and Needs Among Organizations

Seventy-six percent of respondents (n=111) indicated that they currently collaborate with individuals or other organizations to produce guidelines. The majority (74.7%) reported that over the past 5 years, five or fewer guidelines had been produced collaboratively by their organization, while 16.2% reported that at least one collaborative effort had resulted in no publication. With respect to the changes in guideline panels for collaborative guidelines, 28% reported an increased number of panel members, 18.0% reported no change, and 26.9% indicated variability depending on the topic or type of collaboration. When presented with a list of needs unique to collaborating organizations, managing the time required to establish collaboration (median=6; IQR=5–6) and collaborating with an organization that uses a different guideline development process or methodology (median=6; IQR=4–7) were rated as the most relevant. See Figure 4.

Figure 4
figure 4

Perceived needs of participating organizations. The relevance of each need was rated on a Likert scale from 1 (“Not relevant to my organization”) to 7 (“Extremely relevant to my organization”). See Appendix 1 for details about survey questions.

Full size image

Subgroup Analyses

Perceived availability of resources did not vary by type of organization (e.g., professional society, governmental organization, etc.; p ≥ 0.054). The perceived availability of adequate funding, in-house resources such as information specialists to conduct literature searches, the ability to coordinate with journals, and to solicit topic suggestions varied based on organization size (p ≤ 0.031; Fig. 5A). The perceived availability of adequate funding as well as the resources required to effectively coordinate with journals and solicit topic suggestions from an organization's membership base varied based on the collaborative activity of an organization (p ≤ 0.004; Fig. 5B).

Figure 5
figure 5

Perceived availability of resources by organization size (A) and collaboration (B). Organization size categorized into large (>10,000 members), mid-sized (1,000–10,000), and small (<1,000). The availability of resources was rated on a Likert scale from 1 (“Strongly disagree”) to 7 (“Strongly agree”). p < 0.05. See Appendix 1 for details about survey questions.

Full size image

DISCUSSION

Principal Findings

To our knowledge, the findings of this survey are the first to address the perceived resource availability and needs of guideline-producing organizations internationally. While the majority of respondents agreed that systematic and rigorous guidelines added value to their organization, there was significant variability with respect to current guideline development processes (including collaborations), availability of resources, and the ongoing needs identified through this survey. Prior studies have focused on specific challenges of guideline development such as the rigor of guidelines,10,11,12 updating guidelines,13,14,15 or improving their uptake and implementation.16,17 Updating old guidelines and recommendations, access to resources related to data synthesis, time required for high-quality systematic review and guideline development, and tools to understand how guidelines are being implemented in practice were viewed as highly relevant needs among our respondents, in agreement with previous studies on this topic.

These findings indicate that the production of systematic and rigorous guidelines is associated with an organization’s reputation and the satisfaction of its members and is also preferred over the development of other guidance documents, such as expert reviews or consensus statements. However, common challenges such as those listed above may pose barriers to the development of rigorous and trustworthy guidelines in lieu of less time-intensive guidance documents.

The survey also indicated a general need for improvement in conflict of interest (COI) disclosure policies and the external review process, with less than two-thirds of respondents indicating that their organization had a COI process that defined, assessed, and/or managed disclosures. Additionally, less than 41% of respondents indicated a defined interval for COI disclosure that occurred more often than at the beginning of panel onboarding. As guidelines can often take several years from inception to publication, processes for reporting new conflicts as they arise are important to ensure the transparency and trustworthiness of guidelines.

Finally, the results of this survey provide unique insights into the collaboration-related efforts of organizations. Though the majority of respondents reported collaborating with other organizations on guideline development efforts, most of those organizations reported producing no more than one collaborative guideline per year on average. This rate of production may be related to the specific challenges to collaboration indicated by respondents, including the management of differing guideline development processes or methodology between organizations and the time and additional steps required to establish collaboration, such as drafting and signing a memorandum of understanding or co-publishing.

Strengths and Limitations

The strengths of the present study include the process for survey development, which involved multiple iterations based upon feedback from in-person pilot-testing with target audience members, and the wide range of geographic regions, roles, and organizational types and sizes represented in the sample. Potential limitations included small sample size and limited number of responses. Efforts to improve response rates were made by sending reminder e-mails and directly reaching out to guideline-producing organizations to maximize survey completion. There is likely response bias in our sample since organizations that had more formalized guideline development processes may have been more likely to complete the survey or individuals may respond differently depending on their role within the organization (e.g., chair of a guideline committee versus program manager). Additionally, the sample represented in this survey (namely, individuals involved in guideline development) may have been more likely to highly rate the perceived value of rigorous and transparent guidelines than guideline end-users. Lastly, while we performed subgroup analyses focused on organization type, size, resource availability, and collaborations, these were exploratory analyses and further research is needed to determine the reliability of any true subgroup effects.

Implications

This survey provides insights for guideline-producing organizations and can help identify areas for improvement in their guideline development process and collaborations with other organizations. The availability of adequate time, funding, and in-house resources for guideline development were commonly shared concerns among our survey respondents. The use of automation or crowdsourcing to increase the speed of evidence synthesis18,19 is one potential solution to the common challenges identified by our survey. Connecting guideline developers to existing databases of high-quality systematic reviews may also help reduce the need for in-house evidence synthesis, more efficiently preserving time and resources for areas where existing syntheses are sparse. Freely available online platforms such as Rayyan for reference screening and ReviewManager (RevMan; the Nordic Cochrane Centre, the Cochrane Collaboration, Copenhagen, Denmark) for data synthesis can be used in place of more costly tools.

The establishment of stronger policies for the disclosure and management of COI among guideline panel members is a relatively simple change that may help improve guidelines' transparency and trustworthiness, and the use of freely available tools such as the G-I-N/McMaster 2.0 checklist,4 the Reporting Items for Practice Guidelines in HealThcare (the RIGHT statement)20 and their extensions such as those for rapid guidelines21 and patient-facing summaries22 may further improve guideline quality. Future, in-depth research is needed to better understand the challenges faced by guideline developers, especially when collaborating with other organizations, and to provide practical solutions for these issues.

Most importantly, the results of the needs assessment can be used by organizations like USGN, G-I-N, Cochrane Centers, and other national and international organizations invested in the advancement of evidence-based guideline development. Findings from this survey can inform strategic planning around training activities and support for services to improve efficiencies in guideline development and guideline collaboration.