An Empirical Study on Data Validation Methods of Delphi and General Consensus

A comparison of three methods to identify chemicals hazards in French research laboratories

Delphi studies, a type of consensus method, are instrumental in healthcare research for gathering expert perspectives, especially when conclusive evidence is not available. Developed in the 1950s, Delphi methodology is characterized by anonymity, iteration, controlled feedback, and statistical group response. The traditional Delphi method, along with its subforms, policy and decision, has been widely used across various fields, including genetic counseling. In genetic counseling, Delphi studies have been used for guideline development, curriculum design, clinical competency selection, and establishing quality metrics. The overall goal of this research methodology article is to explain the potential benefit of using a Delphi method in the field of genetic counseling and differentiate the Delphi method from other consensus methods available. Educational applications include creating curricula for Master's programs and defining competencies for clinical supervision. Delphi studies have also been used to develop core outcome sets and standardize outcome reporting measures in genetic counseling research. Quality assessment in genetic services has also been studied using Delphi studies. In addition to summarizing Delphi studies in genetic counseling, we provide an overview of the major questions to consider when constructing a Delphi protocol. We discuss common design and provide practical tips for implementation such as: who counts as an expert, how to decide how many rounds to do, how to set up the questionnaire, and how to report findings of a Delphi study. Researchers should thoughtfully consider these many points and the impacts these choices may have on their overall study results.

Bedside incision and drainage (I&D) of skin abscesses is a common medical procedure performed in a variety of medical settings. Yet, there is a paucity of published validated educational tools to teach and assess competency for this procedure. To validate an educational tool to teach and assess competency for bedside I&D of skin abscesses via the Delphi consensus and Angoff standard setting methods. Expert consensus on the importance of each procedural step in the educational tool was obtained using the Delphi method, consisting of four rounds of iterative revisions based on input from a panel of experts. The passing cut-off score for a proficient provider was determined using the modified dichotomous Angoff method. All participants met the minimum criteria of active involvement in resident education and performance of at least 20 skin abscess I&D's within the past 5years. Participant specialties included general surgery, emergency medicine, and internal medicine. The primary outcome was consensus on procedural steps and errors, defined as an interquartile range ≤ 2 on a 9-point Likert scale. A cut-off score was determined by the average across all respondents for the anticipated number of errors that would be committed by a provider with the level of proficiency defined in the survey. Qualitative input was incorporated into the educational tool. At the end of four rounds of review via the Delphi process, participants achieved consensus on 93% of items on the clinical checklist and 85% of errors on the assessment checklist. Via the modified dichotomous Angoff method, the determined passing cut-off for competency was 6 out of 22 errors. An educational and evaluation tool for bedside I&D of skin abscesses was validated via the Delphi and Angoff methods.

Purpose: The Delphi and other consensus methods are a systematic means to measure and develop consensus when empirical evidence is lacking or contradictory. They aim to determine the extent to which experts agree about a particular issue, with the ultimate goal of providing a unified expert opinion. In medical education, there are several important areas of inquiry that are plagued by high levels of uncertainty and limited evidence-based literature. Consequently, consensus group methods are relevant to medical educators. Despite extensive use in other fields, consensus methods are poorly standardized and inconsistently described. Several articles highlight significant deficiencies in methodology and reporting.1,2 Given these deficiencies, the following four questions are addressed: (1) How extensively are consensus methods used in medical education research? (2) What types of methods are used? (3) For what purpose? (4) Is there standardization in the application and reporting of the methods? Method: MEDLINE, Embase, PsycINFO, PubMed, Scopus, and ERIC databases were searched for articles focusing on medical education and using the following keywords: “Delphi,” “RAND,” “nominal group,” and “consensus group methods” (2009–2013). Inclusion criteria included English-language and full-text articles of completed research. A standardized extraction form was developed to evaluate the methodology and quality of reporting. Through an iterative process the form and definitions were refined. The final data extraction form consisted of two parts: (1) a section to gather demographic information, such as type of consensus group used and purpose of the project; and (2) specific features reflecting methodological rigor, such as reporting of literature review, number of participants in each round, type of feedback provided, and definition of consensus. Results: The initial search yielded 692 articles. After removal of duplicates, 143 full-text articles met inclusion criteria. Based on previous reviews, this number was deemed to be a sufficiently representative sample. The consensus methods described were the Delphi (40.6%), modified Delphi (31.5%), nominal group technique (NGT) (11.2%), and various other combinations (e.g., Delphi and NGT) (16.7%). The most common purposes were for curricular development or renewal (25.9%), assessment tool development (21%), and defining competencies (10.5%). The quality of reporting was variable; 107/143 (66.4%) described that a literature review was conducted in preparation for the questionnaire, 36/143 (25.2%) described what background information was provided to participants, 93/143 (65%) provided the response rates, 59/143 (41.3%) reported if private decisions were collected, 50/143 (35%) described formal feedback of group ratings, and 48/143 (33.6%) defined consensus a priori. Conclusions: This study of consensus group methods used in the medical education literature highlights the considerable variability in reporting. Studies do not consistently provide sufficient detail about methods, thus leading to a lack of scientific credibility. If consensus methods should inform best education practice, they must be rigorously conducted.

Cabazitaxel is a second line chemotherapy drug recently approved for the treatment of metastatic castration-resistant prostate cancer. A first panel of French experts and a second independent panel of European experts were convened to assess the conformity of prescription of cabazitaxel with a Delphi consensus method. A two-round modified Delphi consensus process was conducted. This methodology is based on experts’ opinion obtained in a systematic manner. The process was divided into five steps: (i) elaboration of the questionnaire, (ii) rating, (iii) analysis, (iv) discussion of the points with absence of consensus following rating of the questionnaire, and (v) final reporting. Consensus was defined according to RAND method and all analyses were conducted according to the same methodology. At the end of the two rounds of rating and a synthesis meeting, of the 26 items included in the Summary of Product Characteristics (SPC), 11 items were judged appropriate with strong consensus by the two independent panels of experts. These items can therefore be considered of prime importance to evaluate conformity of cabazitaxel prescription in the context of observatory studies as well as in further clinical trials using this new taxane. Our findings further provide important evidence about the value of the Delphi consensus and highlight a requirement for “conformity” standards to assist practitioners in a safe chemotherapy drug prescription.

Consensus group methods, such as the Delphi method and nominal group technique (NGT), are used to synthesize expert opinions when evidence is lacking. Despite their extensive use, these methods are inconsistently applied. Their use in medical education research has not been well studied. The authors set out to describe the use of consensus methods in medical education research and to assess the reporting quality of these methods and results. Using scoping review methods, the authors searched the Medline, Embase, PsycInfo, PubMed, Scopus, and ERIC databases for 2009-2016. Full-text articles that focused on medical education and the keywords Delphi, RAND, NGT, or other consensus group methods were included. A standardized extraction form was used to collect article demographic data and features reflecting methodological rigor. Of the articles reviewed, 257 met the inclusion criteria. The Modified Delphi (105/257; 40.8%), Delphi (91/257; 35.4%), and NGT (23/257; 8.9%) methods were most often used. The most common study purpose was curriculum development or reform (68/257; 26.5%), assessment tool development (55/257; 21.4%), and defining competencies (43/257; 16.7%). The reporting quality varied, with 70.0% (180/257) of articles reporting a literature review, 27.2% (70/257) reporting what background information was provided to participants, 66.1% (170/257) describing the number of participants, 40.1% (103/257) reporting if private decisions were collected, 37.7% (97/257) reporting if formal feedback of group ratings was shared, and 43.2% (111/257) defining consensus a priori. Consensus methods are poorly standardized and inconsistently used in medical education research. Improved criteria for reporting are needed.

Objective: The growing importance of evidence based practice in athletic training is necessitating academics and clinicians to be able to make judgments about the quality or lack of the body of research evidence and peer-reviewed standards pertaining to clinical questions. To assist in the judgment process, consensus methods, namely brainstorming, nominal group technique and the Delphi method can be used. The purpose of this paper is to review the literature related to the Delphi Method and its potential implications for evidence-based practice and peer-reviewed standards in athletic training. Data Sources: We searched PubMed and MEDLINE (1978–2007), CINAHL (1993–2006), Dissertation Abstracts (1979–2006) and Google Scholar (1983–2007) using the terms “Delphi method,” “modified Delphi technique,” “consensus methods,” “Delphi technique,” and combined search terms of “Delphi method AND allied health, AND medicine AND dentistry, AND nursing. Data Synthesis: Textual support for the use of the Delphi Method in athletic training and a brief review of the literature pertaining to the: objectives; advantages; limitations commonly associated with the use of the Delphi Method; and research protocol. Conclusions/Recommendations: The Delphi Method in athletic training has been used to fulfill two objectives; the need for evidence based practice and the need to establish policies and procedures when none are in existence or it is difficult for one individual to make a decision. The Delphi Method and other consensus development methods should not be viewed as a scientific method for creating new knowledge, but rather as processes for making the best use of available information, be that scientific data or the collective wisdom of participants.

Introduction The Nominal Group Technique (NGT) and Delphi Technique are consensus methods used in research that is directed at problem-solving, idea-generation, or determining priorities. While consensus methods are commonly used in health services literature, few studies in pharmacy practice use these methods. This paper provides an overview of the NGT and Delphi technique, including the steps involved and the types of research questions best suited to each method, with examples from the pharmacy literature. Methodology The NGT entails face-to-face discussion in small groups, and provides a prompt result for researchers. The classic NGT involves four key stages: silent generation, round robin, clarification and voting (ranking). Variations have occurred in relation to generating ideas, and how ‘consensus’ is obtained from participants. The Delphi technique uses a multistage self-completed questionnaire with individual feedback, to determine consensus from a larger group of ‘experts.’ Questionnaires have been mailed, or more recently, e-mailed to participants. When to use The NGT has been used to explore consumer and stakeholder views, while the Delphi technique is commonly used to develop guidelines with health professionals. Method choice is influenced by various factors, including the research question, the perception of consensus required, and associated practicalities such as time and geography. Limitations The NGT requires participants to personally attend a meeting. This may prove difficult to organise and geography may limit attendance. The Delphi technique can take weeks or months to conclude, especially if multiple rounds are required, and may be complex for lay people to complete.

El método de consenso es un grupo de técnicas de investigación utilizado con poca frecuencia para la investigación de servicios de salud en América Latina. El método de consenso proporciona a los investigadores la capacidad de lograr una estimación cuantitativa a través de un enfoque cualitativo. El método de consenso representa un enfoque epistemológico de las ciencias inexactas. Este método que incluye el grupo nominal y la técnica Delphi es útil cuando se requiere un acuerdo de expertos sobre la formulación de recomendaciones, especialmente cuando la evidencia científica no está disponible, las evidencia existentes se debate, o si existe un pequeño equilibrio entre el riesgo-beneficio. El objetivo general es llegar a una conclusión válida, o consenso, de una deliberación grupal repetitiva en respuesta a una pregunta de investigación bien definida. La técnica Delphi es un método esencial para lograr consenso en áreas donde no existía anteriormente evidencia, es limitada, de baja calidad y / o sujeta a debate. Existen cuatro áreas generales en las que los investigadores deberían considerar la técnica Delphi, que incluyen 1) El problema de la investigación no se presta a técnicas analíticas precisas, pero puede beneficiarse de juicios subjetivos sobre una base colectiva; 2) La población de investigación presenta diversos antecedentes con respecto a la experiencia y pericia; 3) Se requieren más sujetos que puedan interactuar eficazmente frente a frente; y 4) El tiempo, los costos y la logística harían inviable las reuniones frecuentes de todos los involucrados. La técnica Delphi crea un entorno estructurado para que los expertos evalúen la mejor información disponible relacionada a una pregunta de investigación y formulen soluciones para responder a esta pregunta a través de un proceso concreto. En general, la técnica Delphi es un poderoso método de investigación cualitativo que debe utilizarse en la investigación de servicios de salud en América Latina.

Using Consensus Techniques to Produce Clinical Guidelines for Patients Treated with the Ilizarov Fixator

ObjectiveTo establish an expert consensus on which criteria are the most appropriate in clinical practice to refer patients with systemic sclerosis (SSc) for right heart catheterisation (RHC) when pulmonary hypertension...

Authors' Response.

The Delphi method: A tool to support injury control and trauma care policy

Consensus methods such as Delphi studies or the group nominal method have long been used in healthcare research to develop services, processes or policies and inform further research. However, concept analysis and meta-synthesis tools are used more frequently to seek a collective understanding of subjective terms. To examine the use of consensus methods to develop linguistic clarity in healthcare research. The authors argue that consensus methods are appropriate for determining linguistic clarity when researching subjective terms. The paper includes a sample research design that incorporates this approach. Consensus methods, supported by interpretive synthesis of the concept and research, can enrich our understanding of subjective terms used in healthcare research. Understanding the importance of linguistic clarity is an important step for healthcare researchers. Consensus methods, if managed effectively and conducted in line with the appropriate research guidelines, can bring a richer understanding to concepts. This paper presents a research example that incorporates the use of a consensus method and which healthcare researchers can use to reduce the potential ambiguity of subjective terms in their research.

Risky behaviours from the production to the consumption of bivalve molluscs: Involving stakeholders in the prioritization process based on consensus methods