Developing core outcome sets for clinical trials: issues to consider

Introduction and aims A core outcome set (COS) is defined as an agreed standardised set of outcomes that should be measured and reported, as a minimum, in all clinical trials in specific areas of health or health care. Their use allows research to be compared and combined as appropriate, and may ensure that all studies provide usable information. There is currently no accepted gold standard method for COS development and further work was necessary to explore choices about methods, and what the priorities are for guidance and further research in this area. This thesis aimed to investigate what is currently known about COS development, and explore developers’ experiences of developing COS. Methods A systematic review of studies reporting the development of a COS was undertaken, and the methodological techniques used in these studies was described. A mixed methods approach was undertaken to explore COS development, drawing on qualitative interviews with, and an online web-based survey of, COS developers. This thesis used a Triangulation Design to obtain different but complementary data on the same topic for comprehensiveness. Results The systematic review identified 198 published studies that described the development of COS for clinical trials. The systematic review demonstrated variability in the ways that COS had been developed, particularly the methods used and the stakeholders included as participants in the process. Patient participants had infrequently been included in the development of COS (18%). Key aspects of the process were frequently not reported. Eighty-one (48%) developers completed the survey. The majority of survey respondents (73%) felt that there is a need for methodological guidance or research to inform future activity to develop COS. Areas for future guidance or research included: stakeholder involvement, patient involvement in particular; choice of methodology, and consensus formation. 32 interviews were conducted with COS developers (18 with published, and 14 with ongoing, COS projects). Developers found the process of COS development to be a challenging process, in part due to the nature of COS development being an emerging field of research, but also in part to not always considering important methodological details from the outset, for example their choice of methods and stakeholders. There was a variety of influences on developers’ choice of methods, which included the previous literature on COS development, expert advice, developers’ own experience with methods and the resources available to developers. The absence of guidance in COS development, and the prominence of uncertainties, dominated developers’ accounts. Conclusions The work in this thesis has brought COS together in one place for the first time, summarises key characteristics of COS and their development, and provides the first comprehensive account of COS development. It will inform the development of much needed guidance in this area and help to improve COS development methodology. Guidance needs to determine commonalities across different disease areas, and promote awareness of important issues; encourage COS developers to think about their own contexts and circumstances, and enable COS developers to make decisions about methods that best suit their needs and resources. Guidance seems to be needed for all aspects of COS development, but it was particularly felt that guidance around the systematic review process, conduct of Delphi, and conduct of consensus meetings, are high priority.

The Core Outcomes in Women’s Health (CROWN) Initiative

Clinical trials, systematic reviews and guidelines compare beneficial and non-beneficial outcomes following interventions. Often, however, various studies on a particular topic do not address the same outcomes, making it difficult to draw clinically useful conclusions when a group of studies is looked at as a whole. This problem was recently thrown into sharp focus by a systematic review of interventions for preterm birth prevention, which found that among 103 randomised trials, no fewer than 72 different outcomes were reported. There is a growing recognition among clinical researchers that this variability undermines consistent synthesis of the evidence, and that what is needed is an agreed standardised collection of outcomes - a "core outcomes set" - for all trials in a specific clinical area. Recognising that the current inconsistency is a serious hindrance to progress in our specialty, the editors of over 50 journals related to women's health have come together to support The CROWN (CoRe Outcomes in WomeN's health) Initiative.

Core outcome sets (COS) are an agreed upon set of outcomes that should be evaluated in a clinical trial, allowing data from similar areas of health care to be compared and pooled in meta-analyses, informing whether treatments work and/or harm. Pediatric COS have been developed by identifying outcomes that are important to relevant stakeholders, and recently the importance of including children and young people (CYP) as stakeholders has been recognized. While research on the overall development of pediatric COS exists, no comprehensive review has examined methods implemented by COS developers to include CYP in the development process. This systematic review aims to establish current practice and identify issues in including CYP in developing a pediatric COS, particularly in the context of rare conditions. We conducted a systematic review of studies in the Core Outcome Measures in Effectiveness Trials (COMET) initiative database that developed pediatric COS for children and young people (CYP; 0-18 years). Data were extracted on methods that used to include CYP, barriers to their inclusion and stakeholder participation. Authors were contacted when methodological details were unclear or insufficiently reported. A total of 70 articles corresponding to 53 pediatric COS were analyzed, 18 (34%) included CYP and 16 (30%) were developed for a rare condition/condition with a rare form. CYP were included in advisory groups (six of 18 COS), pilot surveys/questionnaires (three of 18 COS), focus groups (two of 18 COS), workshop (one of 18 COS), qualitative interviews (eight of 18 COS), Delphi (and other) surveys (15 of 18 COS), and consensus meetings (five of 18 COS). COS developers adapted methods for patient and public involvement and engagement work to improve participation in COS, include CYP in generating outcomes, and reach consensus on the most important outcomes to include in the COS. Barriers to including CYP were age and the condition/area of COS development. Our findings show that with appropriate adaptations, CYP can be included in COS development, regardless of age and communication needs, or whether conditions are rare or common. This review proposes adaptations to refine current pediatric COS methodology to increase CYP inclusion. Core outcome sets (COS) are lists of the most important things to measure in a clinical trial-such as symptoms, side effects, or any changes in a person's health-to help show whether a treatment works. When a COS is used in clinical trials for a particular condition, the results are more helpful because they can be combined and compared. This makes it easier to see whether a treatment is helping or harming the people who have that condition. A COS is developed in different stages; first, a list of important things to measure, called outcomes, is put together. One way to make this list is by asking the people who know the condition best-such as patients who live with it and the health-care professionals or researchers who treat or study it-, and what they think should be included. Next, everyone involved votes on which outcomes they think are the most important. The outcomes that receive the most support across all groups are then brought together to form the final COS. To make a COS useful for a condition that affects children and young people (CYP), it is very important to involve CYP with the condition themselves. Some studies described the general steps of creating a COS for CYP, but there is very little information about exactly how COS developers adjusted the methods at each stage so that CYP could take part. This study examines this aspect more closely. We looked at COS that have been created for CYP aged 0 to 18 years and gathered information on how developers adjusted the methods to include CYP and what challenges they faced. Where details were missing or unclear, we contacted the COS developer. Our results showed that one-third of the COS analyzed included CYP at different stages of COS development. Developers used approaches that helped CYP express what outcome mattered most to them, agree on priorities with other groups, and help develop better ways for CYP to take part in research. CYP were not included in COS development when they were young and when their condition made it difficult to participate. Our study shows that by making appropriate adjustments, CYP can be included in COS development, irrespective of age, communication differences, or the type of condition. We propose adaptations to refine current methods to make CYP participation easier and more effective.

Choosing Core Outcomes for Use in Clinical Trials in Ophthalmology: Perspectives from Three Ophthalmology Outcomes Working Groups

There is growing recognition that insufficient attention is paid to the outcomes measured and reported in clinical trials. Selection of outcomes is crucial to trials designed to compare the effects of different interventions. For the findings to influence policy and practice, the chosen outcomes need to be relevant to patients and the public, healthcare professionals and others making decisions about health care. Trials in a specific condition often report different outcomes, or address the same outcome in different ways. Inconsistency in reported outcomes causes well known problems for those who attempt to synthesise evidence, and many meta-analyses have to exclude key studies because relevant outcomes are not reported. Furthermore, the measured outcomes may not always be important to patients or health service users. Much could be gained if an agreed core outcome set (COS) of a minimum number of appropriate and important outcomes was measured and reported in all clinical trials in a specific condition. Key stakeholders, including patients, should be involved in establishing COS, to ensure consideration of appropriate outcomes. COS may encompass all stages or severities of a condition or may focus on a particular disease category. Likewise, a COS may be for use in trials of all treatment types or only trials of a particular intervention. The scope of a COS should be defined to identify the relevant health condition, population and types of interventions. The COMET Initiative (http://www.comet-initiative.org/) aims to foster and facilitate methodological research in the area of standardising outcomes, to develop much needed standards for methods of COS development and to develop and maintain a publically available internet-based resource to collate the knowledge base for COS development.

This paper aims to analyse the topic of the nourishment and its treatment in the Classical Historiography in two complementary aspects: literature and military strategy. Firstly, it is proposed a reading of the topos of the Greek frugality, a sign of identity of the Greek people, and especially of the Spartan culture, as a metaphor of the properly military ἀρeτή and ἐγκράτeια and, therefore, as a suitable elaboration to the historiographical genre itself. Secondly, we carry out a brief approach to the military diet as a reflection of this topos inside the procedure and guidelines imposed by the strategy to the discipline and the cohesion in the army.

The selection of appropriate outcomes is crucial when designing clinical trials in order to compare the effects of different interventions directly. For the findings to influence policy and practice, the outcomes need to be relevant and important to key stakeholders including patients and the public, health care professionals and others making decisions about health care. It is now widely acknowledged that insufficient attention has been paid to the choice of outcomes measured in clinical trials. Researchers are increasingly addressing this issue through the development and use of a core outcome set, an agreed standardised collection of outcomes which should be measured and reported, as a minimum, in all trials for a specific clinical area.Accumulating work in this area has identified the need for guidance on the development, implementation, evaluation and updating of core outcome sets. This Handbook, developed by the COMET Initiative, brings together current thinking and methodological research regarding those issues. We recommend a four-step process to develop a core outcome set. The aim is to update the contents of the Handbook as further research is identified.

On 20-21st May 2015, more than 150 people with an interest in core outcome sets (COS) gathered at the University of Calgary in Alberta, Canada for the fifth meeting of the COMET Initiative. This was the first annual COMET meeting in North America, and the COMET Management Group are grateful to Cochrane Canada for facilitating the meeting and an excellent joint session on the second day. As well as participants from Canada and the USA, people came from Australia, Brazil, Germany, Portugal and the UK. Over the next two days, the invited plenary talks were complemented by workshops, posters and contributed presentations. Theresa Radwell (Alberta Cancer Foundation) opened the meeting, welcoming all to Calgary and introducing the importance of engaging patients within research and outcome selection. Paula Williamson (COMET Management Group) then spoke about the COMET Initiative, emphasising that COMET is keen to avoid unnecessary duplication of effort and to facilitate the development of COS. The participants were then introduced to important methodological issues in COS through a series of presentations. John Marshall (St. Michael's Hospital, Toronto) provided a critical care perspective and highlighted that mortality is not always the most important outcome from a patient perspective. Amy Hoang-Kim (University of Toronto) presented a recommendation for a minimal set of core domains for use in distal radius fracture clinical practice and research. Moving on to nephrology, Jonathan Craig and Allison Tong (University of Sydney) overviewed existing standardized outcomes, with preliminary results showing how dialysis free time was the most important outcome to haemodialysis patients. One of the novel additions in COMET V was a panel discussion showing the importance of COS to different stakeholders. John Fletcher (Canadian Medical Association Journal) described the pros and cons of COS from an editor's perspective. Jordi Pardo (OMERACT) outlined the OMERACT process for developing a COS. Carole Legare (Health Canada) identified the problems seen by regulators because of inconsistency of safety reporting. John Marshall (Canadian Critical Care Trials Group) spoke about challenging issues faced by the critical care research community. Mike Clarke (COMET) brought all of this together by highlighting the resources that are available through COMET to assist in the development and evaluation of COS. The ensuing discussion highlighted the benefits of COS for journals, how stakeholder involvement and international harmonisation are essential to COS development, the need to consider barriers to uptake of COS for researchers, and recognition of the need for a COS that is in no way restrictive. The afternoon began with David Moher (Ottawa Hospital Research Institute) speaking about the EQUATOR network, which aims to maximise the value of research by improving conduct and reporting. David highlighted how the evaluation of reporting guidleines and COS is critical. He was followed by a series of presentations which centred around outcomes for paediatric trials. Zafira Bhaloo (University of Alberta) emphasised how the reporting of primary outcomes in pediatric trials is inadequate and encouraged higher standards for reporting and informed selection of outcomes and their measures. Michele Hamm (University of Alberta) discussed how the use of social media to identify patient-centred outcomes in child health did not result in broad reach as a stakeholder engagement strategy. Mufiza Kapadia (The Hospital for Sick Children) ended the session by stressing the importance of involving parents and children in COS development. Alongside the 17 posters that were available for viewing throughout the first day, four of the people who had submitted abstracts had been selected to give a contributed talk. These began with Carina Benstom (University Hospital RWTH Aachen) who highlighted how the problems caused by inconsistent outcome measures in clinical trials are hardly recognised. Chris Hylton (PaCER) spoke about improvements in the results of patient experiences and outcome analysis, from involving patient and community engagement researchers. Sally Crowe (Crowe Associates Ltd) continued the patient theme, by speaking about how workshops offer context and depth for talking about outcomes. The final contributed talk came from Thomas Kelley (International Consortium for Health Outcomes Measurement) who explained how ICHOM's mission is to define global standard sets of outcome measures that really matter to patients for the most relevant medical conditions. Mike Clarke (COMET Management Group) closed the first day with a presentation about COMET in Canada, which highlighted what Canada can do to increase the use of COS in research, for example by helping to persuade funders that COS should be used in research. The second day (21st May) was shared with Cochrane Canada and the opening of their Annual Symposium. The opening of the joint session was marked by memories of Dave Sackett and his contribution to evidence based healthcare. Following a minute's silence for the many friends, colleagues and admirers of Dave to remember him, the scientific session began with Kay Dickersin (John Hopkins University) highlighting how groundwork needs to be laid in subject areas where there are a lack of COS and how new methods need to be explored for developing COS. Mike presented findings from a survey of outcomes in Cochrane Reviews, showing the wide variation in outcomes and the lack of COS, at least up to 2013. Holger Schunemann (McMaster University) closed the plenary session by providing an overview of the GRADE evidence to decision frameworks. Following a break, the participants headed off for one of three COMET workshops. Paula Williamson led the first of these, which focused on the methods for developing what to measure in COS. The workshop introduced methodological issues and considerations involved in developing COS. Workshop 2 was led by Mike Clarke and looked at how COS might be used for randomised trials and Cochrane Reviews. Bridget Young (University of Liverpool) led the third workshop providing an interactive opportunity for the participants to identify the challenges that researchers may encounter when planning to involve patients and carers in COS development. COMET V allowed a wide variety of stakeholders with an interest in COS development to meet and share experiences, findings, and plans with others. It brought together key scientists and consumers responsible for developing and implementing COS. Patient involvement emerged as a major focus of the meeting with an emphasis on engaging the relevant stakeholders early in the process of COS development. Thoughts were offered for how COMET can evolve both in Canada and the rest of the world. And, challenging questions were posed throughout the meeting, including: How can we ensure that COS are well developed in the first place? Is there a magic number of outcomes to be included in a COS and, if so, what is it? As COMET looks forward to COMET VI, it will seek to meet these challenges, guided by an International Advisory Group, which will include Peter Tugwell (University of Ottawa), one of the founders of OMERACT. The slides from COMET V presentations can be viewed at: http://www.comet-initiative.org/events/FifthCometMeeting.

The induction of labour is one of the most common procedures performed in obstetrics. As such, many clinical trials have been undertaken to try to optimise induction strategies and interventions. Unfortunately, the wide variation in the measuring and reporting of outcomes across induction studies—as well as studies involving different topics—has hampered the process of data synthesis and effective translation into clinical practice. In an effort to mitigate these challenges, there has been a growing call for the development and use of core outcome sets in the field of obstetrics. Core outcome sets that are determined to be important to each clinical question would diminish the heterogeneity between studies and also minimise the risk for reporting bias. This study (Dos Santos et al. BJOG 2018; 125:1673–80) proposes a core outcome set to be used in future clinical trials of the induction of labour. Using the Delphi method (Sinha et al. PLoS Med 2011;8(1): e1000393), the investigators utilised a systematic approach to the collation of core outcomes considered most clinically pertinent for women undergoing induction of labour. The investigators are to be commended for their efforts to use a structured multi-staged methodology that was determined a priori, and to involve multiple relevant stakeholders, including patient representatives who had some personal experience with induction of labour, across several countries. The final developed core outcome set included 28 outcome measures. This may simply reflect the conundrum of the multifaceted process of an induction of labour, and the consideration of outcomes for both pregnant women and their offspring. Such a large number of outcomes will undoubtedly pose significant challenges to future researchers, however, as they may be expected to compile and report, at a minimum, this entire set of outcomes. Furthermore, there is no proposed guideline for how each of the components within the proposed outcome set should be defined and/or measured. Although this may be outside of the scope of their work, this is an essential next step as there is likely to be significant variation in the measurement of these outcomes, which will be problematic. In addition, participants from some large high-income countries, such as the USA, as well as from middle- and low-income areas, were not well represented, and so the generalisability may be limited and hinder the widespread adoption of the proposed core outcome set. There is a growing awareness that the use of agreed-upon standardised outcome sets in clinical trials would improve research into women's health. The establishment of initiatives like The Core Outcomes in Women's and Newborn Health Initiative (www.crown-initiative.org) is encouraging. Although the development of core outcome sets is a challenge, the greater challenge will be the ‘buy-in’: the actual commitment to widespread implementation within the research community. For this to occur, proposed core outcome sets will need to be comprehensive yet feasible, specific yet generalisable, standardised but flexible, and most importantly, transparent and replicable. It is a high bar, but it is necessary if we are to truly advance our field. Dos Santos et al. take an important initial step in this quest. None declared. Completed disclosure of interests form available to view online as supporting information. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.

Core outcome sets (COS) are consensus-driven sets of minimum outcomes that should be measured and reported in all clinical trials. COS aim to reduce heterogeneity in outcome measurement and reporting, and selective outcome reporting. Implementing COS into clinical trials is challenging. Guidance to improve COS uptake in dermatology is lacking. To develop a structured practical guide to COS implementation. Members of the Harmonising Outcome Measurement for Eczema (HOME) executive committee developed an expert opinion-based roadmap founded on a combination of a review of the COS implementation literature, the Core Outcome Measures in Effectiveness Trials (COMET) initiative resources, input from HOME members and experience in COS development and clinical trials. The data review and input from HOME members was synthesized into themes, which guided roadmap development: (a) barriers and facilitators to COS uptake based on stakeholder awareness/engagement and COS features; and (b) key implementation science principles (assessment-driven, data-centred, priority-based and context-sensitive). The HOME implementation roadmap follows three stages. Firstly, the COS uptake scope and goals need to be defined. Secondly, during COS development, preparation for future implementation is supported by establishing the COS as a credible evidence-informed consensus by applying robust COS development methodology, engaging multiple stakeholders, fostering sustained and global engagement, emphasizing COS ease of use and universal applicability, and providing recommendations on COS use. Thirdly, incorporating completed COS into primary (trials) and secondary (reviews) research is an iterative process starting with mapping COS uptake and stakeholders' attitudes, followed by designing and carrying out targeted implementation projects. Main themes for implementation projects identified at HOME are stakeholder awareness/engagement; universal applicability for different populations; and improving ease-of-use by reducing administrative and study burden. Formal implementation frameworks can be used to identify implementation barriers/facilitators and to design implementation strategies. The effect of these strategies on uptake should be evaluated and implementation plans adjusted accordingly. COS can improve the quality and applicability of research and, so, clinical practice but can only succeed if used and reported consistently. The HOME implementation roadmap is an extension of the original HOME roadmap for COS development and provides a pragmatic framework to develop COS implementation strategies.

Core outcome set (COS) development and use enhances comparability of research findings. It may also enhance the translation of research into practice and reduce research waste. However, there is limited involvement of stakeholders from low- and middle-income countries (LMICs) in COS development and use. In this study, we explored the experiences of researchers in COS development projects who included stakeholders from LMICs. Online survey conducted in English of 70 COS developers from HICs who had included LMIC stakeholders in the process of developing a COS, published before the end of 2019. Respondents were identified from the COMET database and sent a link to the survey via a personalised email. Quantitative data were analysed using simple descriptive statistics. Qualitative data analysis was based on qualitative content analysis. There were 37 respondents yielding a 53% overall response rate. Analysis was limited to the responses related to 29 COS developed in the years 2015 to 2019, to reduce the potential for recall bias for earlier COS. Most respondents 20/29 (69%) were researchers. Determining 'what to measure' was reported as the most common stage of inclusion of LMIC stakeholders. Respondents cited (24/29, 83%) their ongoing collaborations with LMIC stakeholders such as clinicians or researchers as their main rationale for including LMICs stakeholders and reported that translation of the Delphi into languages other than English may be useful to enhance wider stakeholder participation. Involvement of LMIC stakeholders only in the later stages of COS development, lack of adequate resources to support their involvement, and lack of networks and contacts were thought to limit fuller participation of stakeholders from LMICs. To improve the involvement of LMIC stakeholders in COS development and use, COS developers need to raise awareness on the utility of COS. The need for and feasibility of translation into multiple languages warrants further discussion.

BackgroundThe selection of appropriate outcomes or domains is crucial when designing clinical trials, to appreciate the effects of different interventions, pool results, and make valid comparisons between trials. If the findings are to influence policy and practice, then the chosen outcomes need to be relevant and important to key stakeholders, including patients and the public, healthcare professionals and others making decisions about health care. There is a growing recognition that insufficient attention has been paid to the outcomes measured in clinical trials. Recent reviews of the measurement properties of patient-reported outcome measures for shoulder disorders revealed a large selection of diverse measures, many with questionable validity, reliability, and responsiveness. These issues could be addressed through the development and use of an agreed standardized collection of outcomes, known as a core outcome set (COS), which should be measured and reported in all trials of shoulder disorders. The purpose of the present project is to develop and disseminate a COS for clinical trials in shoulder disorders.Methods/DesignThe methods for the COS development will include 3 phases: (1) a comprehensive review of the core domains used in shoulder disorder trials; (2) an international Delphi study involving relevant stakeholders (patients, clinicians, scientists) to define which domains should be core; and (3) an international focus group informed by the evidence identified in phases 1 and 2, to determine which measurement instruments best measure the core domains and identification of any evidence gaps that require further empiric evidence.DiscussionThe aim of the current proposal is to convene several meetings of international experts and patients to develop a COS for clinical trials of shoulder disorders and to develop an implementation strategy to ensure rapid uptake of the core set of outcomes in clinical trials. There would be an expectation that the core set of outcomes would always be collected and reported, but it would not preclude use of additional outcomes in a particular trial.

BackgroundAs the development of core outcome sets (COS) increases, guidance for developing and reporting high-quality COS continues to evolve; however, a number of methodological uncertainties still remain. The objectives of this study were: (1) to explore the impact of including patient interviews in developing a COS, (2) to examine the impact of using a 5-point versus a 9-point rating scale during Delphi consensus methods on outcome selection and (3) to inform and contribute to COS development methodology by advancing the evidence base on COS development techniques.MethodsSemi-structured patient interviews and a nested randomised controlled parallel group trial as part of the Pelvic Girdle Pain Core Outcome Set project (PGP-COS). Patient interviews, as an adjunct to a systematic review of outcomes reported in previous studies, were undertaken to identify preliminary outcomes for including in a Delphi consensus survey. In the Delphi survey, participants were randomised (1:1) to a 5-point or 9-point rating scale for rating the importance of the list of preliminary outcomes.ResultsFour of the eight patient interview derived outcomes were included in the preliminary COS, however, none of these outcomes were included in the final PGP-COS. The 5-point rating scale resulted in twice as many outcomes reaching consensus after the 3-round Delphi survey compared to the 9-point scale. Consensus on all five outcomes included in the final PGP-COS was achieved by participants allocated the 5-point rating scale, whereas consensus on four of these was achieved by those using the 9-point scale.ConclusionsUsing patient interviews to identify preliminary outcomes as an adjunct to conducting a systematic review of outcomes measured in the literature did not appear to influence outcome selection in developing the COS in this study. The use of different rating scales in a Delphi survey, however, did appear to impact on outcome selection. The 5-point scale demonstrated greater congruency than the 9-point scale with the outcomes included in the final PGP-COS. Future research to substantiate our findings and to explore the impact of other rating scales on outcome selection during COS development, however, is warranted.

On two sunny