Good Research Practices Task Force Research Articles

A Guide to an Iterative Approach to Model-Based Decision Making in Health and Medicine: An Iterative Decision-Making Framework.

Decision makers frequently face decisions about optimal resource allocation. A model-based economic evaluation can be used to guide decision makers in their choices by systematically evaluating the magnitude of expected health effects and costs of decision options and by making trade-offs explicit. We provide a guide to an iterative approach to the medical decision-making process by following a coherent framework, and outline the overarching iterative steps of model-based decision making. We systematized the framework by performing three steps. First, we compiled the existing guidelines provided by the ISPOR-SMDM Modeling Good Research Practices Task Force, and the ISPOR ValueofInformation Task Force. Second, we identified other previous work related to frameworks and guidelines for model-based decision analyses through a literature search in PubMed. Third, we assessed the role of the evidence and iterative process in decision making and formalized key steps in a model-based decision-making framework. We provide guidance on an iterative approach to medical decision making by applying the compiled iterative model-based decision-making framework. The framework formally combines the decision problem conceptualization (Part I), the model conceptualization and development (Part II), and the process of model-based decision analysis (Part III). Following the overarching steps of the framework ensures compliance to the principles of evidence-based medicine and regular updates of the evidence, given that valueofinformation analysis represents an essential component of model-based decision analysis in the framework. Following the provided guide and the steps outlined in the framework can help inform various health care decisions, and therefore it has the potential to improve decision making.

Updated Recommendations on Evidence Needed to Support Measurement Comparability Among Modes of Data Collection for Patient-Reported Outcome Measures: A Good Practices Report of an ISPOR Task Force

The ISPOR Task Force on measurement comparability between modes of data collection for patient-reported outcome measures (PROMs) has updated the good practice recommendations from the 2009 ISPOR electronic patient-reported outcome and 2014 patient-reported outcome mixed modes Good Research Practices Task Force reports in light of accumulated evidence of measurement comparability among different modes of PROM data collection. Furthermore, with the increasing use of electronic formats of clinical outcome assessments in clinical trials and the US Food and Drug Administration’s encouragement of electronic data collection, this new task force report provides stakeholders with best practice recommendations reflecting the current body of evidence and enables them to respond to future developments in research and technology.This task force recommends an evidence-based approach to determine whether new research is needed to evaluate measurement comparability for a given questionnaire or technology. The suitability of existing evidence depends upon whether it satisfactorily demonstrates that the change in data collection mode has not affected the PROM’s measurement properties. In cases where sufficient evidence of measurement comparability exists and best practices for faithful migration are followed, this task force concludes that further testing of measurement comparability among the data collection modes is unnecessary, including cases of “mixing modes” within clinical trials such as bring your own device designs.

Stone clearance rate and postoperative recovery of mini percutaneous nephrolithotomy: A single-institute study

Introduction: During the last two decades, the evaluation and management of renal and upper ureteric stones have vastly altered. The era of endoscopic surgery replaced open pyelolithotomy or nephrolithotomy which caused significant morbidity. Earlier percutaneous nephrolithotomy (PCNL) replaced open surgery as a safe and effective treatment for renal stones. During this decade, we saw the evolution of endoscopic surgery where Mini PCNL (Mini Perc), Ultra Mini, and Micro Perc have further achieved stone clearance with better outcomes pertaining to reduced morbidity because of reduction in the caliber of tract dilatation, even avoiding a nephrostomy postprocedure in select cases. Aim: The aim of this study summarizes the outcome of 78 cases of patients undergoing Mini Perc. The primary objectives of the study are: 1. Postoperative evaluation of complications associated with Mini Perc by Modified Clavien–Dindo Grading 2. Postoperative pain assessment by visual analog scale 3. Stone clearance rate of the surgical procedure. Settings and Design: The research was carried out in the form of a hospital-based prospective observational study as per the guidelines shared by the Prospective Observational Clinical Studies Good Research Practices Task Force (formed May 16, 2010). Seventy-eight patients who underwent Mini Perc between August 2018 and May 2020 at tertiary care centers and who were vetted against the inclusion criteria were included in the study. Subjects and Methods: This was a hospital-based prospective observational study of 78 consecutive patients who underwent Mini Perc between August 2018 and May 2020 at a tertiary care center and were vetted against the criteria for inclusion and exclusion. A total of 78 patients who met the inclusion criteria were followed up in the study. The management policy for pelvicalyceal calculi at the study center has closely followed those of the American Urological Association (AUA) guidelines for managing renal stones. Results: The mean age of the group was 43.3 years (range: 16–84) with 47 males and 38 females. The mean stone size was 20.11 mm (range 11–8 mm) and the mean operative time was 44.5 min (range: 29–98 min). Double J (DJ) stent was placed in all patients as a prophylaxis to prevent obstruction/postoperative urinary leak. Postoperatively, three patients had bleeding from the operated site, two patients survived sepsis, and one patient each had postoperative fever and pleural injury. Pain assessment done at specified intervals showed a progressive decreasing trend in intensity as evaluated by visual analog scale. The stone clearance rate in our study was 96.15% at the end of 1 month following postoperative period. Conclusion: The study shows that Mini Perc remains the standard of care for the management of renal and upper ureteric stones. The technique beyond doubt is safe, efficient, feasible, and economical in achieving excellent stone clearance rates. The Modified Clavien–Dindo system of grading for perioperative complications is easy to use and reproducible. It can be used as an objective and reliable method for describing the complications of the surgical procedure.

Symptoms and Impacts Reported By Patients with Acute Myeloid Leukemia (AML) in Remission Post-Stem Cell Transplant

BackgroundLittle is known about the burden of symptoms and impacts on quality of life in patients with AML in the posttransplant period. As treatment options for AML increase, it is vital that the patient experience be considered when making treatment decisions. Qualitative methods, largely underutilized in hematologic malignancy research, help capture the patient voice and provide insights into signs, symptoms, and impacts that are most relevant to the patient experience and yield valid results with small sample sizes.Aim/ObjectiveTo gain a deeper understanding of the patient experience in the AML posttransplant period, we conducted patient interviews to identify the most bothersome symptoms and impacts on patients' lived experiences with AML.MethodsPatients diagnosed with AML and currently in remission ≥90 days posttransplant and <1 year from the transplant date were recruited for the study. The number of patients interviewed was estimated based on the projected number of patients needed to reach concept saturation (ie, the point at which no new symptoms or concepts are identified during patient interviews). An interview guide was developed that included a list of bothersome symptoms and life impacts that were identified during a prior study enrolling patients with relapsed/refractory AML and refined for the posttransplant population based on results from a literature review and discussions with clinicians. One-on-one concept elicitation interviews were conducted via telephone in 4 waves. Patient disease experiences were explored, particularly those symptoms and impacts most important to patients during the posttransplant period. Interviews lasted 60 to 90 minutes and were conducted in line with the International Society for Pharmacoeconomics and Outcomes Research Good Research Practices Task Force recommendations. Interviewers asked a series of open-ended questions to elicit spontaneous responses and included prompts to probe patient understanding and experiences in greater depth. When interviewed, patients were asked to rate their symptoms/impacts on a scale of 0 to 10, with 10 being the most bothersome/impactful. Salient symptoms and impacts were those reported by ≥50% of respondents as relevant either spontaneously or upon probing and rated most impactful (average rating of ≥5).ResultsTwenty patients who were in complete remission posttransplant were interviewed. Median age of patients was 59.5 years and 55% of patients were female. Seventeen patients were confirmed FLT3-mutation-positive. Symptom saturation was reached in wave 1 (42 total concepts), with 13 new symptoms identified. Impact saturation was reached in wave 2 (28 total concepts), with 12 new impacts identified. During interviews, patients mentioned both positive and negative impacts. Mean peak symptom scores are shown in Figure A. Salient symptoms identified were fatigue, weakness, nausea, pain, and diarrhea. The disturbance ratings for most salient symptoms decreased from the peak to the current rating, suggesting that symptoms were less bothersome over time. Mean peak impact scores are shown in Figure B. Salient impacts included 1 positive impact (life outlook) and 6 negative impacts (fear, decreased ability to maintain roles, anxiety, appetite loss, decreased ability to function, and change in appearance). Impacts were generally more stable over time than symptoms in terms of their disturbance to patients.ConclusionsBothersome symptoms and impacts continued in patients with AML in the posttransplant period. In general, most salient symptoms were less bothersome over time. Conversely, fear, anxiety, decreased ability to maintain familial roles, decreased ability to function, and change in appearance remained impactful during the posttransplant period, suggesting that fears/concerns about relapse and disease sequelae persist in the posttransplant period. Patients did report positive changes on life outlook, possibly reflecting completion of a stem cell transplant, a potentially curative therapy. The number of salient impacts suggests that while drugs prolong remission and decrease symptoms, emotional impacts continue and are central to the patient experience posttransplant. [Display omitted] DisclosuresCella: FACIT: Membership on an entity's Board of Directors or advisory committees. LeBlanc: Otsuka: Consultancy, Honoraria, Other; AbbVie: Consultancy, Honoraria, Other: Advisory board; Travel fees, Speakers Bureau; Agios: Consultancy, Honoraria, Other: Advisory board; Travel fees, Speakers Bureau; BMS/Celgene: Consultancy, Honoraria, Other: Travel fees, Research Funding, Speakers Bureau; Daiichi-Sankyo: Consultancy, Honoraria, Other: Advisory board; Duke University: Research Funding; NINR/NIH: Research Funding; Amgen: Consultancy, Other: travel; Jazz Pharmaceuticals: Research Funding; UpToDate: Patents & Royalties; Seattle Genetics: Consultancy, Other: Advisory board, Research Funding; Astellas: Consultancy, Honoraria, Other: Advisory board; Helsinn: Consultancy, Research Funding; AstraZeneca: Consultancy, Honoraria, Other: Advisory board, Research Funding; Heron: Consultancy, Honoraria, Other: advisory board; CareVive: Consultancy, Other, Research Funding; Flatiron: Consultancy, Other: Advisory board; Pfizer: Consultancy, Other: Advisory Board; American Cancer Society: Research Funding. Shah: Astellas Pharma, Inc.: Current Employment; University of Michigan School of Public Health Department of Health Management and Policy Alumni Board: Other: Chair-Elect. de la Motte: Astellas Pharma, Inc.: Consultancy. Toublan: Astellas Pharma, Inc.: Consultancy. Kelly: Astellas Pharma, Inc.: Consultancy.

Equivalence testing of a newly developed interviewer-led telephone script for the EORTC QLQ-C30

PurposeThe European Organisation for Research and Treatment of Cancer (EORTC) Quality of Life-Core Questionnaire (QLQ-C30) is a widely used generic self-report measure of health-related quality of life (HRQOL) for cancer patients. However, no validated voice script for interviewer-led telephone administration was previously available. The aim of this study was to develop a voice script for interviewer administration via telephone.MethodsFollowing guidelines from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) PRO Mixed Modes Good Research Practices Task Force, a randomised cross-over equivalence study, including cognitive debriefing, was conducted to assess equivalence between paper and telephone administration modes. Assuming an expected intraclass correlation coefficient (ICC) of 0.70 and a minimally acceptable level of 0.50, a sample size of 63 was required.ResultsCognitive interviews with five cancer patients found the voice script to be clear and understandable. Due to a protocol deviation in the first wave of testing, only 26 patients were available for analyses. A second wave of recruitment was conducted, adding 37 patients (n = 63; mean age 55.48; 65.1% female). Total ICCs for mode comparison ranged from 0.72 (nausea and vomiting, 95% CI 0.48–0.86) to 0.90 (global health status/QoL, 95% CI 0.80–0.95; pain, 95% CI 0.79–0.95; constipation, 95% CI 0.80–0.95). For paper first administration, all ICCs were above 0.70, except nausea and vomiting (ICC 0.55; 95% CI 0.24–0.76) and financial difficulties (ICC 0.60; 95% CI 0.31–0.79). For phone first administration, all ICCs were above 0.70.ConclusionsThe equivalence testing results support the voice script’s validity for administration of the QLQ-C30 via telephone.

Application of Discrete Choice Experiment in Health Care: A Bibliometric Analysis.

Background: Discrete choice experiment (DCE) as a tool that can measure medical stakeholders' preferences especially patients recently has been increasingly applied in health care.Objective: The aim of this study was to examine the hotspots and trends of the application of DCE in health care and to provide reference and direction for further development of DCE in the future.Method: A bibliometric method was implemented using the Web of Science (WoS) Core Collection for the period from the database established to December 8, 2020. The data files are imported into CiteSpace and Excel to analyze and visualize the annual volume of productive, authors, countries, cited journals, cited articles, and keywords.Results: A total of 1,811 articles were retrieved, then we read the abstract of each paper one by one, and 1,562 articles were included after screening, with an exponential increase in publication volume. John F. P. Bridges contributed to 40 publications and ranked first, followed by F. Reed Johnson (n = 37), Julie Ratcliffe (n = 36). The majority of the papers were conducted in the United States (n = 513) and the United Kingdom (n = 433). The top three cited journals were “Health Economics” (n = 981), “Value in Health” (n = 893), and “Pharmaceutical Economics” (n = 774), and the top three articles were “Constructing experimental designs for discrete-choice experiments: report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force,” “Conjoint analysis applications in health-a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force,” and “Discrete choice experiments in health economics: a review of the literature.” The research hotspots and trends included “health technology assessment,” “survival,” “preference based measure,” and “health state valuation.”Conclusion: The size of the literature about DCE studies in health care showed a noticeable increase in the past decade. The application of DCE in health care remains in an early growth phase, and “health technology assessment,” “survival,” “preference based measure,” and “health state valuation” reflected the latest research hotpots and future trends.

The Role of Patients in Alopecia Areata Endpoint Development: Understanding Physical Signs and Symptoms.

Meaningful patient input to understand disease experience and patient expectations for improvement with treatment is essential for the selection and development of outcome measures for alopecia areata (AA) clinical trials. This study explored the physical signs and symptoms of AA through 30 semistructured interviews with adult (n= 25) and adolescent (n= 5) patients experienced with severe or very severe AA. Scalp hair loss was overwhelmingly the most important sign and symptom of AA. Nearly all patients (90%) considered scalp hair loss in their top three most bothersome physical signs and symptoms of AA, with 77% (n= 23) naming scalp hair loss as the most bothersome symptom. Other identified signs and symptoms in the top three most bothersome included eyebrow, eyelash, nose, body, and facial hair loss, as well as eye irritation and nail damage and/or appearance. Eyebrow (16%, n= 4), eyelash (4%, n= 1), nasal (4%, n= 1), and body (4%, n= 1) hair loss were identified by seven adult patients as the most bothersome signs and symptoms of AA. Conceptual saturation confirmed that a comprehensive understanding of this patient population's physical AA-related signs and symptoms was obtained. These findings indicate that the primary objective for new AA treatments for this patient population should be meaningful improvement in scalp hair growth to address the most troubling unmet need.

Measurement Equivalence of Patient-Reported Outcome Measures Migrated to Electronic Formats: A Review of Evidence and Recommendations for Clinical Trials and Bring Your Own Device.

A growing number of clinical trials employ electronic media, in particular smartphones and tablets, to collect patient-reported outcome data. This is driven by the ubiquity of the technology, and an increased awareness of associated improvements in data integrity, quality and timeliness. Despite this, there remains a lingering question relating to the measurement equivalence of an instrument when migrated from paper to a screen-based format. As a result, researchers often must provide evidence demonstrating the measurement equivalence of paper and electronic versions, such as that recommended by the ISPOR ePRO Good Research Practices Task Force. In the last decade, a considerable body of work has emerged that overwhelmingly supports the measurement equivalence of instruments using screen-based electronic formats. Our review of key works derives recommendations on evidence needed to support electronic implementation. We recommend application of best practice recommendations is sufficient to conclude measurement equivalence with paper PROMs. In addition, we recommend that previous usability evidence in a representative group is sufficient, as opposed to per-study testing. Further, we conclude that this also applies to studies using multiple screen-based devices, including bring-your-own-device, if a minimum device specification can be ensured and the instrument is composed of standard response scale types.

PSY1 CONCEPTUALIZING A FRAMEWORK FOR MODELING SYSTEMIC LUPUS ERYTHEMATOSUS

Systemic lupus erythematosus (SLE) has a complex and heterogeneous natural history. Our aim was to develop a conceptual model of SLE that characterizes the relationships between short- and long-term outcomes that could be applied in future economic evaluations and health technology assessments of new therapies. As per the ISPOR-SMDM Modeling Good Research Practices Task Force guidelines, a targeted literature review was conducted to identify any previously developed conceptual or economic models in SLE. A steering committee of clinical and economic experts was convened to review existing model frameworks and brainstorm a new conceptual model, building on revision and refinement of prior models. The relationships between patient characteristics, disease activity, commonly used treatments, organ damage, health-related quality of life (HRQoL), and mortality were all considered by the committee, as well as the likely data available to parameterize the model in the future. Based on the existing literature and consensus among committee members, the key components of the conceptual model included disease activity, corticosteroid use, and organ damage by individual system. Relationships not present in prior models included stratifying short-term disease activity by level (e.g., low, moderate, high)and corticosteroid use influencing subsequent disease activity. Higher disease activity levels (i.e., flare) would trigger higher short-term corticosteroid use and increased risk of long-term organ damage. Higher doses of corticosteroids should decrease disease activity in the short-term, while raising the risk of long-term organ damage. Relevant covariates identified included demographics, age at onset, cardiovascular comorbidities, baseline organ damage, history of lupus nephritis, and anti-malarial/other immunosuppressant use. Long-term outcomes include organ damage, mortality, HRQoL, and costs. By linking short- and long-term outcomes, this conceptual model provides a foundation on which to conduct future economic evaluations of new SLE therapies. The next step is to conduct a model validation using real world data.

Estimating Health-State Utility for Economic Models in Clinical Studies: An ISPOR Good Research Practices Task Force Report

Cost-utility models are increasingly used in many countries to establish whether the cost of a new intervention can be justified in terms of health benefits. Health-state utility (HSU) estimates (the preference for a given state of health on a cardinal scale where 0 represents dead and 1 represents full health) are typically among the most important and uncertain data inputs in cost-utility models. Clinical trials represent an important opportunity for the collection of health-utility data. However, trials designed primarily to evaluate efficacy and safety often present challenges to the optimal collection of HSU estimates for economic models. Careful planning is needed to determine which of the HSU estimates may be measured in planned trials; to establish the optimal methodology; and to plan any additional studies needed. This report aimed to provide a framework for researchers to plan the collection of health-utility data in clinical studies to provide high-quality HSU estimates for economic modeling. Recommendations are made for early planning of health-utility data collection within a research and development program; design of health-utility data collection during protocol development for a planned clinical trial; design of prospective and cross-sectional observational studies and alternative study types; and statistical analyses and reporting.

Development of the Galaxy Chronic Obstructive Pulmonary Disease (COPD) Model Using Data from ECLIPSE: Internal Validation of a Linked-Equations Cohort Model.

The recent joint International Society for Pharmacoeconomics and Outcomes Research / Society for Medical Decision Making Modeling Good Research Practices Task Force emphasized the importance of conceptualizing and validating models. We report a new model of chronic obstructive pulmonary disease (COPD) (part of the Galaxy project) founded on a conceptual model, implemented using a novel linked-equation approach, and internally validated. An expert panel developed a conceptual model including causal relationships between disease attributes, progression, and final outcomes. Risk equations describing these relationships were estimated using data from the Evaluation of COPD Longitudinally to Identify Predictive Surrogate Endpoints (ECLIPSE) study, with costs estimated from the TOwards a Revolution in COPD Health (TORCH) study. Implementation as a linked-equation model enabled direct estimation of health service costs and quality-adjusted life years (QALYs) for COPD patients over their lifetimes. Internal validation compared 3 years of predicted cohort experience with ECLIPSE results. At 3 years, the Galaxy COPD model predictions of annual exacerbation rate and annual decline in forced expiratory volume in 1 second fell within the ECLIPSE data confidence limits, although 3-year overall survival was outside the observed confidence limits. Projections of the risk equations over time permitted extrapolation to patient lifetimes. Averaging the predicted cost/QALY outcomes for the different patients within the ECLIPSE cohort gives an estimated lifetime cost of £25,214 (undiscounted)/£20,318 (discounted) and lifetime QALYs of 6.45 (undiscounted/5.24 [discounted]) per ECLIPSE patient. A new form of model for COPD was conceptualized, implemented, and internally validated, based on a series of linked equations using epidemiological data (ECLIPSE) and cost data (TORCH). This Galaxy model predicts COPD outcomes from treatment effects on disease attributes such as lung function, exacerbations, symptoms, or exercise capacity; further external validation is required.

Value of Information in Asia: Concepts, Current Use, and Future Directions

Health technology assessment is a form of health policy research that provides policymakers with information relevant to decisions about policy alternatives. Findings from cost-effectiveness analysis (CEA) are one of the important aspects of health technology assessment. Nevertheless, the more advanced method of value of information (VOI), which is recommended by the International Society for Pharmacoeconomics and Outcomes Research and Society for Medical Decision Making Modeling Good Research Practices Task Force, has rarely been applied in CEA studies in Asia. The lack of VOI in Asian CEA studies may be due to limited understanding of VOI methods and what VOI can and cannot help policy decision makers accomplish. This concept article offers audiences a practical primer in understanding the calculation, presentation, and policy implications of VOI. In addition, it provides a rapid survey of health technology assessment guidelines and literature related to VOI in Asia and discusses the future directions of VOI use in Asia and its potential barriers. This article will enable health economists, outcomes researchers, and policymakers in Asia to better understand the importance of VOI analysis and its implications, leading to the appropriate use of VOI in Asia.

Translation and psychometric properties of the Persian version of self-management of type 1 diabetes for adolescents.

The present study was conducted to translate and assess the validation of the measure of self-management of type 1 diabetes for adolescents that developed by Schilling et al [Schilling LS, Knafl KA, Grey M. Changing patterns of self-management in youth with type I diabetes. J Pediatr Nurs 2006;21:412-24]. The first stage of the study involved the translation of the measure of self-management of type 1 diabetes for adolescents into Persian based on the model proposed by Wild et al. in two versions [Wild D, Grove A, Martin M, Eremenco S, McElroy S, et al. Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: report of the ISPOR task force for translation and cultural adaptation. Value Health 2005;8:94-104; Wild D, Eremenco S, Mear I, Martin M, Houchin C, et al. Multinational trials - recommendations on the translations required, approaches to using the same language in different countries, and the approaches to support pooling the data: the ISPOR patient-reported outcomes translation and linguistic validation good research practices task force report. Value Health 2009;12:430-40]. The translated versions were reviewed in consultation sessions with experts and the more appropriate items were selected and the final version was translated back into English in two versions and was then sent to the measure's designer for confirmation. The content validity of the measure was then evaluated by a group of experts and found to be acceptable. The next stage evaluated the measure's construct validity. This measure contains 52 items classified under five subscales. The confirmatory factor analysis was performed to assess the measure's construct validity and was found to be acceptable. For evaluating the reliability of the measure, its internal consistency was assessed through calculating its Cronbach's alpha and intra-class correlation. The measure's consistency was measured through calculating its test-retest reliability. The assessment of the measure's content validity revealed a content validity index of 0.98. For the construct validity of the measure using the confirmatory factor analysis, the following figures were obtained: NFI=0.97, RMSA=0.001, AGFI=0.81, IFI=0.833, GFI=0.83. In the assessment of the measure's reliability, the intra-class correlation showed an overall Cronbach's alpha of 0.88. The test-retest showed a consistency of 0.73 for the measure. The validation of the 48-item measure revealed that it can be used in the population of Iranian adolescents with type 1 diabetes (8 items changed their subscales and 4 items were removed).

Psychometric evaluation of the COPD assessment test: Data from the BREATHE study in the Middle East and North Africa region

The objective of this study was to assess the validity and performance of the Arabic and Turkish versions of the COPD Assessment Test (CAT) for evaluating the severity and impact of COPD symptoms. The data were obtained from the BREATHE study in the Middle East and North Africa region, a large general population survey of COPD conducted in ten countries of the region (Algeria, Egypt, Jordan, Lebanon, Morocco, Saudi Arabia, Syria, Tunisia, Turkey and United Arab Emirates), using a standardised methodology. A total of 62,086 subjects were screened, of whom a random sample of 5,681 subjects were administered the CAT by telephone. 5,639 evaluable questionnaires were recovered, representing a completion rate of 99%. In addition, the CAT was administered to an additional 833 subjects fulfilling the epidemiological diagnostic criteria for COPD. Mean scores in the general population were 6.99 ± 6.91 for the Arabic version and 9.88 ± 9.04 for the Turkish version. In patients with COPD, mean scores were 16.2 ± 9.1 and 20.9 ± 10.2 respectively. Scores were consistently higher in smokers than in non-smokers. In the general population, the proportion of respondents fulfilling criteria for COPD rose with higher CAT scores, and particularly above the 80th percentile, where 63% of COPD cases were to be found. This suggests that the CAT may be useful as a case-finding tool in the general population. In the COPD population, healthcare resource consumption rose linearly with CAT score above a threshold score of twenty, arguing in favour of the good criterion validity of the CAT. The internal consistency of the CAT was high (Cronbach's α 0.85 for the Arabic and 0.86 for the Turkish versions) and the factorial structure was unidimensional. In conclusion, this study performed in Arabic and Turkish speaking populations confirms the utility and validity of the CAT as a simple tool to collect data on the severity and impact of COPD symptoms, and suggests that it may potentially be useful as a case-finding tool to identify people at risk for COPD in the general population.

Transparency and Reproducible Research in Modeling

Decision modeling and cost effectiveness analysis have become important tools to inform clinical decision making and policy development, playing roles in policy decisions for human immunodeficiency virus and breast and colon cancer screening, as well as clinical decisions in the prevention of cardiac disease, for example. The establishment and use of best practices for model development, validation, and reporting are key steps in ensuring that model users have information they can trust. In this issue of Medical Decision Making, Eddy and others describe practices for model transparency and validation recently developed by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR)–Society for Medical Decision Making (SMDM) Modeling Good Research Practices Task Force. The authors recommend that modelers 1) provide a freely accessible nontechnical description of the model (VII.1), including a description of the external validation methods and results (VII.1 and VII.6) and verification (internal validation) methods (VII.4); 2) make available (openly or under licensing agreements) sufficiently technical documentation such that expert readers can evaluate and potentially reproduce the model (VII.2); and 3) assess face (VII.3), internal (verification) (VII.4), and external validity (VII.5–VII.11). Previous guidelines and recommendations have discussed the importance of transparency and validation, urging developers to clearly report model structure, data, equations, and assumptions, as well as methods to validate or check model consistency. However, these new ISPOR-SMDM recommendations set a new standard by encouraging the sharing of technical details, including code. The guidelines arrive at a time of heightened interest in transparency in research communities. Earlier this year, in a report investigating difficulties in evaluating and reproducing key translational omics findings that were used for treatment choice in later clinical trials (and that were later retracted), the Institute of Medicine issued a call for greater transparency and sharing of data and code within translational omics and in the broader research community. Eddy and colleagues appropriately note that concerns about intellectual property, model misuse, and costs may limit sharing of model details. The authors also suggest that if a model is accurate (validated), transparency may be of lesser importance: ‘‘Ultimately, what matters most is whether a model accurately predicts what occurs in reality.’’ However, accuracy does not diminish the need for transparency, and there may be ways to address the identified challenges. First, an accurate model still needs to be transparent. What degree of accuracy is sufficient to eliminate the need for transparency? Which are the exact data against which predictions should be measured? Both are matters of judgment. And, if prospective validation (predictive validation) is the highest standard for validation (as implied by these good practice recommendations), this information may only be available years after publication, limiting its usefulness at the time of decision making. Finally, too narrow a focus on predictive accuracy may unnecessarily discount findings from a model that cannot be thoroughly validated in this manner. The primary purpose of decision From VA Palo Alto Health Care System, Palo Alto, California, and Stanford School of Medicine, Stanford, California.

Prospective Observational Studies to Assess Comparative Effectiveness: The ISPOR Good Research Practices Task Force Report

ObjectiveIn both the United States and Europe there has been an increased interest in using comparative effectiveness research of interventions to inform health policy decisions. Prospective observational studies will undoubtedly be conducted with increased frequency to assess the comparative effectiveness of different treatments, including as a tool for “coverage with evidence development,” “risk-sharing contracting,” or key element in a “learning health-care system.” The principle alternatives for comparative effectiveness research include retrospective observational studies, prospective observational studies, randomized clinical trials, and naturalistic (“pragmatic”) randomized clinical trials. MethodsThis report details the recommendations of a Good Research Practice Task Force on Prospective Observational Studies for comparative effectiveness research. Key issues discussed include how to decide when to do a prospective observational study in light of its advantages and disadvantages with respect to alternatives, and the report summarizes the challenges and approaches to the appropriate design, analysis, and execution of prospective observational studies to make them most valuable and relevant to health-care decision makers. RecommendationsThe task force emphasizes the need for precision and clarity in specifying the key policy questions to be addressed and that studies should be designed with a goal of drawing causal inferences whenever possible. If a study is being performed to support a policy decision, then it should be designed as hypothesis testing—this requires drafting a protocol as if subjects were to be randomized and that investigators clearly state the purpose or main hypotheses, define the treatment groups and outcomes, identify all measured and unmeasured confounders, and specify the primary analyses and required sample size. Separate from analytic and statistical approaches, study design choices may strengthen the ability to address potential biases and confounding in prospective observational studies. The use of inception cohorts, new user designs, multiple comparator groups, matching designs, and assessment of outcomes thought not to be impacted by the therapies being compared are several strategies that should be given strong consideration recognizing that there may be feasibility constraints. The reasoning behind all study design and analytic choices should be transparent and explained in study protocol. Execution of prospective observational studies is as important as their design and analysis in ensuring that results are valuable and relevant, especially capturing the target population of interest, having reasonably complete and nondifferential follow-up. Similar to the concept of the importance of declaring a prespecified hypothesis, we believe that the credibility of many prospective observational studies would be enhanced by their registration on appropriate publicly accessible sites (e.g., clinicaltrials.gov and encepp.eu) in advance of their execution.

Content Validity—Establishing and Reporting the Evidence in Newly Developed Patient-Reported Outcomes (PRO) Instruments for Medical Product Evaluation: ISPOR PRO Good Research Practices Task Force Report: Part 1—Eliciting Concepts for a New PRO Instrument

The importance of content validity in developing patient reported outcomes (PRO) instruments is stressed by both the US Food and Drug Administration and the European Medicines Agency. Content validity is the extent to which an instrument measures the important aspects of concepts that developers or users purport it to assess. A PRO instrument measures the concepts most significant and relevant to a patient's condition and its treatment. For PRO instruments, items and domains as reflected in the scores of an instrument should be important to the target population and comprehensive with respect to patient concerns. Documentation of target population input in item generation, as well as evaluation of patient understanding through cognitive interviewing, can provide the evidence for content validity. Developing content for, and assessing respondent understanding of, newly developed PRO instruments for medical product evaluation will be discussed in this two-part ISPOR PRO Good Research Practices Task Force Report. Topics include the methods for generating items, documenting item development, coding of qualitative data from item generation, cognitive interviewing, and tracking item development through the various stages of research and preparing this tracking for submission to regulatory agencies. Part 1 covers elicitation of key concepts using qualitative focus groups and/or interviews to inform content and structure of a new PRO instrument. Part 2 covers the instrument development process, the assessment of patient understanding of the draft instrument using cognitive interviews and steps for instrument revision. The two parts are meant to be read together. They are intended to offer suggestions for good practices in planning, executing, and documenting qualitative studies that are used to support the content validity of PRO instruments to be used in medical product evaluation.

Content Validity—Establishing and Reporting the Evidence in Newly Developed Patient-Reported Outcomes (PRO) Instruments for Medical Product Evaluation: ISPOR PRO Good Research Practices Task Force Report: Part 2—Assessing Respondent Understanding

The importance of content validity in developing patient reported outcomes (PRO) instruments is stressed by both the US Food and Drug Administration and the European Medicines Agency. Content validity is the extent to which an instrument measures the important aspects of concepts developers or users purport it to assess. A PRO instrument measures the concepts most relevant and important to a patient's condition and its treatment. For PRO instruments, items and domains as reflected in the scores of an instrument should be important to the target population and comprehensive with respect to patient concerns. Documentation of target population input in item generation, as well as evaluation of patient understanding through cognitive interviewing, can provide the evidence for content validity. Part 1 of this task force report covers elicitation of key concepts using qualitative focus groups and/or interviews to inform content and structure of a new PRO instrument. Building on qualitative interviews and focus groups used to elicit concepts, cognitive interviews help developers craft items that can be understood by respondents in the target population and can ultimately confirm that the final instrument is appropriate, comprehensive, and understandable in the target population. Part 2 details: 1) the methods for conducting cognitive interviews that address patient understanding of items, instructions, and response options; and 2) the methods for tracking item development through the various stages of research and preparing this tracking for submission to regulatory agencies. The task force report's two parts are meant to be read together. They are intended to offer suggestions for good practice in planning, executing, and documenting qualitative studies that are used to support the content validity of PRO instruments to be used in medical product evaluation.

Interpreting Indirect Treatment Comparisons and Network Meta-Analysis for Health-Care Decision Making: Report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices: Part 1

Evidence-based health-care decision making requires comparisons of all relevant competing interventions. In the absence of randomized, controlled trials involving a direct comparison of all treatments of interest, indirect treatment comparisons and network meta-analysis provide useful evidence for judiciously selecting the best choice(s) of treatment. Mixed treatment comparisons, a special case of network meta-analysis, combine direct and indirect evidence for particular pairwise comparisons, thereby synthesizing a greater share of the available evidence than a traditional meta-analysis. This report from the ISPOR Indirect Treatment Comparisons Good Research Practices Task Force provides guidance on the interpretation of indirect treatment comparisons and network meta-analysis to assist policymakers and health-care professionals in using its findings for decision making. We start with an overview of how networks of randomized, controlled trials allow multiple treatment comparisons of competing interventions. Next, an introduction to the synthesis of the available evidence with a focus on terminology, assumptions, validity, and statistical methods is provided, followed by advice on critically reviewing and interpreting an indirect treatment comparison or network meta-analysis to inform decision making. We finish with a discussion of what to do if there are no direct or indirect treatment comparisons of randomized, controlled trials possible and a health-care decision still needs to be made.

Conducting Indirect-Treatment-Comparison and Network-Meta-Analysis Studies: Report of the ISPOR Task Force on Indirect Treatment Comparisons Good Research Practices—Part 2

Evidence-based health care decision making requires comparison of all relevant competing interventions. In the absence of randomized controlled trials involving a direct comparison of all treatments of interest, indirect treatment comparisons and network meta-analysis provide useful evidence for judiciously selecting the best treatment(s). Mixed treatment comparisons, a special case of network meta-analysis, combine direct evidence and indirect evidence for particular pairwise comparisons, thereby synthesizing a greater share of the available evidence than traditional meta-analysis. This report from the International Society for Pharmacoeconomics and Outcomes Research Indirect Treatment Comparisons Good Research Practices Task Force provides guidance on technical aspects of conducting network meta-analyses (our use of this term includes most methods that involve meta-analysis in the context of a network of evidence). We start with a discussion of strategies for developing networks of evidence. Next we briefly review assumptions of network meta-analysis. Then we focus on the statistical analysis of the data: objectives, models (fixed-effects and random-effects), frequentist versus Bayesian approaches, and model validation. A checklist highlights key components of network meta-analysis, and substantial examples illustrate indirect treatment comparisons (both frequentist and Bayesian approaches) and network meta-analysis. A further section discusses eight key areas for future research.