International Society For Pharmacoeconomics And Outcomes Research Research Articles

Artículo Traducido] Traducción, adaptación cultural y validación de la versión española del Teenagers’ Quality of Life (T-QoL): cuestionario para adolescentes con enfermedades de la piel

BackgroundTeenagers’ Quality of Life (T-QoL) is an age-specific measure to assess QoL of teenagers suffering from different skin diseases. A validated Spanish language version is lacking. We present the translation, cultural adaptation and validation of the T-QoL into Spanish. MethodsA prospective study with 133 patients (between 12 and 19 years old), attended at the dermatology department of Toledo University Hospital, Spain (September 2019–May 2020), was carried out for the validation study. The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) guidelines were used for the translation and cultural adaptation. We evaluated the convergent validity with the Dermatology Life Quality Index (DLQI), the Children's Dermatology Life Quality Index (CDLQI) and with a Global Question (GQ) on self-assessed disease severity. We also analysed internal consistency and reliability of the T-QoL tool and confirmed its structure with a factor analysis. ResultsGlobal T-QoL scores significantly correlated with the DLQI and the CDLQI (r = 0.75) and with the GQ (r = 0.63). The confirmatory factor analysis showed optimal fit for the bi-factor model and an adequate fit for the correlated three-factor model. Reliability indicators were high (Cronbach's α = 0.89; Guttman's Lambda 6 index = 0.91; Omega ω = 0.91) and test–retest showed a high stability (ICC = 0.85). The results were consistent with those found by the authors of the original test. ConclusionOur Spanish version of the T-QoL tool is valid and reliable to assess QoL of Spanish-speaking adolescents with skin diseases.

German translation of the PROMIS® pediatric anxiety, anger, depressive symptoms, fatigue, pain interference and peer relationships item banks

BackgroundThe present study aimed at the translation and cross-cultural adaptation of six PROMIS® pediatric self- and proxy- item banks and short forms to universal German: anxiety (ANX), anger (ANG), depressive symptoms (DEP), Fatigue (FAT), pain interference (P) and peer relationships (PR).MethodsUsing standardized methodology approved by the PROMIS Statistical Center and in line with recommendations of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) PRO Translation Task Force, two translators for each German-speaking country (Germany, Austria, and Switzerland) commented on and rated the translation difficulty and provided forward translations, followed by a review and reconciliation phase. An independent translator performed back translations, which were reviewed and harmonized. The items were tested in cognitive interviews with 58 children and adolescents from Germany (16), Austria (22), and Switzerland (20) for the self-report and 42 parents and other caregivers (Germany (12), Austria (17), and Switzerland (13)) for the proxy-report.ResultsTranslators rated the translation difficulty of most items (95%) as easy or feasible. Pretesting showed that items of the universal German version were understood as they were intended, as only 14 out of 82 items of the self-report and 15 out of 82 items of the proxy-report versions required minor rewording. However, on average German translators rated the items more difficult to translate (M = 1.5, SD = 0.20) than the Austrian (M = 1.3, SD = 0.16) and the Swiss translators (M = 1.2, SD = 0.14) on a three-point Likert scale.ConclusionsThe translated German short forms are ready for use by researchers and clinicians (https://www.healthmeasures.net/search-view-measures).

A Budget Impact Model of Maintenance Treatment of Chronic Inflammatory Demyelinating Polyneuropathy with IgPro20 (Hizentra) Relative to Intravenous Immunoglobulin in the United States.

Chronic inflammatory demyelinating polyneuropathy (CIDP) is a rare, progressive autoimmune disease causing peripheral nervous system dysfunction. Guidelines recommend immunoglobulin (IG) therapy as an immunomodulatory agent in CIDP. Drawbacks and unmet needs with intravenous immunoglobulin (IVIG) include adverse effects and wear-off effects, along with the burden of administration based on site of care. Subcutaneous administration of Hizentra, a subcutaneous immunoglobulin (SCIG) reduces patient burden by allowing self-administration outside the hospital setting and has fewer adverse events (AEs). We aimed to compare the expected cost of treatment and the budget impact of Hizentra compared with IVIG for maintenance treatment of CIDP in the United States. A decision tree model was developed to estimate the expected budget impact of maintenance treatment with Hizentra for US stakeholders. The model adopts primarily a US integrated delivery network perspective and, secondarily, a commercial perspective over a 1-year time horizon. Pharmacy costs were based on a payment mix of average sales price (73%), wholesale acquisition cost (2%), and average wholesale price (25%). Costs in the model reflect 2022 US dollars. In accordance with the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) guidelines and recommendations for budget impact modeling, no discounting was performed. The PATH clinical study of Hizentra maintenance in CIDP was used to determine clinical inputs for relapse rates at initial assessment (24 weeks) and at 52 weeks for Hizentra. The ICE clinical study of Gamunex maintenance in CIDP was the basis of relapse rates for Gamunex (and other IVIGs). Literature-based estimates were obtained for infusion costs by site of care, costs of IVIG infusion-related complications, and significant IVIG AE rates. Hizentra AE rates from the US Hizentra prescribing information were assessed but were not included in the model as the AEs in CIDP were mild, easily treated, and self-limited. Sensitivity analyses and scenario analyses were conducted to evaluate variations from the base case. The model showed that a Hizentra starting dose of 0.2 g/kg is expected to result in annual cost savings of US$32,447 per patient compared with IVIG. For a hypothetical 25-million-member plan, the budget impact of a 10% market share shift from IVIG to Hizentra is expected to result in savings of US$2,296,235. This analysis projects that Hizentra is likely associated with favorable economic benefit compared with IVIG in managing CIDP.

HARPER — protocol template for real-world trials for evaluating treatment efficacy

The International Society of Pharmacoepidemiology (ISPE) and the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) have created a joint task force, including representatives of key international stakeholders, to create a coordinated protocol template for RWE studies that assess the effect of treatment and is designed to inform decision-making. The template is based on existing transparency efforts and includes the state-of-art ideas regarding the level of detail required to ensure reproducibility of the RWE study. The underlying principle was to achieve sufficient clarity regarding research data, design, analysis and implementation to achieve the three main objectives. Firstly, to help researchers carefully consider and document the choice and justification of the key parameters of the study, secondly, to facilitate decision-making by allowing reviewers to easily assess potential bias associated with the choice, and thirdly, to ensure research reproducibility.

Spanish Version of Teenagers’ Quality of Life (T-QoL) for Adolescents with Skin Diseases: Translation, Cultural Adaptation and Validation

BackgroundTeenagers’ Quality of Life (T-QoL) is an age-specific measure to assess QoL of teenagers suffering from different skin diseases. A validated Spanish language version is lacking. We present the translation, cultural adaptation and validation of the T-QoL into Spanish. MethodsA prospective study with 133 patients (between 12 and 19 years old), attended at the dermatology department of Toledo University Hospital, Spain (September 2019–May 2020), was carried out for the validation study. The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) guidelines were used for the translation and cultural adaptation. We evaluated the convergent validity with the Dermatology Life Quality Index (DLQI), the Children's Dermatology Life Quality Index (CDLQI) and with a Global Question (GQ) on self-assessed disease severity. We also analysed internal consistency and reliability of the T-QoL tool and confirmed its structure with a factor analysis. ResultsGlobal T-QoL scores significantly correlated with the DLQI and the CDLQI (r=0.75) and with the GQ (r=0.63). The confirmatory factor analysis showed optimal fit for the bi-factor model and an adequate fit for the correlated three-factor model. Reliability indicators were high (Cronbach's α=0.89; Guttman's Lambda 6 index=0.91; Omega ω=0.91) and test–retest showed a high stability (ICC=0.85). The results were consistent with those found by the authors of the original test. ConclusionOur Spanish version of the T-QoL tool is valid and reliable to assess QoL of Spanish-speaking adolescents with skin diseases.

Self-Efficacy to Manage Chronic Disease (SEMCD) scale: translation and evaluation of measurement properties for a swedish version

BackgroundReinforcing self-efficacy in patients is important in person-centered care; therefore, reliable and valid measures of a person’s self-efficacy is of clinical relevance. A questionnaire suitable for self-efficacy and patient engagement that is not limited to a particular condition is the Self-efficacy to Manage Chronic Disease (SEMCD). This study aims to evaluate the measurement properties of a Swedish translation of the SEMCD with a Rasch analysis.MethodsThe translation and cultural adaptation of the SEMCD was performed according to the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) recommendations. Self-reported data was collected from two cohorts: patients with pituitary tumors (n = 86) and patients on sick leave due to common mental disorders (n = 209). Measurement properties were evaluated with a Rasch analysis in RUMM2030.ResultsThe original six-item SEMCD did not fit to a unidimensional scale. Two items, item 5 and item 6, deviated both statistically and conceptually and were removed. A four-item solution, the SEMCD-4 with collapsed thresholds for mid-range response options, showed good targeting and unidimensionality, no item misfit, and a reliability of 0.83.ConclusionIn a Swedish context with a mix of patients with pituitary tumors or common mental disorders, SEMCD-4 showed satisfactory measurement properties. Thus, SEMCD-4 could be used to identify patient self-efficacy in long-term illnesses. This knowledge about patient self-efficacy may be of importance to tailor person-centered support based on each patient´s resources, needs and goals.

Cross-cultural adaptation of the CRS-PRO questionnaire into French

BackgroundChronic rhinosinusitis (CRS), encompasses many different clinical patterns with variable response to treatment. Precise criteria specifying disease severity and control are lacking in the current literature. Our aim was to perform a cross-cultural adaptation of the CRS-PRO, creating a French version for use as a routine questionnaire in the assessment of patients with CRS.MethodsThe CRS-PRO questionnaire was translated according to the recommendations of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) through a three-step procedure including a backward translation.ResultsSeven of 12 items were initially discordant between the three translators before achieving consensus (Step 1). Two of 12 items were discordant between the backward translation and the initial CRS-PRO version regarding the word “mucus”(Step 2). Step 3 allowed the creation of a French proof-read version of the CRS-PRO questionnaire. Thirty patients were included for initial validation, mean age of 49.2 ± 15 years and 63.3% (19/30) male. It took them 67 ± 23 s to complete the questionnaire without any patients requiring more than 2 min.ConclusionThis study presents the French version of the CRS-PRO questionnaire—an adapted, validated, and well-accepted instrument to evaluate the CRS symptoms in the French speaking population.

OP41 Facilitating Dialogue Of Real-World Evidence Use In Health Technology Assessment: Taxonomy Of Question/Data Source Pairings To Support A Registry Of Studies

IntroductionThis paper reports the results of the collaboration within the European initiative of new Reimbursement and aCCess Approaches (EUreccA) which is concerned with the use of real-world evidence (RWE) in health technology assessment (HTA) decision-making. The work grew from the observation of a large, very experienced group of HTA practitioners which found that the use of RWE varied depending on the type of question asked and the particulars of the data source(s) used. We set out to examine how RWE is used in HTA decision-making and to make proposals on its facilitation.MethodsLiterature reviews covering earlier reviews of RWE use, academic papers, and HTA agency websites were combined with case studies involving interviews with decision-makers in four countries (England, France, Italy, Sweden) to identify the circumstances of breakdown of RWE use and to build a categorization of the uses of RWE and associated difficulties. This evidence supported the creation of a taxonomy of pairings of data sources and the questions they were used to address. The face validity of the approach was tested at an advisory board of senior HTA practitioners.ResultsIn total, 27 questions were identified and 10 types of data source, giving 270 pairings. These pairings were linked to relevant methods guidance and to examples of their use, itemizing HTA issues and decisions made. Reports are being prepared for publication, covering the detail of the methods of the literature searches; methods of the country case studies; a description of the taxonomy; and guidance on governance.ConclusionsWhen using RWE in HTA decision-making, the detail of the particular data sources and question addressed matter. Recently, both the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the Real-World Transparency Initiative have argued for a registry of the uses of RWE. The work described here offers a starting classification of the material that should be held in such a registry, and which in itself could be developed by the stakeholders, both agencies and companies, that use it, furthering trust and confidence.

Adaptation and Validation of the Spanish Version of Decisional Conflict Scale in People with Migraine in Spain.

We aimed to validate the Spanish version of the Decisional Conflict Scale (DCS) and analyze its psychometric properties in people with migraine. The DCS validation comprised two phases. First, a translation and cross-cultural adaptation following the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Good Practices: 1-preparation, 2-independent forward translation, 3-reconciliation, 4-back-translation, 5-harmonization, 6-clinical review, and 7-content validation in a group of migraine patients. Second, the analysis of the psychometric properties. The reliability or internal consistency of the DCS scale and subscales was assessed using Cronbach's α value. The item-subscale correlation was also evaluated. A floor and ceiling effects for DCS score was considered when at least 15% of respondents obtained DCS >90 (ceiling) or <10 (floor). The construct validity was studied through the correlation between the DCS subscales and by the correlation between the DCS and other questionnaires (Decision Self-Efficacy Scale [DSES] and 9-item Shared Decision-Making [SDM-Q-9]). Spearman's coefficients were estimated for the correlations. The cross-cultural adaptation was conducted on 17 patients who completed the questionnaire in a mean of 2.4 ±1.1 minutes. Generally, more than 75% of them considered that DCS items were adequate, easy to understand, and relevant. The psychometric properties were evaluated in a sample of n=128 patients. Accordingly, the internal consistency of DCS was high, with a Cronbach's α of 0.97 for the scale and between 0.87 to 0.96 for subscales. Also, a slight floor effect was observed, with 24.2% of patients having DCS scores <10. The correlation between subscales exceeded Spearman's coefficient of 0.7. Whereas the correlation between the DCS and the other questionnaires was generally moderate (Spearman's coefficient >0.4). The Spanish version (Spain) of the DCS has very acceptable psychometric properties (reliability and construct validity) and good potential for assessing decisional conflicts among migraine patients.

OP12 Findings From the 2022 International Society for Pharmacoeconomics and Outcomes Research Student Benefit Survey

The purpose of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) student benefit survey is to understand perceived benefits of student membership; identify suggested future membership benefits to improve student engagement; and identify drivers of a smooth transition from a student to a new professional member.

Validation of the French version of the ICHOM adult oral health standard set.

The ICHOM Adult Oral Health Standard Set (AOHSS) recently developed by the ICHOM Oral Adult Health Working Group is a standard set of outcomes designed for its collection in clinical practice in dental health. The outcome standard set is made up of clinical-reported outcome measures (CROMs) and patient-reported outcome measures (PROMs). The purpose of this study was to translate and cross-culturally adapt the PROM section of the Standard Set in French for France to enable comprehensive evaluation of the patients' oral health quality of life in a French population. The questionnaire was translated following the guidelines of the International Society for Pharmacoeconomics and Outcome Research (ISPOR). We included patients consulting in a dentistry clinic (n = 127) and seeking dental care. The PROM and CROM data were collected from all patients. Both reliability and the internal consistency were evaluated. The ICHOM AOHSS was successfully translated into French. We sampled and surveyed 126 patients in a dentistry clinic in France using the French translation of the ICHOM AOHSS. Cronbach's α was calculated to measure the internal consistency. The resulting Cronbach's α was 0.8, indicating acceptable homogeneity. The French version of the ICHOM AOHSS shows acceptable psychometric properties in terms of reliability and internal consistency. This translation is suitable for its implementation in a French-speaking patient population.

Value Insider Season 1 Episode 5: What Other Aspects of Value May Be Relevant? (Societal Impact) [Podcast

What other aspects of value may be relevant? In this episode of the Value Insider podcast, host Mike Chambers speaks with Prof. Lou Garrison about the societal perspective on value of healthcare interventions. Prof. Garrison is Professor Emeritus in The Comparative Health Outcomes, Policy, and Economics institute (CHOICE) in the School of Pharmacy at the University of Washington. Lou is past president of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and he is currently co-chair of its Policy Outlook Committee. Going beyond the "conventional" elements of value assessment, Prof. Garrison highlights which additional impacts may be at play. He shares his concept of the "Value Flower", explaining elements such as insurance value and scientific spill-over.

Formulary Submissions: Value Claims, Protocols and Outcomes Based Contracting in Rare Disease.

Outcomes based payments contracting is in its infancy. The increased attention being given to rare disease place a premium on the ability to engage with payers to ensure that there is an analytical framework relevant to value claims contracting. Rare disease is not, of course, alone; many other chronic disease states may be suitable candidates and have been over the past 10 years or more. Rare disease, however stands apart: (i) the evidence base at product launch is limited; (ii) the therapy costs are often considered prohibitive; and (iii) the target patient population is small. At the same time, those seeking to implement an evidence-based engagement with health systems to support innovative rare disease interventions face a substantive technology assessment barrier. The focus in health technology assessment on assumption driven modeled cost-effectiveness simulations that support imaginary recommendations for cost-effective pricing and access is, however, an avoidable barrier. In the US, this barrier is the business model of the Institute for Clinical and Economic Review (ICER) and one endorsed by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR). Rare disease can be better served with other tools at our disposal with a proposed new start analytical framework in health technology assessment. The purpose of this brief note is to make the case that this proposed new start focused on single attribute value claims that meet the standards of normal science and fundamental evidence can not only dispense with the ICER imaginary modeling but, with a new start formulary submission package, integrate value claims with assessment protocols to set the stage for effective outcome-based contracting as the default standard for future payer negotiations.

Value Insider Season 1 Episode 3: How Does Budget Impact and Affordability in Healthcare Work? (BI and Affordability) [Podcast

How does budget impact and affordability in healthcare work? In this episode of the Value Insider podcast, host Mike Chambers speaks with Prof. Sean Sullivan about affordability and budget impact for the "payers" of healthcare interventions. Prof. Sullivan is Dean of the University of Washington School of Pharmacy. He is past president of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and served as chair of the health technology assessment (HTA) committee of US Health Insurer Premera Blue Cross, was part of the US Governmental Medicare coverage evidence committee and led the ISPOR Task Force on Methods for Conducting and Reporting Budget Impact Assessments. Prof. Sullivan explains how budget impact and affordability are intertwined and how this plays a role in decisions in the US, but also the rest of the world.

LIMB-Q Kids—German Translation and Cultural Adaptation

(1) Purpose: Lower limb deformities can have a severe impact on health-related quality of life (HRQL). LIMB-Q Kids is a new patient-reported outcome measure (PROM) aiming to elucidate the experience of 8–18-year-old patients before, during and after treatment, and to measure the different aspects of HRQL. The aim of this study was to translate and culturally adapt LIMB-Q Kids to German. (2) Methods: The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) guidelines were followed. Three forward translations, a backward translation, an expert panel meeting with eight participants, and twenty cognitive debriefing interviews led to the final German version of LIMB-Q Kids. (3) Results: In the forward translations, 4/159 items were difficult to translate, and 2/159 items in the backward translation differed from the original English version. Cognitive debriefing interviews with 20 patients identified 7/159 items that were difficult to comprehend/answer, and 2 of these items were changed. (4) Conclusions: Lower limb deformities can have a great impact on children, and it is important to measure and consider the impact on HRQL. In order to be able to use PROMs in different countries, conceptually equivalent translations and cultural adaptations should be performed in order to ensure comprehensibility. The final German version of LIMB-Q Kids is ready for use in an international field test.

3rd Annual International Pharmacoeconomic Forum Highlights.

3rd Annual International Pharmacoeconomic Forum Highlights.

Dutch translation and cultural adaptation of the LYMPH-Q, a new patient-reported outcome measure for breast cancer-related lymphedema

BackgroundThe LYMPH-Q Upper Extremity module is a new patient-reported outcome measure (PROM) developed to assess patient outcomes of breast cancer-related arm lymphedema (BCRL). Content for the LYMPH-Q Upper Extremity Module was developed from the extensive input of patients and experts in the field of breast surgery and breast cancer-related lymphedema. Rasch Measurement Theory analysis was used to assess psychometric properties. The aim of this study was to perform a Dutch translation and cultural adaptation of the LYMPH-Q Upper Extremity Module.MethodsThe translation process was performed in accordance with the guidelines of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR).The process included two forward translations, two back translations, and cognitive debriefing interviews with patients with BCRL.ResultsComparison of the two forward translations showed that the translations for most items (n = 60; 88.2%) were conceptually consistent between the two translators. Translations of the remaining items were reviewed and discussed until consensus was reached. Three items in the back translation had a different meaning when compared to the original English version and required re-translation. The resultant Dutch version of the LYMPH-Q was tested in a series of cognitive debriefing interviews with seven patients and showed good content validity.ConclusionsThe translation and cultural adaptation process resulted in a conceptually equivalent Dutch version of the LYMPH-Q Upper Extremity Module. This new PROM can now be used in clinical practice and research settings to evaluate outcomes in patients with BCRL.Level of evidence: Not gradable

OP2 Findings from the 2021 International Society for Pharmacoeconomics and Outcomes Research Student Interest Survey

The International Society for Pharmacoeconomics and Outcomes Research (ISPOR) student interest survey is distributed annually to active student ISPOR members. In this study, we describe the findings from the 2021 student interest survey.

Transcultural adaptation and validation of the Spanish version of the Global Pain Scale.

The aim of this study was to perform a cross-cultural adaption and validation of the Global Pain Scale (GPS) to produce a Spanish-language version (GPS-Sp) and to determine the psychometric properties of this instrument. The GPS was cross-culturally translated into Spanish following the guidelines of the International Society for Pharmacoeconomics and Outcomes Research. The initial study population was composed of 384 patients recruited from February to May 2021. All participants were aged at least 18 years and were currently experiencing pain. All gave signed informed consent to take part and completed the Brief Inventory-Sp and GPS-Spain questionnaires. Cronbach'sɑand test/retest reliability values were calculated and floor/ceiling effects analyzed. Construct validity was assessed by confirmatory factor analysis (CFA). The 370 patients included in the final analysis presented the following characteristics: 36.2% were male and 63.8% were female; mean age 42.6 (19-88) years; mean body mass index 24.99. Internal consistency was good. The Cronbach's ɑfor GPS-Sp was 0.86 and the intraclass correlation coefficient was 0.94 (95% CI; 0.87-0.97). Five main explanatory factors were identified by CFA, which produced the following values: RMSEA = 0.057; CFI = 0.807; GFI = 0.809; NFI = 0.763. No floor/ceiling effect was observed. The GPS-Sp is a valid, reliable and sensitive instrument for assessing pain in a Spanish-speaking population and could facilitate pain relief in this population.

Nothing to Cheer About: Endorsing Imaginary Economic Evaluations and Value Claims with CHEERS 22

One of the more unfortunate features of health technology assessment is the tenacity with which leaders in the field and organizations such as the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the Institute for Clinical and Economic Review (ICER) cling to an evaluation framework that fails to meet the standards of normal science. Believers subscribe to a meme that is clearly non-science (metaphysics and pseudoscience) and one that should have been discarded over 30 years ago. Certainly, subscribing to an impossible belief is not unusual; indeed it may make the belief that much stronger. Yet the meme is non-sustainable; it is also pointless as the economic evaluation claims are non-evaluable. There is no acknowledgement of the standards of normal science or the limitations imposed by the axioms of fundamental measurement. The purpose of this commentary is to make the case that the recent release of the Consolidated Health Economic Evaluation Reporting Standards 2022 (CHEERS 22) checklist is misleading; CHEERS 22 fails to address the manifest deficiencies in the approach to economic evaluations endorsed by ISPOR and ICER. Instead, it continues to promote economic evaluations in healthcare that invent evidence and non-empirically evaluable value claims. Given the widespread publicity that has accompanied the release of CHEERS 22, the purpose of this commentary is to detail the deficiencies in CHEERS 22 and propose an alternative framework for economic evaluation in health care to meet the information needs of formulary committees. This means abandoning the standards for economic evaluations that have dominated health technology assessment for 30 years, notably the key role assigned to the mathematically impossible quality adjusted life year (QALY). The proposed new start recommends single attribute evaluable value claims that meet ratio or interval measurement standards and are supported by evaluation protocols.