Multi-item Assessments Research Articles

Leveraging Item Parameter Drift to Assess Transfer Effects in Vocabulary Learning

ABSTRACT Longitudinal models typically emphasize between-person predictors of change but ignore how growth varies within persons because each person contributes only one data point at each time. In contrast, modeling growth with multi-item assessments allows evaluation of how relative item performance may shift over time. While traditionally viewed as a nuisance under the label of “item parameter drift” (IPD), IPD may be of substantive interest if it reflects how learning manifests on different items or subscales at different rates. In this study, we apply the Explanatory Item Response Model to assess IPD in a causal inference context. Simulation results show that when IPD is not accounted for, both parameter estimates and standard errors can be affected. We illustrate with an empirical application to the persistence of transfer effects from a content literacy intervention , revealing how researchers can leverage IPD to achieve a more fine-grained understanding of how vocabulary learning develops over time.

Food Allergy-Related Bullying Among Children and Adolescents.

Food allergy (FA) management requires youth to avoid allergens and carry emergency medication which can impact participation in social activities. Previous research indicates that some youth experience FA-related bullying, but many studies are limited by single-item assessment methods and a narrow definition of bullying. This study describes FA-related bullying among a diverse cohort of youth with FA and evaluates parent-child disagreement and bullying assessment methods. Youth ages 9-15 years (n = 121) diagnosed with an IgE-mediated FA and their primary caregivers were recruited from pediatric FA clinics to complete surveys about their FA-related bullying experiences. Descriptive statistics were conducted to assess overall FA-related bullying and McNemar tests were utilized to assess disagreement among parent-child report and between multi-item and single-item assessment methods. Seventeen percent and 31% of youth reported FA-related bullying on single-item and multi-item assessments, respectively. Twelve percent of parents reported their child had experienced FA-related bullying. Youth reported overt physical (51%), overt non-physical (66%), and relational FA-related bullying (20%). FA-related bullying was most common among classmates. Assessment method significantly affected the rates of FA-related bullying reported by youth, and parents and youth only agreed on FA-bullying experiences when assessed via the single-item measure. A subset of a diverse sample of youth with FA reported FA-related bullying. Clinicians should use multi-item assessment methods and ask both parents and children about their experiences to fully capture the experiences of families managing FA. School policies that facilitate FA safety and social inclusion should be promoted. N/A.

P.323 Assessing the benefit-risk ratio of approved treatments for bipolar depression using likelihood to be helped or harmed (LHH) analyses

Rapid reduction of suicidal thoughts is critical for treating suicidal patients. Clinical trials evaluating these treatments require appropriate measurement. Key methodological issues include: 1) the use of single or multi-item assessments, and 2) evaluating whether suicidal ideation measures can track rapid change over time. The current study presents data from two randomized, placebo-controlled, crossover clinical trials evaluating ketamine in individuals with treatment-resistant depression (n = 60). Participants were assessed for suicidal thoughts using the Hamilton Depression Rating Scale (HAM-D), Montgomery-Asberg Depression Rating Scale (MADRS), Beck Depression Inventory (BDI), and Scale for Suicidal Ideation (SSI) at eight time points over three days. Assessments were compared using correlational analyses and effect sizes at 230 min and three days after ketamine infusion. Linear mixed models evaluated change in ideation across all time points. The HAM-D and MADRS suicide items demonstrated correlations of r > .80 with the first five items of the SSI (SSI5). On linear mixed models, an effect for ketamine was found for the HAM-D, MADRS, BDI items, and SSI5 (p < .001), but not for the full SSI (p = .88), which suggests a limited ability to assess change over time in patients with low levels of suicidal thoughts. Taken together, the results suggest that repeated suicidal assessments over minutes to days appear to detect improvement in suicidal thoughts after ketamine infusion compared to placebo. The MADRS suicide item, BDI suicide item, and SSI5 may be particularly sensitive to rapid changes in suicidal thoughts.

How to conduct a multi-item research assessment in bibliometric studies? Theoretical support and empirical evidence

Purpose – The purpose of this paper is to enrich the understanding of how to form a multi-item assessment and what approaches can be applied for researchers. Design/methodology/approach – This study proposes three ways to form a multi-item assessment (i.e. separate, formative and reflective approaches) and further makes a comparison between these three approaches. Findings – The results show that multi-item assessment, particularly for formative and reflective constructs, has greater explanatory power in the research model. Finally, this study provides a roadmap to guide future researchers’ decision strategy for selecting multi-item assessments. Originality/value – Due to the multi-faceted nature of research, using a single indicator to judge a scholar’s research performance will never reveal a multi-faceted picture and can easily result in measurement bias. In this vein, researchers should use different evaluation approaches and indicators to address various forms of research outcomes. However, prior studies rarely adopt multi-item scales to evaluate research performance and seldom discuss how to build a composite construct of research performance. This study aims to fill this research gap in the literature.

Assessing measures of suicidal ideation in clinical trials with a rapid-acting antidepressant

Rapid reduction of suicidal thoughts is critical for treating suicidal patients. Clinical trials evaluating these treatments require appropriate measurement. Key methodological issues include: 1) the use of single or multi-item assessments, and 2) evaluating whether suicidal ideation measures can track rapid change over time. The current study presents data from two randomized, placebo-controlled, crossover clinical trials evaluating ketamine in individuals with treatment-resistant depression (n = 60). Participants were assessed for suicidal thoughts using the Hamilton Depression Rating Scale (HAM-D), Montgomery-Asberg Depression Rating Scale (MADRS), Beck Depression Inventory (BDI), and Scale for Suicidal Ideation (SSI) at eight time points over three days. Assessments were compared using correlational analyses and effect sizes at 230 min and three days after ketamine infusion. Linear mixed models evaluated change in ideation across all time points. The HAM-D and MADRS suicide items demonstrated correlations of r > .80 with the first five items of the SSI (SSI5). On linear mixed models, an effect for ketamine was found for the HAM-D, MADRS, BDI items, and SSI5 (p < .001), but not for the full SSI (p = .88), which suggests a limited ability to assess change over time in patients with low levels of suicidal thoughts. Taken together, the results suggest that repeated suicidal assessments over minutes to days appear to detect improvement in suicidal thoughts after ketamine infusion compared to placebo. The MADRS suicide item, BDI suicide item, and SSI5 may be particularly sensitive to rapid changes in suicidal thoughts.

Just How Bad Negative Affect Is for Your Health Depends on Culture

Pressman, Lopez and Gallagher (2013) conclude that across the globe negative emotions are bad for one’s health. Yet, just how bad negative emotions are for health depends on culture. In U.S. American contexts, negative feelings are construed as the individual’s responsibility and as harmful. In Japanese contexts, negative feelings are construed as rooted in relationships and as natural. Using six clinically-relevant measures and two representative samples, we tested the hypothesis that negative affect is more strongly associated with poor health in the U.S. (n = 1,741) than in Japan (n = 988). Negative affect more strongly predicted poor health in the U.S. than in Japan for multi-item assessments of physical health (chronic conditions, physical functioning) and mental health (psychological well-being, self-esteem). There were no differences for single-item health assessments (life satisfaction, global health). These findings underscore the need for further theoretically-driven investigations of how cultural construals shape the emotion-health link.

Comparison of Health-Related Quality of Life Measurements Using a Single Value in Patients with Asthma and Chronic Obstructive Pulmonary Disease

Three methods have been developed to measure health-related quality of life (HRQoL) expressed as a single value: the global rating scale, the total score obtained from disease-specific instruments, and the preference-based utility index. We compared these different single HRQoL measurements in patients with asthma and chronic obstructive pulmonary disease (COPD). We recruited 167 patients with asthma and 161 patients with COPD. The global rating HRQoL was assessed by the Hyland scale. The total HRQoL was assessed by the Living With Asthma Questionnaire in asthma and the St. George's Respiratory Questionnaire in COPD. The Quality of Well-being (QWB) scale was used for the utility measurement derived from the Medical Outcome Study Short-form 36. The inter-relationships between these three HRQoL values were weak to moderate in asthma and moderate in COPD. In asthma, the Hyland scale was weakly correlated with the total HRQoL (Spearman's rank correlation coefficients [Rs] = −0.20) and moderately with the QWB score (Rs = −0.43). In the stepwise multiple regression analyses, anxiety on the Hospital Anxiety and Depression scale and the dyspnea score tended to correlate more significantly with the single HRQoL values in both asthma and COPD than physiological measurements such as the forced expiratory volume in one second. The Hyland scale was less correlated with existing parameters (cumulative coefficient determination [R2] = 0.04) than the total HRQoL (cumulative R2 = 0.47) and the QWB scale (cumulative R2 = 0.49) in asthma. The single HRQoL values from the Hyland scale, the total HRQoL and the QWB scale evaluated different aspects of asthma and COPD. The psychological status and dyspnea contributed more significantly to the single HRQoL values in these two disorders than the physiological measurements. In asthma, the Hyland scale was especially different from the other single HRQoL scales and should be evaluated separately from the multi-item HRQoL assessments.

Validation of Single-Item Linear Analog Scale Assessment of Quality of Life in Neuro-Oncology Patients

Assessment of patient quality of life (QOL) requires balancing the details provided by multi-item assessments with the reduced burden of single-item assessments. In this project, we investigated the psychometric properties of single-item Linear Analog Scale Assessments (LASAs) for patients with newly diagnosed high-grade gliomas. Measures included QOL LASAs (overall, physical, emotional, spiritual, intellectual), Symptom Distress Scale (SDS), Profile of Mood States (POMS; overall, confusion, fatigue), and Functional Assessment of Cancer Therapy-Brain (FACT-Br; overall, brain, physical, emotional). Associations of LASA measures with SDS, POMS, and FACT-Br domains and with Eastern Cooperative Oncology Group performance score (PS) and Mini-Mental State Examination (MMSE) were assessed. Repeated measures ANOVA models compared the change over time of LASAs and SDS, POMS, and FACT-Br. Two hundred five patients completed the assessments across three time points. To allow comparison across measures, all scores were converted to a scale of 0–100, with higher scores indicating better QOL. LASA mean scores ranged from 60 to 78; SDS, POMS, and FACT-Br ranged from 62 to 81. FACT-Br physical ( P < 0.001) and POMS fatigue subscale ( P = 0.005) decreased over time, as did LASA physical ( P = 0.08). LASA scales were strongly associated with corresponding scales on SDS, POMS, and FACT-Br (0.44 < rho < 0.65; P < 0.001). LASA was negatively associated with PS and positively with MMSE, with associations similar in magnitude to the other QOL and psychosocial measures. The data suggest that the single-item LASA scales are valid for assessing QOL of cancer patients and are an appropriate alternative when a shorter instrument is warranted.

What is the value added of patient reported outcomes relative to physician rated symptom assessments?

8580 Background: This pooled analysis examined the relationship between single-item and multiple-item quality of life (QOL) measures and assessed the agreement between changes in QOL and patient-reported adverse events (AE). Methods: Data from six lung cancer clinical trials involving 358 patients were pooled. All trials incorporated the UNISCALE and one of three multiple-item assessments: the Functional Assessment for Cancer Therapy-Lung, the Lung Cancer Symptom Scale, or the Symptom Distress Scale. A clinically significant decline (CSD) in QOL was defined as a 10-point drop from baseline. A CSD in AE was defined as a change from ≤ grade 2 at baseline to ≥ grade 3 post-baseline. Spearman rank correlations and Bland-Altman approach were used to assess agreement. The Kaplan-Meier method was used to evaluate the time to the first occurrence of a severe (grade 3+) AE and first CSD in QOL. Results: Correlations between the UNISCALE and multi-item assessments were modest (rho=0.49–0.66). At least one 10-point decline in the UNISCALE and multi-item assessments were reported by 58% and 40% respectively. At least one severe AE was reported in 44% of patients post-baseline. The percent agreement between experiencing a severe AE and a CSD in QOL was 49% and 59% for UNISCALE and the multi-item tools. For individual AE, the percent agreement between a CSD in AE and a CSD in QOL ranged from 37% to 50% for UNISCALE and 44% to 69% for the multi-item tools. The median time to the first CSD in QOL for UNISCALE and multi-item tools was 67 and 142 days respectively. The median time to the first occurrence of a severe AE was 304 days. Conclusions: QOL and AE provide quantifiably different information. A 10-point decline in QOL occurs earlier than CTC AE reporting. Patient reported outcomes may be more sensitive in identifying clinically relevant problems than physician rated symptom assessments. No significant financial relationships to disclose.

Validation of single item Linear Analogue Scale Assessments (LASAs) for assessing quality of life (QOL) in patients with newly diagnosed high-grade gliomas

8583 Background: Patient QOL assessment requires balancing the detail provided by multi-item assessments with the reduced burden of single-item assessments. We investigated the psychometric properties of single-item LASA assessments used in 3 North Central Cancer Treatment Group (NCCTG) phase II trials for patients with newly diagnosed high-grade gliomas. Methods: Measures included QOL LASAs (overall, physical, emotional, spiritual, intellectual), Symptom Distress Scale (SDS), Profile of Mood States (POMS), and Functional Assessment for Cancer Therapy (FACT; overall, physical, emotional). Association of LASA measures with SDS, POMS, and FACT domains and with ECOG performance score (PS) and MMSE was assessed with Spearman’s correlation. Repeated measures ANOVA models compared the change over time of LASAs and SDS, POMS, and FACT. Cox regression modeled the association of baseline QOL and survival. Results: 205 patients completed the QOL assessments across 3 time points. LASA mean scores ranged from 60–78; SDS, POMS, and FACT ranged from 68–81. No significant changes across time for overall and emotional scores were observed. FACT physical decreased over time (p&lt;0.001) as did LASA physical (p=0.08). LASA scales were strongly associated with corresponding scales on SDS, POMS, and FACT (0.44&lt;rho&lt;0.65; p&lt;0.001). LASA was negatively associated with PS and positively with MMSE. Baseline scores for overall FACT (p=0.005) and LASA physical (p=0.015) were associated with better survival. Conclusions: The single-item LASA assessments have comparable psychometric properties as longer assessments for the same constructs. Correlations with PS and MMSE were as expected (convergent/concurrent validity). The discriminant validity and prognostic ability of LASA items were similar to the multi-item instruments. Collectively, the data suggest that the single item LASA scales are valid for assessing QOL in patients with newly diagnosed high grade gliomas and are an appropriate alternative when a shorter instrument is warranted. No significant financial relationships to disclose.

Statistical methods for repeated qualitative assessments on scales

Assessments of auditory communication are commonly performed with questionnaires or multi-item instruments. The item responses generate ordered categorical data. The aim of the work reported here was to present a statistical approach that takes account of the non-metric properties of data from rating scales for description of multi-item assessments and for evaluation of change in a study concerning the function of and satisfaction with hearing aids. A small data set from an ongoing study at the Ahlsén Research Institute, Örebro University Hospital was used in the worked examples. The use of individual and median group profiles provided an interpretable description of the data. By means of the ranking approach, which ties the ranks lo the pairs of data, the systematic group change in hearing function was evaluated separately from the additional individual changes. The use and misuse of parametric statistical methods on ordered categorical data are also discussed.

Guidelines to statistical evaluation of data from rating scales and questionnaires.

Questionnaires and rating scales are commonly used to measure qualitative variables, such as feelings, attitudes and many other behavioural and health-related variables. There are different types of instruments ranging from single scales to multidimensional, multi-item questionnaires. The scaling of the responses can vary from the dichotomous alternatives “yes” and “no” to a mark on a line, as in the visual analogue scale (VAS). Numerical labels are commonly used for the recordings. Nevertheless, irrespective of the type of scaling, the item responses indicate only an ordered structure and not a numerical value in a mathematical sense. Such data are often called ordered categorical or ordinal (1–4). Statistical methods for data from rating scales must take account of the rank-invariant propertiesof ordinal data, whichmeans that the methodsmust be unaffected by a relabelling of the scale categories. Hence, statistical methods applicable to data from rating scales differ completely from the traditional methods for quantitative variables, since calculations based on adding or subtracting ordinal data are not appropriate. Sum scores of multi-item assessments, the mean value, standard deviation and calculation of differences for description of change in score do not have an interpretable meaning and must be avoided in the statistical evaluation of data from rating scales and questionnaires (4–6). Traditionally, in applied research, there is a temptation to treat data from rating scales as numerical on an interval level (4, 5). It should be emphasized, however, that data on an interval level are quantitative, which means that such data have the mathematical properties of well-deŽ ned size and equidistance, but the same variable does not have the same ratio when it is measured in different units (1). Hence, qualitative data could never gain the properties required for being treated as interval data. Statistical methods for quantitative data are valid only when data have the mathematical properties of well-deŽ ned size and distance, and conclusions drawn from such analyses are solely interpretable and reliable for quantitative data. However, quantitative data such as blood pressure could be treated as ordinal when categorized as “low”, “normal” and “high”. Such categorization changes the choice of appropriate statistical methods of analysis. GUIDELINES FOR STUDIES INCLUDING RATING SCALES AND/OR QUESTIONNAIRES