Interpretation Of Test Scores Research Articles

Continuous Norming Approaches: A Systematic Review and Real Data Example.

Norming of psychological tests is decisive for test score interpretation. However, conventional norming based on subgroups results either in biases or require very large samples to gather precise norms. Continuous norming methods, namely inferential, semi-parametric, and (simplified) parametric norming, propose to solve those issues. This article provides a systematic review of continuous norming. The review includes 121 publications with overall 189 studies. The main findings indicate that most studies used simplified parametric norming, not all studies considered essential distributional assumptions, and the evidence comparing different norming methods is inconclusive. In a real data example, using the standardization sample of the Need for Cognition-KIDS scale, we compared the precision of conventional, semi-parametric, and parametric norms. A hierarchy in terms of precision emerged with conventional norms being least precise, followed by semi-parametric norms, and parametric norms being most precise. We discuss these findings by comparing our findings and methods to previous studies.

Validity evidence for the attitudes on person-centered behavior toward students questionnaire (APBS): Internal structure and associations with external variables

AbstractBuilding positive teacher-student relationships (TSR) is a central task for teachers. According to the person-centered approach, teachers create positive relationships by treating students with unconditional positive regard, empathic understanding, and genuineness. Numerous studies demonstrate the impact of person-centered teacher behavior for students’ achievement and well-being. The effectiveness of person-centered behavior is supposed to be determined by underlying attitudes. Accordingly, person-centered attitudes should receive greater focus in both teacher education and TSR research. However, a suitable measurement instrument is currently lacking. Therefore, the APBS instrument was developed, a theoretically-grounded self-report questionnaire that measures pre-service teachers' attitudes on person-centered behavior toward students. In previous studies, the suitability of the test's content and internal structure were empirically examined using EFA. The present study investigated the internal structure using CFA, as well as the associations with external variables, in a sample of N = 1284 pre-service teachers from German universities (Mage = 23.3, SDage = 5.2; 79.8% female). Different factor models were tested and compared. The model with four first-order factors (unconditionality, empathic understanding, trust, genuineness) along with the APBS higher-order factor showed the best fit (χ2/df(1504.868/520) = 2.894, p < .001; CFI = .968; RMSEA = .038). Both the four first-order scales (ωt* = .81-.90), and the second-order scale (ωt* = .95) exhibited satisfactory internal consistency. In addition, structural regression and mediation analyses revealed mostly theory-consistent relationships with respondents' attachment representation, empathy and relational competence. In summary, the findings support the intended interpretation of APBS test scores. The questionnaire can be used as a measurement instrument in TSR research and as a self-exploration tool in teacher education.

Digital Assessment Tools: As a Media for Assesing High School Science Learning

This research is to produce a suitable digital assessment tool for high school students' in chemical literacy with validity, reliability, levels of difficulty and discrimination power. The research metodology is Research and Development (R&D) using the Oriondo and Dallo-Antonio model. The research stages include planning the test instrument, testing, determining empirical validity, determining reliability, and interpreting test scores. In the planning (instrument creation), where the validation score estimates by experts in content, language, construction, and media are above 0.8, indicating that the test instrument is considered suitable for further testing to estimate validity, reliability, difficulty levels, and discrimination power. Out of the 25 developed questions, 24 fit the Rasch model. The reliability of Instrument Part I is 0.67, and Part II is 0.65, falling into the "sufficient" category. The questions exhibit varied difficulty levels, ranging from very easy to very difficult, with a standard deviation (SD) of 0.21 for Part I and 0.14 for Part II. The separation achieved by Instrument Part I is into 4 ability groups, while Instrument Part II separates into 2 ability groups. The distractors for multiple-choice questions fall into the "good" and "very good" categories, fulfilling their intended function effectively.

Validation of the Kimberley Cognitive Assessment (KICA-Cog) for Torres Strait Islander Peoples.

The aim of this study was to validate the Kimberley Indigenous Cognitive Assessment-Cognitive Component (KICA-Cog) adapted for dementia screening in Torres Strait Islander Peoples. Data were obtained from a broader dementia prevalence study completed in the Torres Strait and Northern Peninsula Area between 2015 and 2018. Modifications were made to items from the original KICA-Cog to ensure they were culturally appropriate for the Torres Strait. All participants completed a KICA-Cog and had a comprehensive dementia assessment with a geriatrician experienced in cross-cultural assessment. A total of 255 Torres Strait residents aged 45 years and over completed a KICA-Cog and underwent geriatric assessment. The adapted KICA-Cog showed good validity for dementia diagnosis with a cut point of 33/34 associated with a sensitivity of 81% and specificity of 92% with an area under the ROC curve of 0.91. The KICA-Cog, when modified for the Torres Strait, is a valid cognitive screening tool for dementia. Caution is required when interpreting test scores, as the adapted KICA-Cog had slightly lower sensitivity (ability to detect people with dementia) than the original KICA-Cog. As with all short cognitive tests, individuals with a low KICA-Cog scores should undergo further medical investigations before a dementia diagnosis is considered.

Setting standards for a diagnostic test of aviation English for student pilots

Standard-setting is an essential component of test development, supporting the meaningfulness and appropriate interpretation of test scores. However, in the high-stakes testing environment of aviation, standard-setting studies are underexplored. To address this gap, we document two stages in the standard-setting procedures for the Overseas Flight Training Preparation Test (OFTPT), a diagnostic English test for ab initio pilots, aligned to the International Civil Aviation Organization (ICAO)’s Language Proficiency Rating Scale (LPRS). Performance-level descriptors (PLDs) were empirically generated in Stage 1 in collaboration with six subject matter experts (SMEs). These PLDs made explicit the correspondence between linguistic performance levels within the target language use domain and the ICAO scale. Findings suggest that the ICAO scale is not fine-grained enough to distinguish levels of linguistic readiness among ab initio pilots, nor does it adequately reflect the knowledge, skills, and abilities valued by SMEs within this domain. In Stage 2, 12 SMEs were recruited to set standards and were divided into two groups to investigate the replicability of Ebel method standard-setting procedures. Cut scores were determined for the OFTPT reading and listening tests, which were inferentially linked to the LPRS. There were no significant differences in the cut scores arrived at by both groups and reliability was excellent, suggesting that test users can have confidence in the standards set.

Calibrating Items Using an Unfolding Model of Item Response Theory: The Case of the Trait Personality Questionnaire 5 (TPQue5).

Unfolding item response theory (IRT) models are important alternatives to dominance IRT models in describing the response processes on self-report tests. Their usage is common in personality measures, since they indicate potential differentiations in test score interpretation. This paper aims to gain a better insight into the structure of trait personality, by investigating whether the dominance or alternatively the unfolding IRT model are better descriptors of the response processes on a personality measure constructed under the dominance response theorem. For the assessment of the dominant model, the Graded Response Model (GRM) is used; while for the unfolding model, the Generalized Graded Unfolding Model (GGUM) was examined. All analyses are conducted with the freely available R. A sample of 1340 Greek adults, employed in private and public organizations, fulfilled the Trait Personality Questionnaire 5 short-form (TPQue5). Findings contradict previous research on trait personality. In accordance to the construction method employed, the TPQue5 items are best understood by monotonically increasing item response functions (IRFs). Individuals responding to the TPQue5 increase their probability of endorsing its items as their trait level increases; this stands for all personality dimensions, although Openness to Experience exhibited mixed type of item response patterns. Further research directions, implications and limitations are also discussed.

Digital measurement of hands-on performance? Ecological validation of a computer-based assessment of automotive repair skills

We investigated the ecological validity of performance measures from a computer-based assessment tool that utilises scripted video vignettes. The intended purpose of this tool is to assess the maintenance and repair skills of automotive technician apprentices, complementing traditional hands-on assessment formats from the German journeymen’s exams. We hypothesise that the ability to correctly judge repair actions shown in videos is a good predictor of the ability to perform corresponding actions in hands-on scenarios. Apprentices in the third year of vocational training carried out repairs on real cars or car systems, while experts rated their performance. After this, they worked on our computer-based tests, which utilise videos of very similar repairs. The correlation between video judgement and hands-on performance was lower than expected for most repair actions as well as for overall scores, indicating insufficient ecological validity of the test score interpretations. However, the findings are promising for developing future tests, as the results for some repair actions indicate it is generally possible to develop ecologically valid video-based items focusing on hands-on skills. We discuss the results in the light of a validation framework that combines validity evidence from different sources for the same assessment tool. Finally, we hope our findings contribute to a broader discussion about the psychometric quality of exams.

Preseason Measures of Cervical Spine Function, Vestibulo-Ocular Reflex, Dynamic Balance, and Divided Attention in Youth Ice Hockey Players.

OBJECTIVES: Report typical scores and examine preseason cervical spine, vestibulo-ocular reflex, dynamic balance, and divided attention measures in competitive youth ice hockey players aged 10 to 18 years with and without a previous concussion history. DESIGN: Cross-sectional secondary analysis. METHODS: The exposure of interest was self-reported history of concussion. The main outcomes were cervical spine measures (Cervical Flexor Endurance [CFE; seconds], Cervical Flexion-Rotation Test [normal/abnormal], Anterolateral Cervical Spine Strength [kilograms], Head Perturbation Test (/8), and Joint Position Error [JPE; centimeters]), vestibulo-ocular reflex (Dynamic Visual Acuity [logMAR], Head Thrust Test [Positive/Negative]), dynamic balance (Functional Gait Assessment [/30]) and divided attention (Walking While Talking Test [seconds]). Multivariable linear or logistic regression, adjusted for age-group, sex, level of play, and clustered by team, were used to assess potential differences by concussion history. RESULTS: We included data from 2311 participants in this study (87.2% male, 12.8% female, 39.0% reported a previous concussion). No differences by concussion history were found across any of the measures (P values range: 0.17-0.99). Measures of cervical spine function and divided attention differed by age group (eg, Median Left Anterolateral Cervical Spine Strength [kilograms] for males: U13 = 7.46, U15 = 9.10, U18 = 9.67). CONCLUSION: Clinical outcomes scores in youth ice hockey players did not differ by concussion history. Performance on cervical spine strength, CFE, and JPE test outcomes may improve with age, highlighting the importance of developmental considerations when interpreting test scores. J Orthop Sports Phys Ther 2024;54(3):1-11. Epub 30 November 2023. doi:10.2519/jospt.2023.11958.

Test-retest reliability of the urban outdoor situated phoneme (SiP) test*

Objective To introduce the urban outdoor version of the Situated Phoneme (SiP) test and investigate its test-retest reliability. Design Phonemic discrimination scores in matched-spectrum real-world (MSRW) maskers from an urban outdoor environment were measured using a three-alternative forced choice test paradigm at different phoneme-to-noise ratios (PNR). Each measurement was repeated twice. Test-retest scores for the full 84-trial SiP-test, as well as for four types of contrasting phonemes, were analysed and compared to critical difference scores based on binomial confidence intervals. Study sample Seventy-two adult native speakers of Swedish (26–83 years) with symmetric hearing threshold levels ranging from normal hearing to severe sensorineural hearing loss. Results Test-retest scores did not differ significantly for the whole test, or for the subtests analysed. A lower amount of test-retest score difference than expected exceeded the bounds of the corresponding critical difference intervals. Conclusions The urban outdoor SiP-test has high test-retest reliability. This information can help audiologists to interpret test scores attained with the urban outdoor SiP-test.

Mixed Methods Investigation into Test Score Users’ Perspectives about IELTS Reading Skill Profiles

Research on the use of standardized test scores in higher education reveals significant variations in attitudes and perceptions of language proficiency tests among test score users. Most test score users have limited knowledge about test score interpretations in terms of what English as additional language (EAL) students typically know and can do at the language proficiency levels associated with admission cut scores. To address this critical gap, the language testing field has actively investigated the potential of Diagnostic Classification Models (DCMs) to offer useful information, facilitating test score users in their decision-making processes. The present two-phase mixed methods study examined the characteristics of reading skill profiles across various IELTS band scores, specifically focusing on the most frequently used admission cut scores: 6.0, 6.5, and 7.0. The study further explored test score users’ perspectives about these admission test scores, challenges encountered by EAL students, and the usefulness of reading skill profiles derived from DCMs. Findings from the application of DCMs to IELTS reading test responses (N = 5,222) showed a lack of advanced skills, such as inferential reasoning, at these commonly employed cut scores. Test score users perceived the skill profiles as instrumental in distinguishing reading abilities across various band scores and discussed the EAL students’ lack of critical reasoning, especially in inferential and synthesis tasks in coursework. The results underscore the potential of DCM-based skill profiles in providing test score users with detailed information about test score interpretations.

Does prior law enforcement experience affect scores on preemployment psychological testing? An investigation using the MMPI-3.

Most states require that a police candidate's suitability be assessed through preemployment psychological evaluations. Given the psychological stress officers experience in the line of duty, it is possible that experienced candidates would produce scores indicating greater dysfunction on psychological testing relative to inexperienced candidates. No known research has investigated whether police candidates produce different preemployment test scores across experience levels. The present study sought to address this gap in the literature and offer data to guide evaluators' interpretation of test scores for candidates with prior law enforcement (LE) experience. We used a sample of 400 police candidates (18.5% women) evaluated for positions at several LE agencies for whom Minnesota Multiphasic Personality Inventory-3 (MMPI-3) scores were available. Results showed no practically meaningful differences when comparing MMPI-3 scores of candidates with no prior experience, less than 5 years of experience, and 5 or more years of experience. We also compared frequencies at which the three groups elevated MMPI-3 scales at cutoffs specified in Assessing Police and Other Public Safety Personnel Using the MMPI-3: A Practical Guide (Corey & Ben-Porath, 2022) and found minimal differences. Together, these findings indicate that MMPI-3 scores can be interpreted consistently with published guidelines regardless of a candidate's prior LE experience. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

5 The impact of recreational cannabis use on neuropsychological function in epilepsy

Objective:Cannabis is classified as a class B drug in the UK with penalties for possession of up to 5 years in prison, an unlimited fine or both. Nevertheless it is widely available and is the most commonly used drug in the UK with approximately 2.6 million (7.6%) of adults reporting that they sometimes or regularly use it. It is not uncommon for people who present in our epilepsy clinic to report regular use of cannabis; some use it recreationally whilst others report ‘self-medicating’ based on the belief that it has a beneficial impact on their seizures. The aim of this study was to establish the prevalence of cannabis use in people with epilepsy referred for a neuropsychological assessment and to examine the impact of cannabis use on cognitive function in this group.Participants and Methods:All patients who attend for a neuropsychological assessment are routinely asked about illegal drug use in their clinical interview. This information is also captured in the medical and neuropsychiatric assessments they undergo when assessed by the multidisciplinary team. The electronic medical records of 800 consecutive patients who had undergone a neuropsychological assessment between 2019 and 2022 were searched for references to cannabis use. The neuropsychological profiles of patients reporting cannabis use were compared to those seen in the larger series across multiple cognitive domains.Results:Seventy (8.75%) of the patients in the series reported past or present cannabis use. Cannabis users were more likely to be male (p<0.01) and were younger (p<0.01) than those who did not report use. Reading IQ was significantly lower in the cannabis group (p<0.001). Patients who were regularly using cannabis at the time of the neuropsychological assessment did not differ from the rest of the cohort on tests of processing speed, working memory, naming or verbal fluency. There were no differences between the groups in their performance on an embedded measure of performance validity. However the patients who were regularly using cannabis at the time of their neuropsychological assessments scored significantly lower on tests of verbal learning (p<0.05) and reported significantly greater subjective memory problems in everyday life (p=0.02) than the non-cannabis group. The group using cannabis also scored significantly more highly on the depression (p<0.01) and anxiety scales (p=0.02) on the Hospital Anxiety and Depression Scale.Conclusions:The prevalence and patterns of cannabis use in the epilepsy population mirror those seen in the wider population. The impact on regular cannabis use on neuropsychological function appears to be most evident on measures of new learning and subjective measures of memory disturbance. Cannabis use is significantly associated with lower levels of cognitive reserve and elevated levels of anxiety and low mood. Whilst caution must be employed with respect to any direct attribution in these complex clinical presentations, these findings may be helpful in the interpretation of neuropsychological test scores and the planning of interventions, particularly with respect to subjective memory complaints in this group.

Recovery after a suicidal episode: Developing and validating the Recovery Evaluation and Suicide Support Tool (RESST).

The personal recovery movement advocates for shifting emphasis from clinical symptom reduction toward reclaiming personal agency and creating meaning-filled lives despite the presence of distressing experiences. Corresponding personal recovery measures have been developed; however, there is no established psychometric assessment of personal recovery following a suicidal episode. This study addressed this gap by developing the Recovery Evaluation and Suicide Support Tool (RESST) and assessing its test score reliability, test score interpretations' validity, and psychometric properties. Throughout RESST's development, input from diverse stakeholders-including clinicians, researchers, and individuals with lived experience-was gathered to ensure a meaningful and useful scale. Exploratory factor analysis techniques were used with adults with a suicidal episode history (N = 502) to select and refine items, culminating in a 21-item scale with four distinct subscales: Self-Worth, Life Worth, Social Worth, and Self-Understanding. Confirmatory factor analysis techniques demonstrated model fit across three samples of adults with a suicidal episode history (combined N = 1,523), and test-retest reliability was obtained (N = 204). The results revealed that RESST scores exhibit an internally consistent and replicable factor structure, consistent with personal recovery theory. Additionally, the interpretation of test scores exhibited both convergent and discriminant validity. Mental health indices related to recovery, negative mood states, suicidality, and meaning in life had significant moderate-to-strong correlations with the RESST, supporting the validity of the test score interpretations and clinical relevance. This measure should aid research into recovery processes and understanding how recovery following a suicidal episode may be enhanced clinically and personally. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Examining Appropriacy of CFI and TLI Cutoff Value in Multiple-Group CFA Test of Measurement Invariance to Enhance Accuracy of Test Score Interpretation

The most common effect size when using a multiple-group confirmatory factor analysis approach to measurement invariance is ΔCFI and ΔTLI with a cutoff value of 0.01. However, this recommended cutoff value may not be ubiquitously appropriate and may be of limited application for some tests (e.g., measures using dichotomous items or different estimation methods, sample sizes, or model complexity). Moreover, prior cutoff value estimations often have ignored consequences resulting in using measures that more accurately estimate countries’ or learners’ proficiency for some countries or groups versus others. In this study, we investigate whether the cutoff value proposed by Cheung and Rensvold (ΔCFI or ΔTLI > 0.01) is appropriate across educational measurement contexts. Specifically, we investigated the performance of ΔCFI and ΔTLI in capturing LOI at the scalar level in dichotomous items within item response theory on groups whose test characteristic curves differed by 0.5. Simulation results showed that the proposed cutoff value of 0.01 in ΔCFI and ΔTLI was not appropriate to capture LOI under the study conditions, which may result in the misinterpretation of test results or inaccurate inferences.

Interpretation and Use of a Workplace English Language Proficiency Test Score Report: Perspectives of TOEIC® Test Takers and Score Users in Taiwan

AbstractResearch in validity suggests that stakeholders' interpretation and use of test results should be an aspect of validity. Claims about the meaningfulness of test score interpretations and consequences of test use should be backed by evidence that stakeholders understand the definition of the construct assessed and the score report information. The current study explored stakeholders' uses and interpretations of the score report of a workplace English language proficiency test, the TOEIC® Listening and Reading (TOEIC L&R) test. Online surveys were administered to TOEIC L&R test takers and institutional and corporate score users in Taiwan to collect data about their uses and interpretations of the test score report. Eleven survey respondents participated in follow‐up interviews to further elaborate on their uses of the different score reporting information within the stakeholders' respective contexts. Results indicated that the participants used the TOEIC L&R test scores largely as intended by the test developer although some elements of the score report appeared to be less useful and could be confusing for stakeholders. Findings from this study highlight the importance of providing score reporting information with clarity and ease to enhance appropriate use and interpretation.

The Influence of Age, Gender and Education on Neuropsychological Test Scores: Updated Clinical Norms for Five Widely Used Cognitive Assessments.

Sociodemographic effects (i.e., age, gender, education) have been shown to influence neuropsychological test scores. The current retrospective, quasi-epidemiological work provides age-, gender- and education-corrected clinical norms for five common cognitive assessments. In total, test scores of 4968 patients from the University Hospital of Cologne (Department of Neurology), recruited between 2009 and 2020, were analyzed retrospectively. Conducted tests were the Mini-Mental State Examination (MMSE), F-A-S Test (FAS), Rey-Osterrieth Complex Figure Test (ROCFT) and Trail Making Test, Part A and B (TMT-A/-B). Using multiple linear regression analyses, test scores were analyzed for sociodemographic influences (age, gender, education). Based on these analyses, norms were generated by first separating patients into different age groups stratified by educational level and (if necessary) gender. Subsequently, percentile ranks and z-scores for a subsample including only individuals without dementia were calculated. Lower age and higher educational level predicted better test scores (MMSE, FAS, ROCFT) and completion times (TMT-A/-B). Additionally, produced words on the FAS and remembered drawings from the ROCFT were influenced by gender, with females having better FAS but lower ROCFT (delayed recall) scores than males. Considering these effects, clinical norms were provided for the five cognitive assessments. We found influences of age, gender and education on test scores, although they are frequently not or only partially considered for test score interpretation. With the provided norms, neuropsychologists can make more profound evaluations of cognitive performance. A user-friendly Microsoft Excel file is offered to assist this process.

On Validity

The way in which validity has been conceptualized has changed throughout the years. The focus in validation studies shifted from evaluating distinct components of validity to developing a comprehensive argument for the use and interpretations of test scores. The argument-based approach to validity incorporates the distinct types of the componential approach, underscores Messick’s attention to construct validity, highlights the need for evidence of the evidence-gathering approach, and combines it all into one logical argumentative structure. Such a comprehensive argument is now an indispensable part of validity studies (Dursun & Li, 2021).

Investigation into the factor structure of a local English placement test of oral communication

AbstractThis study describes the development process and examines the construct validity of an English placement test of oral communication (EPT OC) developed at a Midwestern university in the United States. This test includes a one‐on‐one oral interview and paired discussion task, and test performance is judged on an analytic rating scale. A confirmatory factor analysis conducted on the ratings of 338 students who took the initial fully operational EPT OC revealed the test structure was represented by a correlated four‐factor model with interactional competence, fluency, pronunciation/comprehensibility, and grammar/vocabulary as sub‐constructs, in line with its targeted theoretical framework. Both tasks were effective in measuring the targeted sub‐constructs, but the sub‐constructs were not sufficiently distinct from each other to completely justify a four‐factor model. The findings provide some support for the proposed interpretations of the EPT OC test scores but indicate the need for some modifications to the assessment, such as more thorough rater training and/or revised rating scales to better distinguish the targeted sub‐constructs.

Developer involvement and COI disclosure in high-stakes English proficiency test validation research: A systematic review

Language test developers play a critical role in gathering evidence to support test score interpretations and real-world uses, but independent research is necessary to put claims of validity to the test (Kane, 2013). This study systematically reviews the prevalence of developer involvement in high-stakes English proficiency test research published in five peer-reviewed language testing journals from 2016 to 2021, focusing on aspects of validity addressed, research methodology, and disclosure of relevant conflicts of interest (COI). Nearly half of the 181 included studies featured some degree of developer involvement, in terms of authorship or funding, and developer-involved studies had some notable methodological advantages, including larger sample sizes and access to official test materials/data. COI reporting was generally poor, with only 4-7% of studies with potential conflicts making adequate disclosures. We discuss implications of these findings in terms of evaluating tests and make recommendations for greater transparency in COI reporting and validation research.

Social consequences of educational measures: A commentary.

This commentary discusses Messick's (1995) unified validity framework, which broadly characterizes the interpretation of test scores in terms of their social consequences. Scores and their interpretation can lead to unintended consequences when they capture only part of the specified construct or characteristics unrelated to the construct. The evaluation of construct underrepresentation and irrelevance requires careful investigation and rational argument about the construct and its theoretical basis, as well as any planned uses, contexts, scores, or samples. Developers also validate an assessment for specific purposes, and users share responsibility for validation for any novel use or interpretation of scores. This commentary also considers the consequences of decisions based on assessments and the consequences of local and national norms. (PsycInfo Database Record (c) 2023 APA, all rights reserved).