Modern Psychometrics Research Articles

How Modern Psychometrics Can Change Intelligence Testing of Children and Adolescents in Russia

The assessment of intellectual development in children and adolescents is an essential component of psychodiagnostics, facilitating an understanding of the nature of learning difficulties. Intelligence tests have a history that spans more than a century, and even contemporary versions of widely used intelligence tests (such as the Wechsler or Stanford-Binet tests) are limited by the influence of their traditional paper-based formats. Nevertheless, the advent of computational capabilities and the digitalization of tests have led psychometrics to experience substantial development, opening new possibilities for assessment. The complete application of modern psychometrics holds the promise of revolutionizing the measurement of intelligence. This paper presents the current situation of measuring intelligence in children and adolescents in the world and in Russia. It describes the potentialities presented by modern psychometrics, including the utilization of innovative task formats such as technology-enhanced items, the integration of universal design principles in test development, the adoption of computerized adaptive testing (CAT) and multistage testing (MST) methodologies, the application of multidimensional IRT models to establish both construct and criterion validity, the shift from discrete norms to continuous scaling, and the incorporation of longitudinal measurements. The combination of these features within a computerized intelligence testing framework will enhance respondents' motivation and simplify the test administration process. It will facilitate frequent assessments of a child's progress by presenting varying test versions on each occasion, thereby significantly reducing errors in administration and scoring, while also safeguarding against misuse by non-professionals.

DIGITALIZATION AT THE SERVICE OF PSYCHODIAGNOSTICS: MODERN OPPORTUNITIES IN INTELLIGENCE TESTING

The assessment of intellectual development in children and adolescents is an essential component of psychodiagnostics, facilitating an understanding of the nature of learning diffi culties. Intelligence tests have a history that spans more than a century, and even contemporary versions of widely used intelligence tests (such as the Wechsler or Stanford-Binet tests) are limited by the infl uence of their traditional paper-based formats. Nevertheless, the advent of computational capabilities and the digitalization of tests have led psychometrics to experience substantial development, opening new possibilities for assessment. The complete application of modern psychometrics holds the promise of revolutionizing the measurement of intelligence. Present-day psychometrics is deeply entwined with computational behavioral sciences, going beyond outdated principles in the development of psychological tests. Methodological approaches that notably enhance the accuracy, reliability, and equity of measurement instruments currently exist. Furthermore, the accumulation of supplementary validity evidence, simplifi - cation of result interpretation, and practical implementation far exceed conventional psychodiagnostic approaches. This presentation will show the potentialities presented by modern psychometrics, including the utilization of innovative task formats such as technology-enhanced items, the integration of universal design principles in test development, the adoption of computerized adaptive testing (CAT) and multistage testing (MST) methodologies, the application of multidimensional IRT models to establish both construct and criterion validity, the shift from discrete norms to continuous scaling, and the incorporation of longitudinal measurements. The combination of these features within a computerized intelligence testing framework will enhance respondents’ motivation and simplify the test administration process. It will facilitate frequent assessments of a child’s progress by presenting varying test versions on each occasion, thereby signifi cantly reducing errors in administration and scoring, while also safeguarding against misuse by non-professionals.

Rasch analysis of the Oxford shoulder score in a non-surgical occupational population.

The Oxford shoulder score (OSS) questionnaire for measuring patient perception of shoulder disability, has not tested specifically in a non-surgical population and no study has assessed the OSS with modern psychometrics based on Rasch model (RM). To assess the psychometric properties of the OSS using RM among health-care workers with shoulder disorders and to verify its interest in a non-surgical population. In an occupational health department of a French hospital center, a retrospective review was performed of the medical records from June 2019 to October 2020. Responses to 110 questionnaires were examined from 55 subjects (97% of women). A polytomous Rasch model based on the Partial Credit Model was used. Overall fit was satisfactory, the reliability coefficient was high and an ascending order was observed with the 5 categories of the scale. Analysis of the residuals supports unidimensionality and the local independence assumption. Item performance remained stable across the subgroup examined (DIF measures). Scale to-sample targeting indicated a substantial floor effect, and the mildest impairments were not well discriminated. OSS presents good psychometric qualities. However, it does not clearly discriminate subjects presenting the lowest levels of impairment. Its use in a non-surgical population is questionable.

Patient-Reported Measures of the Effects of Vision Impairments and Low Vision Rehabilitation on Functioning in Daily Life.

The quantification of vision impairments dates to the mid-nineteenth century with standardization of visual acuity and visual field measures in the eye clinic. Attempts to quantify the impact of vision impairments on patients' lives did not receive clinical attention until the close of the twentieth century. Although formal psychometric theories and measurement instruments were well developed and commonplace in educational testing, as well as in various areas in psychology and rehabilitation medicine, the late start applying them to clinical vision research created a vacuum that invited poorly developed and poorly functioning instruments and analytic methods. Although this research is still burdened with legacy instruments, mandates by regulatory agencies to include the patients' perspectives and preferences in the evaluation of clinical outcomes have stimulated the development and validation of self-report instruments grounded in modern psychometric theory and methods. Here I review the progress and accomplishments of applying modern psychometrics to clinical vision research.

Cognitive diagnostic modelling in healthcare professions education: an eye-opener

Criticisms about psychometric paradigms currently used in healthcare professions education include claims of reductionism, objectification, and poor compliance with assumptions. Nevertheless, perhaps the most crucial criticism comes from learners' difficulty in interpreting and making meaningful use of summative scores and the potentially detrimental impact these scores have on learners. The term "post-psychometric era" has become popular, despite persisting calls for the sensible use of modern psychometrics. In recent years, cognitive diagnostic modelling has emerged as a new psychometric paradigm capable of providing meaningful diagnostic feedback. Cognitive diagnostic modelling allows the classification of examinees in multiple cognitive attributes. This measurement is obtained by modelling these attributes as categorical, discrete latent variables. Furthermore, items can reflect more than one latent variable simultaneously. The interactions between latent variables can be modelled with flexibility, allowing a unique perspective on complex cognitive processes. These characteristic features of cognitive diagnostic modelling enable diagnostic classification over a large number of constructs of interest, preventing the necessity of providing numerical scores as feedback to test takers. This paper provides an overview of cognitive diagnostic modelling, including an introduction to its foundations and illustrating potential applications, to help teachers be involved in developing and evaluating assessment tools used in healthcare professions education. Cognitive diagnosis may represent a revolutionary new psychometric paradigm, overcoming the known limitations found in frequently used psychometric approaches, offering the possibility of robust qualitative feedback and better alignment with competency-based curricula and modern programmatic assessment frameworks.

Development of the Myasthenia Gravis (MG) Symptoms PRO: a case study of a patient-centred outcome measure in rare disease

BackgroundMyasthenia gravis (MG) is a chronic autoimmune neuromuscular disease, characterised by fluctuating muscle weakness which makes it challenging to assess symptom severity. Mixed methods psychometrics (MMP), which combines evidence from qualitative research and modern psychometrics, is a versatile approach to the development of patient-centred outcome measures (PCOM) in the context of rare disease. Our objective was to develop the MG Symptom patient-reported outcome (PRO) to assess key aspects of MG severity from the patient perspective.MethodsWe used MMP to develop a novel PRO instrument in a multi-step process. An initial conceptual model for MG patient experience was developed and expanded based on preliminary literature review and two waves of concept elicitation interviews with people with MG (Step 1). Based on this, the novel PRO instrument, the MG Symptoms PRO, was drafted. The draft instrument was refined by combining qualitative and quantitative data collected in a Phase 2 clinical study (Step 2).ResultsFindings from the literature review and concept elicitation interviews (n = 96) indicated that patient experience in MG includes proximal muscle weakness symptoms related to several body parts, along with muscle weakness fatigability and general fatigue. Then, a set of 42 items across five scales (ocular-, bulbar-, and respiratory muscle weakness, physical fatigue, and muscle weakness fatigability) was developed. Qualitative evidence endorsed its relevance, clarity, and ease of completion; quantitative analysis with Rasch measurement theory methods demonstrated strong measurement properties, including good targeting and high reliability. Classical test theory analyses showed adequate reliability of the instrument and mild to moderate correlations with other widely used MG-specific outcome measures.ConclusionsThe MG Symptoms PRO has potential to be used both to measure treatment benefit in clinical trials and monitor symptom severity in clinical practice. Its component scales were purposefully designed to stand alone, enhancing interpretability of scores given the heterogeneity of MG, and enabling modular use. Compared with existing MG PROs, it contains more detailed assessments of muscle weakness and muscle weakness fatigability symptoms, which are of key importance to people with MG. The MMP approach used may serve as a case study for developing PCOMs across rare disease indications.

Clinical validation of the scales of depression, neuroticism, somatization, personal anxiety and validity of the modified questionnaire for neurotic disorders

The classical theory of tests, in the framework of which a questionnaire for neurotic disorders was created, does not meet the requirements of modern psychometrics. In this regard, it became necessary to use new technologies for its modification and effective use in the clinic. The Rush metric system, a variation of modern test theory, is the only measurement model that meets the criteria for a joint additive measurement. On the basis of the claims of the questionnaire for neurotic disorders, 13 scales with proven psychometric characteristics were formed in the framework of the Rush model. This article evaluated the clinical efficacy of five of them.The purpose of the study: to conduct clinical validation of the scales of depression, neuroticism, somatization, personal anxiety and validity of the modified version of the questionnaire for neurotic disorders.Materials and Methods: the sample consisted of research data from a group of subjects in the amount of 296 people. Among them, 167 women and 129 men. Statistical methods of work—factor analysis of standardized residuals, ROC analysis, correlation analysis of the total indicator.Results: 1. the points of each investigated scale characterize various aspects of one construct, that is, the scales are homogeneous; 2. through ROC analysis, high diagnostic properties of the scales were identified and cut-off values for each scale were calculated, which were the criteria for interpretation; 3. the criterion validity of the studied scales is statistically significantly higher or not different from the validity of the well-known questionnaires, which are the «gold standard» of measurement, which demonstrated a statistical comparison of their ROC curves; 4. the correlation analysis of the total indicator revealed statistically significant (p <0.05) correlation coefficients of the scales of the modified questionnaire among themselves, as well as with the assessments according to the MMPI scales, QIDS-SR16 questionnaire and the Hamilton anxiety score scale (HADS).Conclusion: the scales of depression, neuroticism, somatization, personal anxiety and validity of the modified version of the neurotic disorders questionnaire are clinically valid, and therefore, can be used to solve various practical problems in psychodiagnostics.

ОЦЕНКА КЛИНИЧЕСКОЙ ЭФФЕКТИВНОСТИ ЧЕТЫРЕХ ШКАЛ МОДИФИЦИРОВАННОГО ВАРИАНТА ОПРОСНИКА НЕВРОТИЧЕСКИХ РАССТРОЙСТВ

The neurotic disorders questionnaire was originally created on the basis of the classical test theory and does not meet the requirements of modern psychometrics. Within the framework of our research, this technique has been modified and consists of 13 scales, the psychometric analysis of which included all the technical stages of iterative analysis and scale modeling based on the Rush metric system. This article presents the results of clinical validation of four of them. Purpose of the study. To assess the clinical effectiveness of the scales of coping deficit, anankasticity, the scale of impulsivity and addictive reactions, as well as the scale of general personal disorganization of the modified version of the questionnaire of neurotic disorders. Material and methods. 296 people were examined. Among them, 167 are women and 129 are men. Statistical methods of the study are factor analysis of standardized residuals, ROC analysis, correlation analysis. Results: The studied scales are homogeneous in their structure. Using the ROC-analysis, high and moderate differentiating properties of the scales were revealed, cutoff values were calculated, which were the criteria for interpretation. Correlation analysis of the total indicator revealed statistically significant correlations between the studied scales and the MMPI scores, the QIDS-SR16 questionnaire, and the Hamilton Anxiety Rating Scale (HADS). Conclusions: The scales of coping deficit, anankasticity, the scale of impulsivity and addictive reactions, as well as the scale of general personal disorganization of the modified version of the neurotic disorders questionnaire are clinically valid and can be used to solve various practical problems.

Composition of practice-oriented tasks for autonomous learning within massive open online courses (some aspects of the modern psychometrics)

The article deals with the aspects of the development of practical tasks for massive open online courses. The article also covers the importance of psychometrics for the development of digital courses and technical capabilities of modern popular MOOC platforms. The article describes the method of practical tasks selection to the educational materials named after G. Roche, which allows composing practical framework for the studied courses. Despite the common tools of pedagogical design of academic and digital courses, the latter are characterized by technical constraints. In order to cover cognitive competencies of all levels it is proposed to use Bloom’s taxonomy, which is used in this article within the framework of popular MOOC platforms.

Book Review: Modern Psychometrics With R

Book Review: Modern Psychometrics With R

ПСИХОДІАГНОСТИЧНА МЕТОДИКА "ПАРЦІАЛЬНІ ПОЗИЦІЇ ЖИТТЄСТІЙКОСТІ ОСОБИСТОСТІ": РЕЗУЛЬТАТИ РОЗРОБКИ ТА АПРОБАЦІЇ

The article presents the theoretical construction, stages of creation and results of psychometric verification of the test questionnaire "Partial positions of personality hardiness" (PPH). Hardiness stands for the ability of an individual to assimilate experience and development during life's difficulties. Partial positions of personality hardiness cover a specific manifestation of the characteristics in different types of hardships, which is implemented in a range of emotional reactions and feelings, the nature of psychological difficulties, mechanisms of assimilation of problem experience, coping strategies and transformation of unpleasant circumstances. The developed test questionnaire contains 100 statements, distributed on 5 scales of 20 statements each: "hardiness in a situation of stress", "hardiness in a situation of frustration", "hardiness in a situation of conflict", "hardiness in a crisis", "resilience in situations of uncertainty". The algorithm of the original method construction provided for several stages that meet the requirements of modern psychometrics. At the stage of approbation of the test questionnaire its reliability (method of division in half, parallel test, check of independence of results from the personality of the diagnostician), obvious, constructive, convergent; competitive validity (methods of correlation, factor analysis), discriminativeness has been verified. The presence of correlations between the indicators of the PPH test questionnaire and the S. Muddy Test of hardiness (adapted by D. Leontiev, O. Rasskazova) indicates that the developed method measures the same psychological quality as the reference test, but low (from 0.20 to 0.30) and average (from 0.30 to 0.50) values of correlation coefficients indicate that the new method has different theoretical construction grounds. Comparison of data between the test-questionnaire of PPH and parallel tests ("Test-questionnaire of indicators' diagnostics of psychological crisis experience" (PE) (O. Sannikova, I. Brynza), and a method of "Self-assessment of mental states according to G. Eysenck") showed existence of negative correlations between of the majority of comparative methods' indicators of (p≤0.01; p≤0.05). The results of approbation approved the conformity of the created method in accordance with psychometric requirements, which allows to use it both for scientific and practical purposes.

Exploring of mathematics classroom goal structures in senior high school: An engaging in academic work for student

This study aimed to investigate classroom goal structures based on the teacher’s learning objectives for students in the mathematics classroom. The study used survey methods with involved 319 students at senior high school level. Data analysis based upon Rasch model as the framework of modern psychometrics approach. The internal consistency and psychometrics properties for quality of instruments, was appropriate. The results of this study showed that students tend to refuse to labelling as embarrassment behaviour if any student who cannot answer or do math problems in the classroom. The students argue although answering math problems properly is a pride or achievement; but, create a judge “poor students” in mathematics should not occur in the classroom because typically all students are still in the learning stage. Another interesting side of this research findings is that male and female students have differences in looking at the learning climate of mathematics in relation to engaging in academic work. In the context of teacher-student interaction in the classroom, it is vital to consider that a constructive classroom situation and solid learning experienced can avoid to declining students’ academic motivation and achievement.

Neuropsychological tests of the future: How do we get there from here?

Objective: This article reviews current approaches to neuropsychological assessment, identifies opportunities for development of new methods using modern psychometric theory and advances in technology, and suggests a transition path that promotes application of novel methods without sacrificing validity. Methods: Theoretical/state-of-the-art review.Conclusions: Clinical neuropsychological assessment today does not reflect advances in neuroscience, modern psychometrics, or technology. Major opportunities for improving practice include both psychometric and technological strategies. Modern psychometric approaches including item response theory (IRT) enable linking procedures that can place different measures on common scales; adaptive testing algorithms that can dramatically increase efficiency of assessment; examination of differential item functioning (DIF) to detect measures that behave differently in different groups; and person fit statistics to detect aberrant patterns of responding of high value for performance validity testing. Opportunities to introduce novel technologies include computerized adaptive testing, Web-based assessment, healthcare- and bio-informatics strategies, mobile platforms, wearables, and the ‘internet-of-things’. To overcome inertia in current practices, new methods must satisfy requirements for back-compatibility with legacy instrumentation, enabling us to leverage the wealth of validity data already accrued for classic procedures. A path to achieve these goals involves creation of a global network to aggregate item-level data into a shared repository that will enable modern psychometric analyses to refine existing methods, and serve as a platform to evolve novel assessment strategies, which over time can revolutionize neuropsychological assessment practices world-wide.

Chop and Change: A Commentary and Demonstration of Classical vs. Modern Measurement Models for Interpreting Latent-Stability of Occupational-Future Time Perspective.

This commentary article was initially motivated by an empirical paper published in the journal of Work, Aging, and Retirement that reported support for stability (non-decreasing) future time perspectives (FTP) over two repeated-measurements. That is, empirical evidence supporting the temporal stability of an adapted measure (occupational-FTP [O-FTP]) serves as guiding framework for demonstrating limitations of classical test theory (CTT) and modern psychometrics’ (IRT) enabling extension for stronger substantive inferences from response data. The focal authors’ quantitative attention to study design and statistical analysis is commendable. In this commentary, I aim to complement their efforts from a measurement perspective. This is accomplished through four sections. In the first section, I summarize some well-known limitations to CTT measurement models for assessing change. Then, I briefly introduce item response theory (IRT) as an alternative test theory. In the second section, Chop, I review the empirical evidence for FTP and O-FTP’s latent-factor structure. Then, I bring evidence from modern psychometric methods to bear on O-FTP, specifically, a model-comparisons approach was adopted for comparing relative fit of 1-factor, 2-factor, and bifactor solutions in cross-sectional data (N = 511). Findings supported retention of the bifactor solution. In the third section, Change, I extend the bifactor model to two-wave FTP data over approximately 2 years (N = 620) as an instructive application for assessing temporal stability. The fourth section concludes with a brief discussion of substantive implications and meaningful interpretation of (O)-FTP scores over time.

The EFPA Test-Review Model: When Good Intentions Meet a Methodological Thought Disorder.

The European Federation of Psychologists’ Associations (EFPA) has issued sets of test standards and guidelines for psychometric test reviews without any attempt to address the critical content of many substantive publications by measurement experts such as Joel Michell. For example, he has argued that the psychometric test-theory which underpins classical and modern IRT psychometrics is “pathological”, with the entire profession of psychometricians suffering from a methodological thought disorder. With the advent of new kinds of assessment now being created by the “Next Generation” of psychologists which no longer conform to the item-based, statistical test theory generated last century, a new framework is set out for constructing evidence-bases suitable for these “Next Generation” of assessments, which avoids the illusory beliefs of equal-interval or quantitatively structured psychological attributes. Finally, with no systematic or substantive refutations of the logic, axioms, and evidence set out by Michell and others; it is concluded psychologists and their professional associations remain in denial. As with the eventual demise of a similar attempt to maintain the status quo of professional beliefs within forensic clinical psychology and psychiatry during the last century, those following certain EFPA guidelines might now find themselves required to justify their professional beliefs in legal rather than academic environments.

The Psychometric Properties of an Internet-Administered Version of the Depression Anxiety and Stress Scales (DASS) in a Sample of Dutch Adults

Psychometric work on the widely used Depression Anxiety and Stress Scales (DASS) has mostly used classical psychometrics and ignored common internet-administered versions. Therefore, the present study used not only classical, but also modern psychometrics based on item response theory (IRT) to evaluate an internet-administered version of the DASS (Dutch translation). Internet-administered DASS data were collected as part of a large internet-based study in the Dutch adult population (n = 7972). Initially, external correlates (i.e. demographics other measures) and some classical psychometrics (internal consistency, convergent/divergent validity) of the DASS scales were evaluated. Next, IRT was used to investigate the scales’ dimensionality, discrimination and item-functioning. Finally, the DASS depression scale was further investigated by linking it to the more clinically-oriented Quick Inventory of Depressive Symptomatology (QIDS) using item response theory (IRT). Initial classical psychometric analyses supported the scales’ internal consistency (alpha = 0.94–0.98) and convergent/divergent validity. IRT analyses showed that each of the DASS scales was only suitable to measure variations in a very narrow and rather mild severity range. Linking the DASS depression scale with the QIDS also showed that the DASS depression scale discriminated best in the mild-moderate severity range, but not at higher severity levels that were covered by the QIDS. In conclusion, the scales of the internet-administered DASS show good internal consistency and validity. However, users should be aware that the scales discriminate best at mild-moderate severity ranges in the general population.

ПСИХОМЕТРИЧЕСКИЕ СВОЙСТВА ШКАЛЫ КОНСТРУКТИВНЫХ СТРАТЕГИЙ МЕТОДИКИ ДИАГНОСТИКИ КОПИНГ-ПОВЕДЕНИЯ

Background. The problem of coping behavior diagnostics is of particular interest due to the requirements of modern psychometrics. At the moment, diagnostic methods of this construct with reasonable psychometric characteristics are lacking in clinical practice.Purpose. The research purpose was to analyze the psychometric properties of the scale of constructive strategies of the coping behavior questionnaire based on the Rasch model.Material and methods. A total of 270 respondents were assessed with the help of the coping behavior diagnostic technique. The difficulty of the scale items was calculated. The construct validity of the scale was assessed. The reliability and separation statistics values were determined.Results. The scale item difficulty measures ranged from -1.04 to 1.20 logits. The UMS and WMS indices had acceptable values (in the range of 0.7 – 1.3). The index of reliability was 0.81.Conclusions. The scale items of the designed questionnaire are balanced by the degree of difficulty. The construct validity of the items is sufficient. The index of reliability of the method has an acceptable significance (0.81). The using of the scale makes it possible to diagnose three levels of severity of the construct.

Resolving Uncertainty About the Intolerance of Uncertainty Scale–12: Application of Modern Psychometric Strategies

abstractIn this study, we evaluated the factor structure, reliability estimates, item parameters, and differential correlates of the short form of the Intolerance of Uncertainty Scale (Carleton, Norton, & Asmundson, 2007) in samples of undergraduate women (n = 387) and men (n = 276) ranging in age from 18 to 49 years (M = 20.20, SD = 3.91). This instrument was designed to measure 2 facets of intolerance of uncertainty—prospective anxiety and inhibitory anxiety—although total scores on the measure are often used. A major objective of this study was to determine the degree to which derivation of total versus subscale scores is empirically permissible. Comparison of a bifactor model to a unidimensional model and a 2-factor correlated traits model indicated that the bifactor model exhibited superior fit to the sample data. This model provided evidence of a strong general intolerance of uncertainty factor that was more reliable and accounted for significantly more common variance than either subscale factor. Examination of the item response theory slope parameters revealed negligible bias in the measure's items across genders. Finally, a series of simultaneous regression analyses was conducted to examine differential correlates of the measure's total scale scores for men and women.

What does the beck depression inventory measure in myocardial infarction patients? a psychometric approach using item response theory and person-fit.

Observed associations between depression following myocardial infarction (MI) and adverse cardiac outcomes could be overestimated due to patients' tendency to over report somatic depressive symptoms. This study was aimed to investigate this issue with modern psychometrics, using item response theory (IRT) and person-fit statistics to investigate if the Beck Depression Inventory (BDI) measures depression or something else among MI-patients. An IRT-model was fit to BDI-data of 1135 MI patients. Patients' adherence to this IRT-model was investigated with person-fit statistics. Subgroups of "atypical" (low person-fit) and "prototypical" (high person-fit) responders were identified and compared in terms of item-response patterns, psychiatric diagnoses, socio-demographics and somatic factors. In the IRT model, somatic items had lower thresholds compared to depressive mood/cognition items. Empirically identified "atypical" responders (n = 113) had more depressive mood/cognitions, scored lower on somatic items and more often had a Comprehensive International Diagnostic Interview (CIDI) depressive diagnosis than "prototypical" responders (n = 147). Additionally, "atypical" responders were younger and more likely to smoke. In conclusion, the BDI measures somatic symptoms in most MI patients, but measures depression in a subgroup of patients with atypical response patterns. The presented approach to account for interpersonal differences in item responding could help improve the validity of depression assessments in somatic patients.

The RDoC framework: continuing commentary.

The RDoC framework: continuing commentary.