Psychometric Models Research Articles

Investigating heterogeneity in IRTree models for multiple response processes with score-based partitioning.

Item response tree (IRTree) models form a family of psychometric models that allow researchers to control for multiple response processes, such as different sorts of response styles, in the measurement of latent traits. While IRTree models can capture quantitative individual differences in both the latent traits of interest and the use of response categories, they maintain the basic assumption that the nature and weighting of latent response processes are homogeneous across the entire population of respondents. In the present research, we therefore propose a novel approach for detecting heterogeneity in the parameters of IRTree models across subgroups that engage in different response behavior. The approach uses score-based tests to reveal violations of parameter heterogeneity along extraneous person covariates, and it can be employed as a model-based partitioning algorithm to identify sources of differences in the strength of trait-based responding or other response processes. Simulation studies demonstrate generally accurate Type I error rates and sufficient power for metric, ordinal, and categorical person covariates and for different types of test statistics, with the potential to differentiate between different types of parameter heterogeneity. An empirical application illustrates the use of score-based partitioning in the analysis of latent response processes with real data.

The Children’s Somatic Symptoms Inventory-8: Psychometric Properties of a Brief Measure of Somatic Distress

Background: Children often present to primary and specialty care clinics with multiple somatic symptoms of nonspecific origin that can be highly distressing and prompt significant health service use. We evaluated the psychometric properties of the eight-item Children’s Somatic Symptoms Inventory (CSSI-8) as a brief measure of somatic distress that could be easily integrated into clinical systems. Method: Eight items from the 24-item CSSI were selected based on their representation of multiple bodily systems, association with high base rates, and ability to maximize the separation of the items’ Rasch measure scores. The psychometric quality of the eight-item scale was evaluated in 876 pediatric patients with chronic abdominal pain and a nonclinical sample of 954 school children using methods from three psychometric models (the classical test theory, Rasch modeling, and confirmatory factor analysis). Results: The CSSI-8 showed good measurement properties on an extensive array of psychometric criteria, had adequate Rasch person separation reliability for a brief instrument (rsep = 0.74–0.75), and distinguished between clinical and nonclinical youth. Girls in both groups had significantly higher CSSI-8 scores than boys. Norms for the clinical sample are presented. Conclusions: The CSSI-8 is a psychometrically sound measure suitable for use as a brief dimensional assessment of pediatric somatic distress.

Is Bilinguality an Illusive Concept?

Background and Objective: In an effort to study bilinguality, researchers in this field have often compared bilinguals with monolinguals, generally considering each group to be a homogeneous, monolithic body with no discernable within-group individual differences. This approach overlooks the various levels of proficiency among bilinguals, the age and order of acquisition of each of their languages, as well as their language history, and daily usage. Although monolinguals may not be conversant, some may still at least partially comprehend a second language. By failing to actively consider and collect more details about a variety of participant language attributes, researcher assumptions can lead to conclusions that are not a true reflection of the similarities and differences between monolinguals and bilinguals. A number of studies have found bilingual advantage in health-related outcomes as well as performance on cognitive tasks. All these reported findings are based on the performance of participants who may or may not be true bilinguals consequently impacting the results of this research [9,11]. Recommendations: The impact of considering a wide variable of participant characteristics in bilinguality research, including age and order of acquisition, daily usage, and fluency is examined. In addition, a recently proposed psychometric model that evaluates the above variables is discussed. Finally, recommendations are made on how to further improve the standards of research in bilinguality, including performing objective fluency testing, and asking detailed questions about participants’ complete language history.

Customizing Bayesian multivariate generalizability theory to mixed-format tests.

Mixed-format tests, which typically include dichotomous items and polytomously scored tasks, are employed to assess a wider range of knowledge and skills. Recent behavioral and educational studies have highlighted their practical importance and methodological developments, particularly within the context of multivariate generalizability theory. However, the diverse response types and complex designs of these tests pose significant analytical challenges when modeling data simultaneously. Current methods often struggle to yield reliable results, either due to the inappropriate treatment of different types of response data separately or the imposition of identical covariates across various response types. Moreover, there are few software packages or programs that offer customized solutions for modeling mixed-format tests, addressing these limitations. This tutorial provides a detailed example of using a Bayesian approach to model data collected from a mixed-format test, comprising multiple-choice questions and free-response tasks. The modeling was conducted using the Stan software within the R programming system, with Stan codes tailored to the structure of the test design, following the principles of multivariate generalizability theory. By further examining the effects of prior distributions in this example, this study demonstrates how the adaptability of Bayesian models to diverse test formats, coupled with their potential for nuanced analysis, can significantly advance the field of psychometric modeling.

Critical Consciousness Scale and Career Contexts: Cross-Cultural Adaptation and Psychometric Properties of the Brazilian Version

According to the Brazilian pedagogue Paulo Freire, Critical Consciousness aims to understand people’s ability to critically analyze social and structural inequities and their role in social power relations, while also promoting political and social engagement, both individually and collectively. In career studies, the Psychology of Working Theory views Critical Consciousness as an individual moderator of variables that predict perception and access to decent work. This article presents the cross-cultural adaptation procedures and psychometric properties of the Brazilian version of the Critical Consciousness Scale. The sample consisted of 554 Brazilian workers, with women making up the majority (62.5%), and an average age of 35.8 years (SD = 14.3). The analysis of the internal structure (exploratory and confirmatory factor analyses) revealed a 24-item measure with three factors: critical reflection ( α = 0.93; ω = 0.93), critical responsibility ( α = 0.73; ω = 0.77), and critical action ( α = 0.84; ω = 0.85). The instrument showed partial invariance between groups based on gender, subjective social class, and education. Further investigations into the construct in the career field with different groups of workers are discussed, along with the need for caution when interpreting the Freirean conception of consciousness based on psychometric models.

Applying Cross-Validated Psychometric Models to the Bavarian Reading Test (BYLET)

Abstract: In this study, we compared three different models of reading comprehension on a large dataset of more than 6,500 students. We compared two models representing four comprehension processes as well as the influence of text difficulty to a one-factor model by applying multi-dimensional item response models to the newly developed Bavarian reading test (BYLET). Cross-validation indicated the best generalizability for the one-factor model, but factor loadings and global model fit showed some evidence for the influence of text difficulty measured by word and sentence length and the process structure. All psychometric models tested had a good fit, as indicated by global fit indices and loading patterns. The general factor scores point to reliability and validity. We conclude that theories of reading comprehension processes also apply to some extent to the measurement of reading comprehension as a trainable skill and that the general factor score of the BYLET is suitable for a reading comprehension screening between grades two and six. The study is preregistered. The analysis code is available at https://osf.io/xw9bv/?view_only=21549993ef79426eb4006ef82415f25c . Test materials can be sent to interested researchers on request.

Exploring Estimation Procedures for Reducing Dimensionality in Psychological Network Modeling.

To understand psychological data, it is crucial to examine the structure and dimensions of variables. In this study, we examined alternative estimation algorithms to the conventional GLASSO-based exploratory graph analysis (EGA) in network psychometric models to assess the dimensionality structure of the data. The study applied Bayesian conjugate or Jeffreys' priors to estimate the graphical structure and then used the Louvain community detection algorithm to partition and identify groups of nodes, which allowed the detection of the multi- and unidimensional factor structures. Monte Carlo simulations suggested that the two alternative Bayesian estimation algorithms had comparable or better performance when compared with the GLASSO-based EGA and conventional parallel analysis (PA). When estimating the multidimensional factor structure, the analytically based method (i.e., EGA.analytical) showed the best balance between accuracy and mean biased/absolute errors, with the highest accuracy tied with EGA but with the smallest errors. The sampling-based approach (EGA.sampling) yielded higher accuracy and smaller errors than PA; lower accuracy but also lower errors than EGA. Techniques from the two algorithms had more stable performance than EGA and PA across different data conditions. When estimating the unidimensional structure, the PA technique performed the best, followed closely by EGA, and then EGA.analytical and EGA.sampling. Furthermore, the study explored four full Bayesian techniques to assess dimensionality in network psychometrics. The results demonstrated superior performance when using Bayesian hypothesis testing or deriving posterior samples of graph structures under small sample sizes. The study recommends using the EGA.analytical technique as an alternative tool for assessing dimensionality and advocates for the usefulness of the EGA.sampling method as a valuable alternate technique. The findings also indicated encouraging results for extending the regularization-based network modeling EGA method to the Bayesian framework and discussed future directions in this line of work. The study illustrated the practical application of the techniques to two empirical examples in R.

Latent growth mixture models as latent variable multigroup factor models: Comment on McNeish et al. (2023).

McNeish et al. argue for the general use of covariance pattern growth mixture models because these models do not involve the assumption of random effects, demonstrate high rates of convergence, and are most likely to identify the correct number of latent subgroups. We argue that the covariance pattern growth mixture model is a single random intercept model. It and other models considered in their article are special cases of a general model involving slope and intercept factors. We argue growth mixture models are multigroup invariance hypotheses based on unknown subgroups. Psychometric models in which trajectories are modeled using slope factor loadings which vary by latent subgroup are often conceptually preferable. Convergence rates for mixture models can be substantially improved by using a variance component start value taken from analyses with one fewer class and by specifying multifactor models in orthogonal form. No single latent growth model is appropriate across all research contexts and, instead, the most appropriate latent mixture model must be "right-sized" to the data under consideration. Reanalysis of a real-world longitudinal data set of posttraumatic stress disorder symptomatology reveals a three-group model involving exponential decline, further suggesting that the four-group "cat's cradle" pattern frequently reported is artefactual. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Testing the convergent validity, domain generality, and temporal stability of selected measures of people’s tendency to explore

Given the ubiquity of exploration in everyday life, researchers from many disciplines have developed methods to measure exploratory behaviour. There are therefore many ways to quantify and measure exploration. However, it remains unclear whether the different measures (i) have convergent validity relative to one another, (ii) capture a domain general tendency, and (iii) capture a tendency that is stable across time. In a sample of 678 participants, we found very little evidence of convergent validity for the behavioural measures (Hypothesis 1); most of the behavioural measures lacked sufficient convergent validity with one another or with the self-reports. In psychometric modelling analyses, we could not identify a good fitting model with an assumed general tendency to explore (Hypothesis 2); the best fitting model suggested that the different behavioural measures capture behaviours that are specific to the tasks. In a subsample of 254 participants who completed the study a second time, we found that the measures had stability across an 1 month timespan (Hypothesis 3). Therefore, although there were stable individual differences in how people approached each task across time, there was no generalizability across tasks, and drawing broad conclusions about exploratory behaviour from studies using these tasks may be problematic. The Stage 1 protocol for this Registered Report was accepted in principle on 2nd December 2022 https://doi.org/10.6084/m9.figshare.21717407.v1. The protocol, as accepted by the journal, can be found at https://doi.org/10.17605/OSF.IO/64QJU.

Enhancing Workplace Communications Through the Applications of Coloured Brain

Miscommunication happens when there are discrepancies between what is said and what is understood. Effective communication is essential at work as it not only generates stronger interpersonal relationships but also impacts the business’s overall productivity. It is important therefore to overcome this situation. The Directive Communication Coloured Brain psychometric model by Arthur Carmazzi is the organization of the different classifications of “genetic neuro-processing” and defines the way each person experiences their environment and takes in and interprets information, which results in specific ways of perceiving and evaluating events and situations. There are 4 colours; red, blue, green and purple. Each of these colours have different characteristics. As such, their understanding of subjects will be different from those with different colours. A study was conducted to executives in a manufacturing organisation. The employees were trained and tested on the application of colored brain. They found that it was easier to communicate with their colleagues.

Certification exams evaluated with serial case reports: theoretical and problematic approach.

The objective of this issue is a theoretical approach to the disadvantages involved in taking the certification exams of medical specializations through serial case reports and analyzing them with the Classical Test Theory (TCT). The certification exams of medical specializations correspond to criterial summative exams with high consequences. Therefore, it is imperative to maintain the highest quality standards in all the processes involved in preparing the exam. At present, it has been detected that some councils of medical specialties conduct the certification exams with tests that contain serial case reports and the psychometric analysis that they conduct is supported by the TCT; however, the structure of this type of test violates the fundamental assumptions of the TCT. The violation of the fundamental assumptions of the TCT in the tests constructed through serial case reports can lead to biases or misinterpretations of the results. Due to the above, it is advisable to use other psychometric models for the analysis of this type of test, or to avoid the use of serial case reports in the certification exams of medical specialties.

A data-driven typology of emotion regulation profiles.

Typologies serve to organize knowledge and advance theory for many scientific disciplines, including more recently in the social and behavioral sciences. To date, however, no typology exists to categorize an individual's use of emotion regulation strategies. This is surprising given that emotion regulation skills are used daily and that deficits in this area are robustly linked with mental health symptoms. Here, we attempted to identify and validate a working typology of emotion regulation across six samples (collectively comprised of 1,492 participants from multiple populations) by using a combination of computational techniques, psychometric models, and growth curve modeling. We uncovered evidence for three types of regulators: a type that infrequently uses emotion regulation strategies (Lo), a type that uses them frequently but indiscriminately (Hi), and a third type that selectively uses some (cognitive reappraisal and situation selection), but not other (expressive suppression), emotion regulation strategies frequently (Mix). Results showed that membership in the Hi and Mix types is associated with better mental health, with the Mix type being the most adaptive of the three. These differences were stable over time and across different samples. These results carry important implications for both our basic understanding of emotion regulation behavior and for informing future interventions aimed at improving mental health. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Animation and Manga on Improvement in Students’ Problem-Solving Capabilities: Comparison of Two Psychometric Models

To improve the problem-solving capabilities of students, this study explored using interactive digital teaching materials based on animation and manga. A classroom teaching system was developed based on teaching material using a manga format. Students from a national university and a science and technology university in southern Taiwan were recruited. A questionnaire survey was administered for data collection. A total of 104 valid samples were collected; the data were analysed using SPSS 17.0. Two statistically significant models could be used to predict learning outcomes, and the highest R2 value was 52.5%. The utilisation of interactive multimedia teaching materials based on animation and manga helped students apply knowledge to solve practical problems, cultivating their problem-solving capabilities. The findings differed from previous studies; models developed in this study could be used to determine the effects and relationships of multiple factors.

A Note on Ising Network Analysis with Missing Data.

The Ising model has become a popular psychometric model for analyzing item response data. The statistical inference of the Ising model is typically carried out via a pseudo-likelihood, as the standard likelihood approach suffers from a high computational cost when there are many variables (i.e., items). Unfortunately, the presence of missing values can hinder the use of pseudo-likelihood, and a listwise deletion approach for missing data treatment may introduce a substantial bias into the estimation and sometimes yield misleading interpretations. This paper proposes a conditional Bayesian framework for Ising network analysis with missing data, which integrates a pseudo-likelihood approach with iterative data imputation. An asymptotic theory is established for the method. Furthermore, a computationally efficient Pólya-Gamma data augmentation procedure is proposed to streamline the sampling of model parameters. The method's performance is shown through simulations and a real-world application to data on major depressive and generalized anxiety disorders from the National Epidemiological Survey on Alcohol and Related Conditions (NESARC).

Improving Instructional Decision‐Making Using Diagnostic Classification Models

AbstractIn recent years, educators, administrators, policymakers, and measurement experts have called for assessments that support educators in making better instructional decisions. One promising approach to measurement to support instructional decision‐making is diagnostic classification models (DCMs). DCMs are flexible psychometric models that facilitate fine‐grained reporting on skills that students have mastered. In this article, we describe how DCMs can be leveraged to support better decision‐making. We first provide a high‐level overview of DCMs. We then describe different methods for reporting results from DCM‐based assessments that support decision‐making for different stakeholder groups. We close with a discussion of considerations for implementing DCMs in an operational setting, including how they can inform decision‐making at state and local levels, and share future directions for research.

Management styles and innovation

PurposeUsing a sample of about 323 Italian companies and 423 managers from July 2020–March 2021, the work tests hypotheses on reliability of The Open MOOd Questionnaire. Data collected are analyzed by a psychometric model and a latent trait is found. Results are verified and validated. Moreover, external robustness is assured by comparing the Rasch Analysis results with other statistical methods.Design/methodology/approachThe purpose of this paper is to measure the level of openness of managers through the Open MOOd Questionnaire. Which aspects allow us to identify different management styles, and if the “openness” characterizes specific clusters of managers.FindingsThe study shows that the openness attitude in the professional activity of the managers is very high. The Open MOOd Questionnaire can be improved by considering some dichotomic items.Practical implicationsThe study highlights the model proposed identifies the Open Manager’s competencies, and identifies teaching methods for their development.Originality/valueThe authors propose a new model to study openness. Diagnostic tools of consolidated statistical methods show that the Open MOOd Questionnaire is an effective tool to evaluate the openness.

A Statistical Model Regarding the Solving Problems Behavior in Formal Groups

Abstract Formal group behaviour is based on following specific norms and rules wherein the participants are expected to be well organised proving respect and accomplish orders, working to the requirements of the organization. When different tasks are essential to be solved, each member of the group must adopt proactive behaviour, so the communication process to be productive. The paper presents a practical approach about how the gender of students influences organisational group communication with the main objectives to evaluate the solving problems abilities on two groups (M/F) and create a psycho-metric model for solving problems inside formal groups. The authors used the questionnaire of Cândea R.M. and Cândea D. as an instrument that includes three types of behaviour that may appear during solving problem process - approaching a routine problem to solve it; approaching a problem that is not routine and measuring aspects regarding decision process encouraging the creativity and the spirit of innovation in a formal group. Furthermore, the authors used statistical methods - Shapiro-Will and Man-Whitney U tests- and statistical factorial modelling – PCA method to identify the most critical factors that influence solving problems in formal groups. The necessity of the research comes from the dynamic of changes affecting the group from external and internal environments during the decision process.

New Psychometric Evidence of the Grief Impairment Scale (GIS) in People Who Have Experienced the Death of a Loved One From a Network Psychometric Approach in Two Latin American Countries.

This study aimed to evaluate the psychometric properties of the Grief Impairment Scale (GIS) using a network psychometric model. A total of 1048 individuals from Peru and El Salvador participated. A network psychometric model was used to determine internal structure, reliability, and cross-country invariance. The results indicate that the GIS items were grouped into a single network structure through Exploratory Graph Analysis. Reliability was estimated by structural consistency, and it was found that when replicating the network structure within an empirical dimension, a single network structure was consistently obtained, and all items remained stable. Furthermore, the network structure was invariant, thus functioning similarly across the different country groups. In conclusion, the GIS presented solid psychometric evidence of validity based on its internal structure, reliability, and cross-country invariance. Therefore, the GIS is a psychometrically sound measure of functional impairment symptoms due to grief for Peruvian and Salvadoran individuals.

Investigating the capability of ChatGPT for generating multiple-choice reading comprehension items

The development of multiple-choice reading comprehension items based on specific reading subskills is crucial in teaching, learning, and testing reading. But it remains a challenging task because this process is time-consuming and costly. This study aims to investigate the capability of ChatGPT (Chat Generative Pre-trained Transformer) for generating multiple-choice reading comprehension items. Psychometric models and human review were adopted to evaluate the item quality based on the benchmark of human-authored items. The results showed that ChatGPT-authored multiple-choice items were acceptable and comparable to human-authored items in terms of psychometric properties, and human review by questionnaire, expert judgment, and interview found that ChatGPT had the potential capability to serve as a test developer and assistant for teaching and learning reading. However, some shortcomings and potential pitfalls were also identified and room for improvement was discussed when ChatGPT is applied to generate items for educational purposes.

A Diagnostic Facet Status Model (DFSM) for Extracting Instructionally Useful Information from Diagnostic Assessment.

Modern assessment demands, resulting from educational reform efforts, call for strengthening diagnostic testing capabilities to identify not only the understanding of expected learning goals but also related intermediate understandings that are steppingstones on pathways to learning goals. An accurate and nuanced way of interpreting assessment results will allow subsequent instructional actions to be targeted. An appropriate psychometric model is indispensable in this regard. In this study, we developed a new psychometric model, namely, the diagnostic facet status model (DFSM), which belongs to the general class of cognitive diagnostic models (CDM), but with two notable features: (1) it simultaneously models students' target understanding (i.e., goal facet) and intermediate understanding (i.e., intermediate facet); and (2) it models every response option, rather than merely right or wrong responses, so that each incorrect response uniquely contributes to discovering students' facet status. Given that some combination of goal and intermediate facets may be impossible due to facet hierarchical relationships, a regularized expectation-maximization algorithm (REM) was developed for model estimation. A log-penalty was imposed on the mixing proportions to encourage sparsity. As a result, those impermissible latent classes had estimated mixing proportions equal to 0. A heuristic algorithm was proposed to infer a facet map from the estimated permissible classes. A simulation study was conducted to evaluate the performance of REM to recover facet model parameters and to identify permissible latent classes. A real data analysis was provided to show the feasibility of the model.