Item Usage Research Articles

Effect of Differential Item Functioning on Computer Adaptive Testing Under Different Conditions.

This study provides an overview of the effect of differential item functioning (DIF) on measurement precision, test information function (TIF), and test effectiveness in computer adaptive tests (CATs). Simulated data for the study was produced and analyzed with the Rstudio. During the data generation process, item pool size, DIF type, DIF percentage, item selection method for CAT, and the test termination rules were considered changed conditions. Sample size and ability parameter distribution, Item Response Theory (IRT) model, DIF size, ability estimation method, test starting rule, and item usage frequency method regarding CAT conditions were considered fixed conditions. To examine the effect of DIF, measurement precision, TIF and test effectiveness were calculated. Results show DIF has negative effects on measurement precision, TIF, and test effectiveness. In particular, statistically significant effects of the percentage DIF items and DIF type are observed on measurement precision.

Hiding in plain sight: The distinct importance of low-arousal positive affect

Most past research on positive affect and emotion has focused exclusively on high-arousal positive affect (HAPA: e.g., excited), however, low-arousal positive affect (LAPA: e.g., calm) increasingly is included in emotion research. As such, there is a need to synthesize knowledge about the similarities and differences between LAPA and HAPA, the operationalization of LAPA and HAPA, and the distinct characteristics and importance of LAPA within emotional life. A systematic search identified 226 research papers comparing LAPA with HAPA from a broad spectrum of research topics; this review provides a narrative summary of their findings. Indications of differences between LAPA and HAPA were found in 89% of comparisons, with LAPA having a consistently distinguishable relationship to variables such as brain activity, cardiovascular health, decision-making, memory, mindfulness, personality, and solitude, among others. Other notable aspects of LAPA were found, including its role in stress, work, positive sociality, and well-being, as well as its importance in older adults and women. An analysis of items used to measure LAPA and HAPA revealed nuanced differences in conceptualizations, as well as emerging consensus around specific item usage. While considering item use in light of approach-avoidance motivation, we identified three possible LAPA subtypes: calm (a steady state of neither approach nor avoidance), satisfaction (having successfully approached), and relief (having successfully avoided). This review clarifies LAPA’s role in affective life, underscoring that LAPA’s differences from HAPA should be considered in research involving positive affect.

Generating Usage-related Questions for Preference Elicitation in Conversational Recommender Systems

A key distinguishing feature of conversational recommender systems over traditional recommender systems is theirability to elicit user preferences using natural language. Currently, the predominant approach to preference elicitation is to ask questions directly about items or item attributes. Users searching for recommendations may not have deep knowledge of the available options in a given domain. As such, they might not be aware of key attributes or desirable values for them. However, in many settings, talking about the planned use of items does not present any difficulties, even for those that are new to a domain. In this article, we propose a novel approach to preference elicitation by asking implicit questions based on item usage. As one of the main contributions of this work, we develop a multi-stage data annotation protocol using crowdsourcing, to create a high-quality labeled training dataset. Another main contribution is the development of four models for the question generation task: two template-based baseline models and two neural text-to-text models. The template-based models use heuristically extracted common patterns found in the training data, while the neural models use the training data to learn to generate questions automatically. Using common metrics from machine translation for automatic evaluation, we show that our approaches are effective in generating elicitation questions, even with limited training data. We further employ human evaluation for comparing the generated questions using both pointwise and pairwise evaluation designs. We find that the human evaluation results are consistent with the automatic ones, allowing us to draw conclusions about the quality of the generated questions with certainty. Finally, we provide a detailed analysis of cases where the models show their limitations.

Inventory Management with Advance Booking Information: The Case of Surgical Supplies and Elective Surgeries

Problem definition: Medical operations require a large volume and variety of consumable supplies that are kept in hospital inventory and replenished on a regular basis. Stringent requirements on the availability of these supplies, together with high variability in their daily usage, contribute to the high inventory costs of the surgical departments in hospitals. We investigate the value of utilizing Advance Booking Information (ABI) on elective surgeries—which are often booked up to months in advance—in reducing inventory costs. Methodology/results: We study a single-item, periodic-review, stochastic inventory control problem, where the item demand in each period is driven by the number and type of surgeries requiring the item, and with the available information on elective surgeries integrated into the ordering decisions. Given that item usage from each case is uncertain and only realized after the surgery, ABI provides imperfect information on future demand. Through exact analysis of a simplified version of the problem, as well as extensive numerical experiments using synthetic and real data, enabled using a state aggregation technique, we provide insights on and quantify the value of using ABI as a function of the number of periods of ABI integrated into the ordering decisions. We identify a relevant parameter regime—namely, high backlog (relative to holding) costs and when surgeries are booked sufficiently in advance—where the value of using ABI could be significant and the majority of the benefits can be gained through incorporating only one period of ABI beyond the order lead time. In a case study conducted using real data, we observe up to 26% reduction in average inventory levels, without violating the service levels. Managerial implications: By incorporating readily available elective surgery schedules into replenishment decisions of surgical supplies, hospitals could significantly reduce inventory costs without compromising the availability of the supplies. Funding: This work was partially funded by The Ontario Ministry of Government and Consumer Services (MGCS). The views expressed in the paper are the views of the authors and do not necessarily reflect those of the Province. Supplemental Material: The e-companion is available at https://doi.org/10.1287/msom.2021.0063 .

An Efficient Non-parametric Item Selection Method for Polytomous Scoring CD-CAT

In educational evaluations at home and abroad, polytomous scoring items are becoming increasingly important. They can provide richer and more valuable information, with unmatched advantages compared to binary (0-1 scoring) items. If used as a tool for teachers to diagnose and assess students in the classroom, CD-CAT (Cognitive Diagnostic Computerized Adaptive Testing) has significant implications for improving teaching effectiveness. However, in classroom teaching situations, it is not feasible to estimate item parameters accurately with a large sample, as in large-scale assessments. In such cases, non-parametric CD-CAT becomes the only viable option. Compared to parametric CD-CAT, non-parametric CD-CAT started later and is particularly lacking in research related to polytomous scoring. Item selection method is at the core of CD-CAT, so it is essential to develop a non-parametric item selection method suitable for polytomous scoring CD-CAT. This study proposes a non-parametric item selection method for polytomous scoring cognitive diagnostic computerized adaptive testing (PCD-CAT): the Manhattan Distance Non-parametric Difference index item selection method (MD-NDI). The results of simulation studies indicate: (1) MD-NDI item selection method is suitable for PCD-CAT scenarios and exhibits better performance when the item bank quality is poor or the sample size for estimating item parameters is limited. (2) MD-NDI does not require pre-testing of items and distributes item usage more evenly, effectively ensuring the security of the item bank. (3) Even in cases of incorrectly specified item bank of Qc-matrix, MD-NDI still shows higher pattern correct classification rates. (4) In the study of variable-length PCD-CAT, MD-NDI not only reduces the test length in most conditions but also has a higher pattern correct classification rates when reaching the test termination rule.

Controlling the Minimum Item Exposure Rate in Computerized Adaptive Testing: A Two-Stage Sympson-Hetter Procedure.

Computerized adaptive testing (CAT) can improve test efficiency, but it also causes the problem of unbalanced item usage within a pool. The effect of uneven item exposure rates can not only induce a test security problem due to overexposed items but also raise economic concerns about item pool development due to underexposed items. Therefore, this study proposes a two-stage Sympson-Hetter (TSH) method to enhance balanced item pool utilization by simultaneously controlling the minimum and maximum item exposure rates. The TSH method divides CAT into two stages. While the item exposure rates are controlled above a prespecified level (e.g., rmin) in the first stage to increase the exposure rates of the underexposed items, they are controlled below another prespecified level (e.g., rmax) in the second stage to prevent items from overexposure. To reduce the effect on trait estimation, TSH only administers a minimum sufficient number of underexposed items that are generally less discriminating in the first stage of CAT. The simulation study results indicate that the TSH method can effectively improve item pool usage without clearly compromising trait estimation precision in most conditions while maintaining the required level of test security.

Feasibility of implementing a surgical patient safety checklist: prospective cross-sectional evaluation

BackgroundThe World Health Organization’s Global Patient Safety Action Plan 2021–2030 call for attention to patient and family involvement to reduce preventable patient harm. Existing evidence indicates that patients’ involvement in their own safety has positive effects on reducing hospitalisation time and readmissions. One intervention reported in the literature is the use of checklists designed for patients’ completion. Studies on such checklists are small scale, but they are linked to reduction in length of hospital stay and readmissions. We have previously developed and validated a two-part surgical patient safety checklist (PASC). This study aims to investigate the feasibility of the PASC usage and implementation prior to its use in a large-scale clinical trial.MethodsThis is a prospective cross-sectional feasibility study, set up as part of the design of a larger stepped-wedge cluster randomised controlled trial (SW-CRCT). Descriptive statistics were used to investigate patient demographics, reasons for not completing the PASC and percentage of PASC item usage. Qualitative patient interviews were used to identify barriers and drivers for implementation. Interview was analysed through content analysis.ResultsOut of 428 recruited patients, 50.2% (215/428) used both parts of PASC. A total of 24.1% (103/428) of the patients did not use it at all due to surgical or COVID-19-related cancellations. A total of 19.9% (85/428) did not consent to participate, 5.1% (22/428) lost the checklist and 0.7% (3/428) of the patients died during the study. A total of 86.5% (186/215) patients used ≥ 80% of the checklist items. Barriers and drivers for PASC implementation were grouped into the following categories: Time frame for completing the checklist, patient safety checklist design, impetus to communicate with healthcare professionals and support throughout the surgical pathway.ConclusionsElective surgical patients were willing and able to use PASC. The study further revealed a set of barriers and drivers to the implementation. A large-scale definitive clinical-implementation hybrid trial is being launched to ascertain the clinical effectiveness and scalability of PASC in improving surgical patient safety.Trial registrationClinicaltrials.gov: NCT03105713. Registered 10.04.2017

Welcome to Volume 13 of Neurodegenerative Disease Management.

Welcome to Volume 13 of Neurodegenerative Disease Management.

Item selection methods with exposure and time control for computerized classification test.

Computerized classification testing (CCT) commonly chooses items maximizing information at the cut score, which yields the most information for decision-making. However, a corollary problem is that all examinees will be given the same set of items, resulting in high test overlap rate and unbalanced item bank usage, which threatens test security. Moreover, another pivotal issue for CCT is time control. Since both the extremely long response time (RT) and large RT variability across examinees intensify time-induced anxiety, it is crucial to reduce the number of examinees exceeding the time limitation and the differences between examinees' test-taking times. To satisfy these practical needs, this paper proposes the novel idea of stage adaptiveness to tailor the item selection process to the decision-making requirement in each step and generate fresh insight into the existing response time selection method. Results indicate that a balanced item usage as well as short and stable test times across examinees can be achieved via the new methods.

A Newsvendor Approach to Design of Surgical Preference Cards

Surgical procedures require a large number of consumable supplies that need to be kept in hospital inventory and transported to the operating rooms (OR) before the surgery. A surgical preference card (SPC) provides a list of items to be prepared for each surgery. For each item, a SPC also specifies how many should be taken to the OR (fill quantity). As the usage of most consumables in the OR is subject to uncertainty, the cards also specify how many of the filled items should be opened at the beginning of the surgery (open quantity). The fill and open quantities control the flow of consumables between the hospital inventory and the ORs and directly affect the wastage in ORs. In this work, we formulate the problem of determining the fill and open quantities on the preference cards as a stochastic optimization problem, where the objective is to minimize a weighted sum of the expected wastage and operational costs. We show that, as in the newsvendor problem, the optimal solution for the fill and open quantities takes the form of critical quantiles of the item usage distribution in the OR. The solution form together with historical usage data provide a data-driven approach to design of SPCs, as well as insights on the value of including an open decision. We demonstrate our approach using extensive numerical experiments and real usage data from a Canadian hospital. The results suggest a potential for significant reduction of wastage and operational costs in the ORs.

How socio-demographic and mass media factors affect sanitary item usage among women in rural and urban India

Our study’s objective is to examine the various socio-demographic and mass media factors of sanitary item usage among rural and urban women in India. We have used data from the Indian version of the Demographic and Health Survey, conducted in 2015–16 for this study, with a sample of 247,833 women in the age group of 15 to 24 years. Binary logistics regression was performed separately for urban and rural women to understand the various determinants of sanitary item usage. We found that three-quarters of women in urban areas use sanitary items compared to half of the women in rural areas. Indian women in rural areas appear to be at a considerable disadvantage compared to their urban counterparts with regard to sanitary item usage. Factors determining sanitary item usage did not vary between urban and rural areas. However, education, wealth, mass media, and toilet facility were the major factors in understanding the improvement in sanitary item usage in urban and rural areas. At the policy level, there is need to focus on (a) disseminating knowledge about menstrual hygiene management (MHM) practices and provisioning of sanitary items; and (b) providing exposure to traditional media channels to potentially reduce the social stigma associated with these issues and bring critical conversations to the forefront, especially among the socio-economically disadvantaged groups.

Recommendation System With Hierarchical Recurrent Neural Network for Long-Term Time Series

Nowadays, recommendation systems are widely used on various services. This system predicts what item a user will use next using large amounts of stored user history. Recommendation systems are commonly applied in various fields as movies, e-commerce, and social services. However, previous researches on recommendation systems commonly overlooked the importance of item usage sequence and time intervals of the time series data from users. We provide a novel recommendation system that incorporates these temporal properties of the user history. We design a recurrent neural network (RNN) model with a hierarchical structure so that the sequence and time intervals of the user’s item usage history can be considered. The model is divided into two layers: a layer for a long time and a layer for a short time. We conduct experiments on real-world data such as Movielens and Steam datasets, which has a long-time range, and show that our new model outperforms the previous widely used recommendation methods, including RNN-based models. We also conduct experiments to find out the influence of the length and time interval of sequences in our model. These experimental results show that both sequence length and time interval are influential, indicating that it is important to consider the temporal properties for long-term sequences.

Dual Adversarial Variational Embedding for Robust Recommendation

Robust recommendation aims at capturing true preference of users from noisy data, for which there are two lines of methods have been proposed. One is based on noise injection, and the other is to adopt the generative model Variational Auto-encoder (VAE). However, the existing works still face two challenges. First, the noise injection based methods often draw the noise from a fixed noise distribution given in advance, while in real world, the noise distributions of different users and items may differ from each other due to personal behaviors and item usage patterns. Second, the VAE based models are not expressive enough to capture the true preference since VAE often yields an embedding space of a single modal, while in real world, user-item interactions usually exhibit multi-modality on user preference distribution. In this paper, we propose a novel model called Dual Adversarial Variational Embedding (DAVE) for robust recommendation, which can provide personalized noise reduction for different users and items, and capture the multi-modality of the embedding space, by combining the advantages of VAE and adversarial training between the introduced auxiliary discriminators and the variational inference networks. The extensive experiments conducted on real datasets verify the effectiveness of DAVE on robust recommendation.

Procedures to Develop a Computerized Adaptive Testing to Advance the Measurement of Narcissistic Personality.

Narcissistic personality (NP) has recently attracted a great deal of attention. In this study, we mainly investigated the feasibility and effectiveness of using computerized adaptive testing (CAT) to measure NP (CAT-NP). The CAT in this study was simulated by the responses of several NP questionnaires of 1,013 university students as if their responses were collected adaptively. The item bank (85 items) that met the requirements of the psychometric properties of Item Response Theory (IRT) was first established, and then the CAT dynamically selected items according to the estimates of current trait level until the prespecified measurement precision is achieved. Finally, the efficiency and validity of the CAT were verified. The results showed that the proposed CAT-NP had reasonable reliability, validity, predictive utility, and high correlation. In addition, the CAT-NP could significantly save item usage without losing measurement accuracy, which greatly improves the test efficiency. The advantages and limitations of CAT in measuring NP and other psychological tests are discussed in the final section.

Item selection for 12-item short forms of the Knee injury and Osteoarthritis Outcome Score (KOOS-12) and Hip disability and Osteoarthritis Outcome Score (HOOS-12)

To develop 12-item short forms (KOOS-12, HOOS-12) of the 42-item Knee injury and Osteoarthritis Outcome Score (KOOS) and 40-item Hip disability and Osteoarthritis Outcome Score (HOOS) that represent the full-length instruments sufficiently to provide joint-specific pain, function and quality of life (QOL) domain and summary joint impact scores. This paper describes KOOS-12 and HOOS-12 item selection. Subsequent papers will examine KOOS-12 and HOOS-12 reliability, validity and responsiveness. Items were selected based on qualitative information from patients, clinicians and KOOS/HOOS translators and analysis of data from 1,395 knee osteoarthritis (OA) and 1,281 hip OA patients from the FORCE-TJR cohort who completed KOOS or HOOS before and after total joint replacement (TJR). Item response theory models and computerized adaptive test (CAT) simulations were used to identify items that best measured patients' levels of pain and function pre- and post-TJR. KOOS-12/HOOS-12 items were selected based on content, coverage of a wide measurement range, high item information, item usage in CAT simulations, scale-level properties (reliability, validity, responsiveness), and qualitative information. KOOS-12 and HOOS-12 each included a pain frequency item and three items measuring pain during increasingly difficult activities (sitting/lying, walking, up/down stairs); function items about standing, rising from sitting, getting in/out of a car, and twisting/pivoting (KOOS-12) or walking on an uneven surface (HOOS-12); and the original 4-item QOL scale. This study demonstrated the benefits of examining patient-reported outcome measures using modern psychometric methods, to create short forms with diverse content that provide domain-specific and summary joint impact scores.

Computerized Adaptive Testing for Cognitively Based Multiple-Choice Data.

Cognitive diagnosis models (CDMs) are latent class models that hold great promise for providing diagnostic information about student knowledge profiles. The increasing use of computers in classrooms enhances the advantages of CDMs for more efficient diagnostic testing by using adaptive algorithms, referred to as cognitive diagnosis computerized adaptive testing (CD-CAT). When multiple-choice items are involved, CD-CAT can be further improved by using polytomous scoring (i.e., considering the specific options students choose), instead of dichotomous scoring (i.e., marking answers as either right or wrong). In this study, the authors propose and evaluate the performance of the Jensen-Shannon divergence (JSD) index as an item selection method for the multiple-choice deterministic inputs, noisy "and" gate (MC-DINA) model. Attribute classification accuracy and item usage are evaluated under different conditions of item quality and test termination rule. The proposed approach is compared with the random selection method and an approximate approach based on dichotomized responses. The results show that under the MC-DINA model, JSD improves the attribute classification accuracy significantly by considering the information from distractors, even with a very short test length. This result has important implications in practical classroom settings as it can allow for dramatically reduced testing times, thus resulting in more targeted learning opportunities.

Components of the item selection algorithm in computerized adaptive testing.

Computerized adaptive testing (CAT) greatly improves measurement efficiency in high-stakes testing operations through the selection and administration of test items with the difficulty level that is most relevant to each individual test taker. This paper explains the 3 components of a conventional CAT item selection algorithm: test content balancing, the item selection criterion, and item exposure control. Several noteworthy methodologies underlie each component. The test script method and constrained CAT method are used for test content balancing. Item selection criteria include the maximized Fisher information criterion, the b-matching method, the a-stratification method, the weighted likelihood information criterion, the efficiency balanced information criterion, and the Kullback-Leibler information criterion. The randomesque method, the Sympson-Hetter method, the unconditional and conditional multinomial methods, and the fade-away method are used for item exposure control. Several holistic approaches to CAT use automated test assembly methods, such as the shadow test approach and the weighted deviation model. Item usage and exposure count vary depending on the item selection criterion and exposure control method. Finally, other important factors to consider when determining an appropriate CAT design are the computer resources requirement, the size of item pools, and the test length. The logic of CAT is now being adopted in the field of adaptive learning, which integrates the learning aspect and the (formative) assessment aspect of education into a continuous, individualized learning experience. Therefore, the algorithms and technologies described in this review may be able to help medical health educators and high-stakes test developers to adopt CAT more actively and efficiently.

Electronic document flow between project, production and operating enterprises in the context of Industry 4.0 digital economy

Subject of Research.We propose organizational schemesforelectronic document flow of design, program and technological documentation for instrument-making items in the companies, which run according to the rules of Industry 3.0 and according to the digital design and production rules of Industry 4.0. The most important point in the item life cycle is the exploitation documentcirculation at the stage of «designing-manufacturing-exploitation». Method.We used organization methods of project and manufacturing works for the instrument-making items production within the conditions of pilot-scale and full-scale production based on the general theory of automation design. Main Results.It is shown that the effect of the most advanced industrial technology implementation in the electronic document flow can be achieved through the application of project and manufacturing company archives based on the technologies of the Internet of things and cloud-based computing as means of technical support. The most important research result is the development of some organizational schemes for interaction between manufacturer, designer and exploiter in the view of exploitation documentation for the instrument-making item usage. Practical Relevance.The results of this research can be implemented for the creation of instrument-making (machine manufacturing) automation design algorithms for digital production, operating within the conditions of Industry 4.0 digital economy. The presented organizational schemes of the electronic exploitation document flow can be reasonably implemented for the practice of situational support in the exploitation of instrument-making items

Item usage in a multidimensional computerized adaptive test (MCAT) measuring health-related quality of life

PurposeExamining item usage is an important step in evaluating the performance of a computerized adaptive test (CAT). We study item usage for a newly developed multidimensional CAT which draws items from three PROMIS domains, as well as a disease-specific one.MethodsThe multidimensional item bank used in the current study contained 194 items from four domains: the PROMIS domains fatigue, physical function, and ability to participate in social roles and activities, and a disease-specific domain (the COPD-SIB). The item bank was calibrated using the multidimensional graded response model and data of 795 patients with chronic obstructive pulmonary disease. To evaluate the item usage rates of all individual items in our item bank, CAT simulations were performed on responses generated based on a multivariate uniform distribution. The outcome variables included active bank size and item overuse (usage rate larger than the expected item usage rate).ResultsFor average θ-values, the overall active bank size was 9–10%; this number quickly increased as θ-values became more extreme. For values of −2 and +2, the overall active bank size equaled 39–40%. There was 78% overlap between overused items and active bank size for average θ-values. For more extreme θ-values, the overused items made up a much smaller part of the active bank size: here the overlap was only 35%.ConclusionsOur results strengthen the claim that relatively short item banks may suffice when using polytomous items (and no content constraints/exposure control mechanisms), especially when using MCAT.

Item-network-based collaborative filtering: A personalized recommendation method based on a user's item network

Recommendation systems are becoming important with the increased availability of online services. A typical approach used in recommendations is collaborative filtering. However, because it largely relies on external relations, such as items-to-items or users-to-users, problems occur when the relations are biased or insufficient. Focusing on that limitation, we here suggest a new method, item-network-based collaborative filtering, which recommends items through four steps. First, the system constructs item networks based on users’ item usage history and calculates three types of centrality: betweenness, closeness, and degree. Next, the system secures significant items based on the betweenness centrality of the items in each user's item network. Then, by using the closeness and degree centrality of the secured items, the algorithm predicts preference scores for items and their rank orders from each user's perspective. In the last step, the system organizes a recommendation list based on the predicted scores. To evaluate the performance of our system, we applied it to a sample dataset of 196 Last.fm users’ listening history and compared the results with those from existing collaborative filtering methods. The results showed that the suggested method performed better than the basic item-based and user-based collaborative filtering methods in terms of Accuracy, Recall, and F1 scores for top-k recommendations. This indicates that an individual user's item relations can be utilized to remedy the problems occurring when the external relations are biased or insufficient.