Pairs Of Items Research Articles

Matrix Factorization and Prediction for High-Dimensional Co-Occurrence Count Data via Shared Parameter Alternating Zero Inflated Gamma Model

High-dimensional sparse matrix data frequently arise in various applications. A notable example is the weighted word–word co-occurrence count data, which summarizes the weighted frequency of word pairs appearing within the same context window. This type of data typically contains highly skewed non-negative values with an abundance of zeros. Another example is the co-occurrence of item–item or user–item pairs in e-commerce, which also generates high-dimensional data. The objective is to utilize these data to predict the relevance between items or users. In this paper, we assume that items or users can be represented by unknown dense vectors. The model treats the co-occurrence counts as arising from zero-inflated Gamma random variables and employs cosine similarity between the unknown vectors to summarize item–item relevance. The unknown values are estimated using the shared parameter alternating zero-inflated Gamma regression models (SA-ZIG). Both canonical link and log link models are considered. Two parameter updating schemes are proposed, along with an algorithm to estimate the unknown parameters. Convergence analysis is presented analytically. Numerical studies demonstrate that the SA-ZIG using Fisher scoring without learning rate adjustment may fail to find the maximum likelihood estimate. However, the SA-ZIG with learning rate adjustment performs satisfactorily in our simulation studies.

The quadratic knapsack problem with setup

The Quadratic Knapsack Problem is a well-known generalization of the classical 0-1 knapsack problem, in which any pair of items produces a pairwise profit if both are selected. Another relevant generalization of the knapsack problem is the Knapsack Problem with Setup, in which the items are partitioned into classes, the items of a class can only be inserted into the knapsack if the corresponding class is activated, and activating a class involves a setup cost and a setup capacity reduction.Despite a rich literature on these two problems, their obvious generalization, i.e., the Quadratic Knapsack Problem with Setup, was never investigated so far. We discuss applications, mathematical models, deterministic matheuristic algorithms, and computationally evaluate their performance.

Rasch validation of the revised body awareness rating questionnaire (BARQ-R) in adults with and without musculoskeletal pain

BackgroundThe Revised Body Awareness Rating Questionnaire (BARQ-R) is a self-report measure of body awareness. First aim: evaluate the structural validity of BARQ-R with Rasch analysis in community-dwelling Americans with and without musculoskeletal pain. Subaim: validate a Rasch analysis of BARQ-R done in Norwegian adults with musculoskeletal pain, through a secondary analysis in our sample of Americans with musculoskeletal pain.MethodsBARQ-R has 12 items with scores ranging from 0 (completely disagree) to 3 (completely agree), with higher total scores reflecting lower degrees of body awareness. Through Rasch analysis, we evaluated unidimensionality, item hierarchy, and structural validity with item and person fit, targeting, person separation reliability (PSR), local item dependence (LID), differential item functioning (DIF), and principal components analysis of residuals (PCAR).ResultsWe recruited 623 adults with and without musculoskeletal pain (average age = 50.27 ± 17.25 years). After rescoring 1 item and deleting 3 items, the 9-item Rasch-based BARQ-R had no misfitting items, the hierarchical ordering of the items followed clinical expectations, 3 (0.48%) misfitting persons, person mean location: -0.62 ± 1.03 logits (max -0.53, min 0.72 logits), minimal floor effect (1.93%) and ceiling effect (0.48%), no DIF, and PSR = 0.72. LID was found in 5 item pairs. The PCAR’s eigenvalue was 2.18. The secondary Rasch analysis in 152 adults with musculoskeletal pain (average age = 52.26 ± 16.13 years), demonstrated that, after rescoring 2 items, BARQ-R had no misfitting items and only 2 (1.32%) misfitting persons, good targeting (person mean location: -0.36 ± 0.88 logits), minimal floor effect (0.01%), no ceiling effect (0.00%), and PSR = 0.75. LID was found in 6 item pairs. The PCAR’s eigenvalue was 2.47.ConclusionsBARQ-R had good item and person fit. PSR with items covering a limited logit range suggests that differing levels of body awareness are measured with only modest precision. Adding and revising items to cover a wider range of body awareness and to better address concepts of internal body awareness and body movements would improve BARQ-R’s utility. Further analyses are needed before BARQ-R’s use for research or in the clinic. In addition, future BARQ-R Rasch validation is needed in other populations with body awareness deficits, such as stroke or spinal cord injury.

Psychometric Properties of the Saudi Version of the Mathematics Confidence Scale using the Modern Test Theory

The aim of the study was to analyze the psychometric properties of the Arabic version of the Math Confidence Rating Scale (MCS), using Item Response Theory (IRT) and Exploratory Factor Analysis (EFA). The participants were (315) from Imam Mouhammad Ibn Saud Islamic University; They are distributed into (92) female and (223) male, selected through a convenience sampling. The EFA models indicated that the one-dimensional structure better represents the data, and the pairs of items were considered locally independent, IRT findings indicate that all items had a very high discrimination parameter and all items had acceptable threshold parameters, And that MCS is more informative for mid-level ability of Math confidence. The MCS in Arabic has adequate psychometric properties to be used as a short measure of Math confidence.

Development of an Inconsistent Responding Scale for the Big Five Inventory-2

The Big Five Inventory − 2 (BFI-2) is a commonly used self-report assessment of normal personality trait domains (Extraversion, Agreeableness, Conscientiousness, Negative Emotionality, Open-Mindedness) and facets. To date, however, no direct measures of response distortion have been developed for it to identify potentially invalid responses. Such distortions (e.g., careless or random responding) can adversely impact data quality. The current study developed and provided initial validation data for an inconsistent responding scale within the BFI-2 to identify careless responders using two large undergraduate samples and a community sample. To create the scale, we first identified highly correlated BFI-2 item pairs in one undergraduate sample (N = 1,461) and then computed a total score by summing the absolute differences of these item pairs. This scale, the Detection of Response Inconsistency Procedure (DRIP), differentiated randomly generated and genuine data and generally correlated as expected with personality domains and other inconsistent responding scales across samples. The DRIP also incrementally predicted random data beyond a composite of items with exceptionally high or low base rates of endorsement from the Comprehensive Infrequency/Frequency Item Repository. We provide recommendations for DRIP cut scores that can detect careless responding while balancing sensitivity and specificity.

Learning bivariate scoring functions for ranking

State-of-the-art Learning-to-Rank algorithms, e.g., λ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\lambda$$\\end{document}MART, rely on univariate scoring functions to score a list of items. Univariate scoring functions score each item independently, i.e., without considering the other available items in the list. Nevertheless, ranking deals with producing an effective ordering of the items and comparisons between items are helpful to achieve this task. Bivariate scoring functions allow the model to exploit dependencies between the items in the list as they work by scoring pairs of items. In this paper, we exploit item dependencies in a novel framework—we call it the Lambda Bivariate (LB) framework—that allows to learn effective bivariate scoring functions for ranking using gradient boosting trees. We discuss the three main ingredients of LB: (i) the invariance to permutations property, (ii) the function aggregating the scores of all pairs into the per-item scores, and (iii) the optimization process to learn bivariate scoring functions for ranking using any differentiable loss functions. We apply LB to the λ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\lambda$$\\end{document}Rank loss and we show that it results in learning a bivariate version of λ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\lambda$$\\end{document}MART—we call it Bi-λ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\lambda$$\\end{document}MART—that significantly outperforms all neural-network-based and tree-based state-of-the-art algorithms for Learning-to-Rank. To show the generality of LB with respect to other loss functions, we also discuss its application to the Softmax loss.

Towards similar alignment and unique uniformity in collaborative filtering

Representation learning, with its desired properties, including Alignment and Uniformity, has recently emerged as an effective approach in collaborative filtering (CF) for recommender systems. Alignment measures the distance between positive pairs, and uniformity describes the distribution of all samples on the unit hypersphere. Despite the effectiveness of existing research, the challenges of insufficient alignment and uniformity bias persist in studies on the desired properties of recommender systems: (1) Sparse interaction information leads to an insufficient alignment representation. (2) Calculating the uniformity loss based on duplicate samples introduces bias, which affects the overall representation uniformity.Building on aforementioned challenges, we propose a Similar Alignment and Unique Uniformity model called SUAU. SUAU mitigates insufficient alignment by introducing additional item-similar item pairs for model training and employing a uniqueness strategy to prevent uniformity bias. More specifically, we propose a similar items generation method that identifies the most similar items based on the interaction behavior of the original items’ related users to form item-similar item pairs. The experimental results obtained on three highly sparse datasets demonstrate that the propose model achieves superior alignment and uniformity compared with state-of-the-art models, notably surpassing the best CF methods in terms of recommendation performance. We have open-sourced the source code on a publicly accessible website: https://github.com/ZzYUuuu/SUAU.

Rasch Measurement Theory (RMT) Analyses of the Huntington's Disease Everyday Functioning (Hi-DEF) to Evaluate Item Fit and Performance.

The Huntington's Disease (HD) Everyday Functioning (Hi-DEF) is a new patient-reported outcome (PRO) instrument designed to measure the impact of cognitive impairment on daily functioning in the early stages of HD. To assess the measurement properties and finalize item content of the Hi-DEF. A cross-sectional, observational psychometric validation study was conducted among individuals with early stages of HD at 9 US centers of excellence. Rasch Measurement Theory (RMT) analysis of the initial draft version of the Hi-DEF (47 items) and subscales (i.e., 'Home', 'At work', 'Driving', and 'Communication') was conducted to examine measurement properties including sample-to-scale targeting, suitability of response scale (ordering of response thresholds), scale cohesiveness (item fit), local independence, and person fit. 151 participants (mean age 47 years (SD 12), 59% female) were included. Seven items were removed based on dependency and item fit. The remaining 40-item version of the Hi-DEF demonstrated good measurement properties. Across the four subscales, targeting ranged from 49-70% (72% full scale), reliability ascertained by person separation index ranged from 0.53-0.87 (0.92 full scale), response scales were ordered for 25-100% of items (75% full scale), 0-12% items displayed misfit (2% full scale), and 0-1% (2% full scale) item pairs displayed dependency. Our study supports the psychometric integrity of the Hi-DEF as a reliable and valid new PRO instrument designed to assess the impact of cognitive impairment on daily functioning in the early stages of HD. Future work will evaluate the external validity and utility in clinical trial applications.

Modeling Sequences as Star Graphs to Address Over-Smoothing in Self-Attentive Sequential Recommendation

Self-attention (SA) mechanisms have been widely used in developing sequential recommendation (SR) methods, and demonstrated state-of-the-art performance. However, in this article, we show that self-attentive SR methods substantially suffer from the over-smoothing issue that item embeddings within a sequence become increasingly similar across attention blocks. As widely demonstrated in the literature, this issue could lead to a loss of information in individual items, and significantly degrade models’ scalability and performance. To address the over-smoothing issue, in this article, we view items within a sequence constituting a star graph and develop a method, denoted as \(\mathop{\mathtt{MSSG}}\limits\) , for SR. Different from existing self-attentive methods, \(\mathop{\mathtt{MSSG}}\limits\) introduces an additional internal node to specifically capture the global information within the sequence, and does not require information propagation among items. This design fundamentally addresses the over-smoothing issue and enables \(\mathop{\mathtt{MSSG}}\limits\) a linear time complexity with respect to the sequence length. We compare \(\mathop{\mathtt{MSSG}}\limits\) with eleven state-of-the-art baseline methods on six public benchmark datasets. Our experimental results demonstrate that \(\mathop{\mathtt{MSSG}}\limits\) significantly outperforms the baseline methods, with an improvement of as much as 10.10%. Our analysis shows the superior scalability of \(\mathop{\mathtt{MSSG}}\limits\) over the state-of-the-art self-attentive methods. Our complexity analysis and runtime performance comparison together show that \(\mathop{\mathtt{MSSG}}\limits\) is both theoretically and practically more efficient than self-attentive methods. Our analysis of the attention weights learned in SA-based methods indicates that on sparse recommendation data, modeling dependencies in all item pairs using the SA mechanism yields limited information gain, and thus, might not benefit the recommendation performance. Our source code and data are publicly accessible through GitHub .

The Chinese Mandarin COMHON Index and Braden Scale to assess pressure injury risk in intensive care: An inter-rater reliability and convergent validity study

BackgroundThe COMHON Index is an intensive-care-specific pressure injury risk assessment tool, which has demonstrated promising psychometric properties. It has been translated into Chinese Mandarin but requires inter-rater reliability testing and comparison to the standard care instrument (Braden Scale) before clinical use. ObjectivesThis study aimed to test and compare the inter-rater reliability and convergent validity of the Chinese Mandarin versions of the COMHON Index and Braden Scale. MethodsThe study was conducted in a Chinese comprehensive intensive care unit. Based on a sample size calculation, five registered nurse raters with at least 6-months experience independently conducted risk assessments for 20 adult patients using both the COMHON Index and Braden Scale. Intraclass correlations (ICC) for inter-rater reliability, standard errors of measurement (SEM), and minimally detectable change (MDC) were calculated. Convergent validity was assessed using Pearson Product Moment Correlation for sum scores and Spearman's rho for subscales. ResultsInter-rater reliability of COMHON Index and Braden Scale sum scores was very high (ICC [1,1] = 0.973; [95% confidence interval 0.949–0.988]; SEM 0.54; MDC 1.50) and high (ICC [1,1] = 0.891; [95% confidence interval 0.793-0.951]; SEM 0.93; MDC 2.57), respectively. All COMHON-Index subscales demonstrated ICC values >0.6, whereas two Braden Scale subscales (Mobility, Activity) were below this threshold. Instrument sum scores were strongly correlated (Pearson's r = −0.76 [r2 = 0.58]; p < 0.001), as were three subscale item pairs (mobility rs= −0.56 [r2 = 0.32]; nutrition rs= −0.63 [r2 = 0.39]; level of consciousness/sensory perception rs= −0.67 [r2 = 0.45] p < 0.001). ConclusionBoth the COMHON Index and Braden Scale demonstrated high levels of inter-rater reliability and measured similar constructs. However, the COMHON Index demonstrated superior inter-rater reliability and the results suggest that it better detects changes in patient condition and subsequently pressure injury risk. Further testing is recommended.

Utility-Oriented Reranking with Counterfactual Context

As a critical task for large-scale commercial recommender systems, reranking rearranges items in the initial ranking lists from the previous ranking stage to better meet users’ demands. Foundational work in reranking has shown the potential of improving recommendation results by uncovering mutual influence among items. However, rather than considering the context of initial lists as most existing methods do, an ideal reranking algorithm should consider the counterfactual context— the position and the alignment of the items in the reranked lists . In this work, we propose a novel pairwise reranking framework, Utility-oriented Reranking with Counterfactual Context (URCC), which maximizes the overall utility after reranking efficiently. Specifically, we first design a utility-oriented evaluator, which applies Bi-LSTM and graph attention mechanism to estimate the listwise utility via the counterfactual context modeling. Then, under the guidance of the evaluator, we propose a pairwise reranker model to find the most suitable position for each item by swapping misplaced item pairs. Extensive experiments on two benchmark datasets and a proprietary real-world dataset demonstrate that URCC significantly outperforms the state-of-the-art models in terms of both relevance-based metrics and utility-based metrics.

The influence of representations on task difficulty in organic chemistry: an exploration using a novel paired-items test instrument

ABSTRACT The intricate relationship between representational competence and content knowledge in organic chemistry has been widely debated, and the ways in which representations contribute to task difficulty, particularly in assessment, remain unclear. This paper presents a multiple-choice test instrument for assessing individuals’ knowledge of fundamental concepts – such as nomenclature and basic reaction mechanisms – in organic chemistry using item pairs with and without representations. We conducted a pilot test of the instrument with 55 undergraduate students and administered the final version to 357 undergraduate chemistry students from seven German universities. We used the Rasch methodology for psychometric analysis to identify sound and reliable test properties. A task-difficulty comparison revealed no differences between the item pairs with and those without representations. Hierarchical regression suggests that the complexity and cognitive processes involved in a task influence its difficulty. Our data suggest that the relationship between representational competence and content knowledge may not be disentangled when item difficulty is considered. We argue that representational competence and content knowledge might not represent separate constructs, which is in line with the literature on representational dilemmas: representations cannot be taught without content knowledge, and content knowledge cannot be taught without the use of representations.

Using natural language processing to facilitate the harmonisation of mental health questionnaires: a validation study using real-world data

BackgroundPooling data from different sources will advance mental health research by providing larger sample sizes and allowing cross-study comparisons; however, the heterogeneity in how variables are measured across studies poses a challenge to this process.MethodsThis study explored the potential of using natural language processing (NLP) to harmonise different mental health questionnaires by matching individual questions based on their semantic content. Using the Sentence-BERT model, we calculated the semantic similarity (cosine index) between 741 pairs of questions from five questionnaires. Drawing on data from a representative UK sample of adults (N = 2,058), we calculated a Spearman rank correlation for each of the same pairs of items, and then estimated the correlation between the cosine values and Spearman coefficients. We also used network analysis to explore the model’s ability to uncover structures within the data and metadata.ResultsWe found a moderate overall correlation (r = .48, p < .001) between the two indices. In a holdout sample, the cosine scores predicted the real-world correlations with a small degree of error (MAE = 0.05, MedAE = 0.04, RMSE = 0.064) suggesting the utility of NLP in identifying similar items for cross-study data pooling. Our NLP model could detect more complex patterns in our data, however it required manual rules to decide which edges to include in the network.ConclusionsThis research shows that it is possible to quantify the semantic similarity between pairs of questionnaire items from their meta-data, and these similarity indices correlate with how participants would answer the same two items. This highlights the potential of NLP to facilitate cross-study data pooling in mental health research. Nevertheless, researchers are cautioned to verify the psychometric equivalence of matched items.

ATLAS: Mapping ATtention’s Location And Size to probe five modes of serial and parallel search

Conventional visual search tasks do not address attention directly and their core manipulation of ‘set size’ – the number of displayed items – introduces stimulus confounds that hinder interpretation. However, alternative approaches have not been widely adopted, perhaps reflecting their complexity, assumptions, or indirect attention-sampling. Here, a new procedure, the ATtention Location And Size (‘ATLAS’) task used probe displays to track attention’s location, breadth, and guidance during search. Though most probe displays comprised six items, participants reported only the single item they judged themselves to have perceived most clearly – indexing the attention ‘peak’. By sampling peaks across variable ‘choice sets’, the size and position of the attention window during search was profiled. These indices appeared to distinguish narrow- from broad attention, signalled attention to pairs of items where it arose and tracked evolving attention-guidance over time. ATLAS is designed to discriminate five key search modes: serial-unguided, sequential-guided, unguided attention to ‘clumps’ with local guidance, and broad parallel-attention with or without guidance. This initial investigation used only an example set of highly regular stimuli, but its broader potential should be investigated.

Label-semantics enhanced multi-layer heterogeneous graph convolutional network for Aspect Sentiment Quadruplet Extraction

Aspect-based sentiment analysis (ABSA) is a pivotal area within natural language processing (NLP) that focuses on extracting fine-grained sentiment information from text. A particularly demanding task within ABSA is Aspect Sentiment Quadruplet Extraction (ASQE), which aims to identify quadruplets comprising aspect terms, their corresponding opinion terms, sentiment polarity, and categories. This level of analysis is crucial for downstream applications such as sentiment monitoring and user experience research, offering nuanced insights into textual data. However, current methodologies fall short of fully leveraging the rich linguistic features of sentences and the semantic information of labels embedded within sentences. To address this gap, this paper introduces a novel framework, the Label-Semantics Enhanced Multi-layer Heterogeneous Graph Convolutional Network (LSEMH-GCN), specifically designed for ASQE. Our model integrates sentence linguistic features and label semantics to construct a graph neural network tailored for this task. It employs a multi-layer graph convolutional network that synergizes various linguistic features, and utilizes Biaffine attention to enrich the label probability distribution for token pairs with semantic label information. Furthermore, our approach introduces a token pair vector concatenation strategy alongside an advanced asymmetric label tagging scheme to enhance quadruplet extraction. Comprehensive evaluations on benchmark datasets reveal that LSEMH-GCN significantly surpasses existing state-of-the-art models, establishing a new benchmark for ASQE. Our model achieves an average F1 score improvement of 15.52% and 12.30% on the Restaurant-ACOS and Laptop-ACOS datasets, respectively.

Adaption and validation of the Rwandese version of the Young Mania Rating Scale to measure the severity of a manic or hypomanic episode

BackgroundBipolar Disorder is one of the most incapacitating diseases among young persons, leading to cognitive and functional impairment and raised mortality, particularly death by suicide. Managing a manic episode and developing new and more effective treatment modalities requires sensitive and reliable instruments. This study aims to translate the English version of the YMRS questionnaire into Kinyarwanda, adapt it to the Rwandan context, and assess its validity.MethodsThe original English version of The Young Mania Rating Scale questionnaire was translated into Kinyarwanda. The translation process followed a standardized approach, including back-translation, cross-cultural adaptation, and final adjustments. A total of 130 inpatients with bipolar disorder in a manic episode from CARAES Ndera Teaching Hospital were included. The descriptive statistics and test–retest correlations were carried out, as well as the CFA for validation and Rasch-analysis.ResultsThe Rwandese version of The Young mania rating scale had an adequate internal consistency (Cronbach’s alpha = 0.90). Item 11 provided the lowest standardized loading in both ratings (0.51 and 0.55). The second lowest loading involved the highly correlated item pairs 5 & 9, with item 5 loading 0.51 in rating 1 and item 9 loading 0.57 in rating 2. The remaining loadings ranged from 0.59 to 0.79. This relatively narrow range indicated that a fit to a Rasch model was plausible if excluding item 11.ConclusionThe findings demonstrate that the translated YMRS, the R-YMRS, can be used as a reliable and valid instrument for assessing mania in the Rwandese population in clinical and research settings. However, the results supported using an unweighted total score of 32 and removing items 5, 9, and 11. Studies on this revised scale with an added interview guide for less-trained clinical staff are recommended.

Awareness and Calibration: The Role of Descriptive Norms and Social Desirability in Accurate IAT Score Predictions of Food Items vs. Social Groups.

Extending research that people are able to predict the patterns of their social group biases on Implicit Association Tests (IATs), we let participants predict and complete IATs toward five different food item pairs and compared the results with the social-groups domain. Participants predicted the patterns of their IAT scores with similar accuracy in both domains, suggesting similar internal awareness (evidenced by comparable within-subjects correlations), even though food evaluations followed less descriptively-normative patterns. At the same time, participants were better-calibrated in communicating their evaluations in the domain of food than social groups (evidenced by higher between-subjects correlations). This discrepancy may partly stem from participants' tendency to refrain from using harsh labels when predicting social group biases, despite their demonstrated awareness of them: IAT scores predicted as "moderate" or "strong" for food preferences tended to be labeled "mild" for social groups. Discussion centers on the importance of distinguishing between awareness and calibration.

Automatic Enemy Identification—Are We There Yet?

Enemy items refer to any two items that should not appear on the same test form (Weir, 2019). Accurately identifying enemy pairs is critical for ensuring the quality and fairness of exams, but it can also be challenging and time-consuming given the large number of possible item pairs in the exam item bank. Various enemy identification approaches have been explored to automate or semi-automate this task. In this process, the critical component is the encoding technique. The better the encoding technique captures the meaning of the sentences, the more accurate the similarity index and enemy classification results will be. This study focuses on evaluating the performance of a transformer-based model against the results from a string-based vector-space model (VSM) encoding technique under different research conditions for multiple-choice and multiple-response items used in a foundational IT certification exam. The results suggest that when using sufficient representative training data and conducting fine-tuning, the transformer-based model significantly outperforms the VSM model for enemy identification.

A Bayesian active learning approach to comparative judgement within education assessment

Assessment is a crucial part of education. Traditional marking is a source of inconsistencies and unconscious bias, placing a high cognitive load on the assessors. One approach to address these issues is comparative judgement (CJ). In CJ, the assessor is presented with a pair of items of work, and asked to select the better one. Following a series of comparisons, a rank for any item may be derived using a ranking model, for example, the Bradley-Terry model, based on the pairwise comparisons. While CJ is considered to be a reliable method for conducting marking, there are concerns surrounding its transparency, and the ideal number of pairwise comparisons to generate a reliable estimation of the rank order is not known. Additionally, there have been attempts to generate a method of selecting pairs that should be compared next in an informative manner, but some existing methods are known to have created their own bias within results inflating the reliability metric used within the process. As a consequence, a random selection approach is usually deployed.In this paper, we propose a novel Bayesian approach to CJ (which we call BCJ) for determining the ranks of a range of items under scrutiny alongside a new way to select the pairs to present to the marker(s) using active learning, addressing the key shortcomings of traditional CJ. Furthermore, we demonstrate how the entire approach may provide transparency by providing the user insights into how it is making its decisions and, at the same time, being more efficient. Results from our synthetic experiments confirm that the proposed BCJ combined with entropy-driven active learning pair-selection method is superior (i.e. always equal to or significantly better) than other alternatives, for example, the traditional CJ method with differing selection methods such as uniformly random, or the popular no repeating pairs where pairs are selected in a round-robin fashion. We also find that the more comparisons that are conducted, the more accurate BCJ becomes, which solves the issue the current method has of the model deteriorating if too many comparisons are performed. As our approach can generate the complete predicted rank distribution for an item, we also show how this can be utilised in probabilistically devising a predicted grade, guided by the choice of the assessor. Finally, we demonstrate our approach on a real dataset on assessing GCSE (UK school-level) essays, highlighting the advantages of BCJ over CJ.

An examination of individual differences in levels of processing

ABSTRACT The present study examined individual differences in levels of processing. Participants completed a cued recall task in which they made either rhyme or semantic judgements on pairs of items. Pupillary responses during encoding were recorded as a measure of the allocation of attentional effort and participants completed multiple measures of working and long-term memory. The results suggested levels of processing effect in both accuracy and pupillary responses with deeper levels of processing demonstrating higher accuracy and larger pupillary responses than shallower levels of processing. Most participants demonstrated levels of processing effect, but there was substantial variability in the size of the effect. Variation in levels of processing was positively related to individual differences in long-term memory and the magnitude of the pupillary levels of processing effect, but not working memory. These results suggest that some of the variation in levels of processing is likely due to individual differences in the allocation of attentional effort (particularly to items processed deeply) during encoding.