True Score Research Articles

Reducing the number of impulses in video head impulse testing – It's the quality not the numbers

ObjectivesThe video-Head-Impulse-Test (vHIT) is widely used to evaluate vestibular function. Nevertheless, there is no consensus on the necessary or ideal number of impulses performed for robust VOR gains. Therefore, the aim of our study is to analyze how many impulses are needed to receive reliable VOR gains in difficult testing situations like testing children younger than 48 months and in children with vestibular loss. MethodsA retrospective data analysis was performed in which existing vHIT results of 25 healthy children aged 5–48 months were included as well as vHIT results of 25 children with vestibular loss aged 2–16 years. Descriptive data analysis was performed and further statistical analysis was conducted to determine if the number of head impulses could be reduced using internal consistency (Cronbach's alpha) and paired t-test. ResultsMedian gain was 0.95 (±0.16) for impulses to the right and 0.97 (±0.16) for impulses to the left in healthy children and ranged from 0.01 (±0.11) to 0.75 (+/- 0.23) in children with vestibular hypfunction. Analyzing Cronbach's Alpha, a 99.6% (α = 0.996) true score variance was achieved when two impulses were performed to the right and 98,1% to the left in healthy children and 99.9% in children with impaired vestibular function. ConclusionThese results indicate that two high velocity artifact-free impulses from an experienced tester are sufficient to evaluate vestibular function in difficult vHIT testing conditions like testing very young children. Further impulses do not improve test reliability.

Investigating the Test-Retest Reliability and Validity of Hand-Held Dynamometry for Measuring Knee Strength in Older Women with Knee Osteoarthritis.

Purpose: Hand-held dynamometry (HHD) can be used to evaluate strength when gold-standard isokinetic dynamometry (IKD) is not feasible. HHD is useful for measuring lower limb strength in a healthy population; however, its reliability and validity in individuals with knee osteoarthritis (OA) has received little attention. In this research, we examined the test-retest reliability and validity of HHD in older women with knee OA. We also examined the associations between reliability and symptom and disease severity. Method: A total of 28 older women with knee OA completed knee extension and flexion exertions measured using HHD and IKD. Intra-class correlation coefficients (ICC2,3), standard error of measurement, and minimal detectable change were calculated. Correlation coefficients and regressions evaluated the relationships between inter-trial differences and symptom and disease severity. Results: High test-retest reliability was demonstrated for both exertions with each device (ICC2,3 = 0.83-0.96). Variance between trials was not correlated with OA symptoms. Criterion validity was good (ICC2,3 = 0.76), but extension yielded lower agreement than flexion. Regression analysis demonstrated that true strength can be predicted from HHD measurements. Conclusions: HHD is a reliable tool for capturing knee extension and flexion in individuals with OA. Because of lower agreement, HHD might be best suited for evaluating within-subject strength changes rather than true strength scores. However, gold-standard extension strength magnitudes may reasonably be predicted from regression equations (r 2=0.82).

Prediction of Flares within 10 Days before They Occur on the Sun

Abstract Prediction of solar flares due to the effects on Earth and satellites is an important topic for scientists. We develop a method and a tool for flare prediction by applying the support vector machine classifier to unique and independent Zernike moments extracted from active region (AR) images. In the analysis, we used the Helioseismic and Magnetic Imager (HMI) line-of-sight magnetograms, the Atmospheric Imaging Assembly (AIA) ultraviolet (UV at 1600 Å) and extreme ultraviolet (EUV at 304, 171, 193, 211, 335, 94, and 131 Å) images for a period of eight years of the solar cycle 24 (2010 June to 2018 September). The power-law behavior for the frequency distribution of the large flaring time window—the time interval between the occurrence of an AR and first large flare (X- and M-class) therein—indicated that most of the large flares appeared within 150 hr. The True Skill Score (TSS) metric for the performance of the win classifier that (uses the outputs of the HMI and AIA at 193, 211, 94, and 131 Å classifiers) was obtained as 0.86 ± 0.04. We also showed that the maximum value of the TSS for prediction of large flares for the win classifiers was about 0.95 ± 0.03 on the flaring day and decreased to 0.76 ± 0.1 within 4 to 10 days before flaring.

Peer-Rated Organizational Citizenship Behavior

Abstract. Organizational Citizenship Behavior (OCB) is valuable to organizations and has become an important focus of employee performance evaluation. Employees’ peers may be particularly well-situated to rate their OCB. We investigated the proportion of variance in peer-rated OCB attributable to the ratee (true score) versus the rater (rater bias). Furthermore, we investigated whether these proportions were affected by the familiarity of the peer with the ratee. We found that high familiarity was associated with a greater proportion of ratee variance (.43 vs. .18), and a lower proportion of rater bias (.30 vs. .51), than was the case with low-to-moderate familiarity. Thus, when choosing peers as raters of OCB, there may be value in carefully considering the peers’ familiarity with the ratees.

Modeling of Item Response Functions Under the D-Scoring Method.

This study presents new models for item response functions (IRFs) in the framework of the D-scoring method (DSM) that is gaining attention in the field of educational and psychological measurement and largescale assessments. In a previous work on DSM, the IRFs of binary items were estimated using a logistic regression model (LRM). However, the LRM underestimates the item true scores at the top end of the D-scale (ranging from 0 to 1), especially for relatively difficult items. This entails underestimation of true D-scores, inaccuracy in the estimates of their standard errors, and other psychometric issues. The inverse-regression adjustments used to fix this problem are too complicated for regular applications of the DSM and not in line with its simplicity. This issue is resolved with the IRF models proposed in this study, referred to as rational function models (RFMs) with one parameter (RFM1), two parameters (RFM2), and three parameters (RFM3). The proposed RFMs are discussed and illustrated with simulated and real data.

Efficiency of Average Treatment Effect Estimation When the True Propensity Is Parametric

It is well known that efficient estimation of average treatment effects can be obtained by the method of inverse propensity score weighting, using the estimated propensity score, even when the true one is known. When the true propensity score is unknown but parametric, it is conjectured from the literature that we still need nonparametric propensity score estimation to achieve the efficiency. We formalize this argument and further identify the source of the efficiency loss arising from parametric estimation of the propensity score. We also provide an intuition of why this overfitting is necessary. Our finding suggests that, even when we know that the true propensity score belongs to a parametric class, we still need to estimate the propensity score by a nonparametric method in applications.

Quality evaluation of decoction pieces of Rhizoma Atractylodis Macrocephalae by near infrared spectroscopy coupled with chemometrics.

To establish a fast, simple and reliable method for quality evaluation of decoction pieces of Rhizoma Atractylodis Macrocephalae (referred as BZ below) by near infrared spectroscopy coupled with chemometrics. Twelve batches of raw medicinal materials of BZ were collected from three main producing location in China. According to the Pharmacopoeia of the People's Republic of China, these raw decoction pieces were stir-fried in wheat bran using a stir-frying machine for 3, 6, 9, 12 and 15 min, respectively. The resulted 60 samples were categorized into three classes (i.e., light, moderate and dark) by experienced pharmacists according to their surface color. After that, these slices were smashed to acquire near infrared spectra and to determine the contents of atractylenolide I, II and III by HPLC method. Qualitative and quantitative models were constructed to relate the spectra to the color labels and to the contents of three atractylenolides. Various chemometrics methods, including calibration methods like principal component analysis, partial least squares discriminant analysis (PLS-DA) and partial least squares regression (PLSR), spectra pretreatment methods like standard normal variate, multiplicative scatter correction, derivation and smoothing, feature selection methods like particle swarm optimization, genetic algorithm (GA) and other fourteen methods were compared in detail. The PLS-DA models were evaluated by jackknife tests with calculating parameters such as error rate (ERR), true positive rate (TPR), true negative rate (TNR) and F1 score, meanwhile the PLSR models were evaluated by five fold cross-validation tests with calculating parameters such as coefficients of determination (R2), root mean square error (RMSE), mean absolute error (MAE), and residual predictive deviation (RPD). The PLS-DA models with spectra pretreated by 1D5S or 1D9S and wavelengths selected by InfFS, Relief-F, MutInfFS, fisher or CFS performed best, yielding 0.00 of ERR, 1.00 of TPR, 1.00 of TNR, and 1.00 of F1 for all three classes. As for quantitative models, the PLSR models by 1D5S spectra pretreatment and GA wavelengths selection performed best, where R2C and R2P were all >0.95, RMSEC and RMSEP were all <0.04%, MAEC and MAEP were all <0.04%, and RPD were all >5. The present qualitative and quantitative models can be successfully used to distinguish the degree of suitability of processed BZ, and to determine the contents of three atractylenolides, which thus are of great help for quality evaluation and control of processed BZ and other decoction pieces.

Indices of Subscore Utility for Individuals and Subgroups Based on Multivariate Generalizability Theory.

Conventional methods for evaluating the utility of subscores rely on traditional indices of reliability and on correlations among subscores. One limitation of correlational methods is that they do not explicitly consider variation in subtest means. An exception is an index of score profile reliability designated as , which quantifies the ratio of true score profile variance to observed score profile variance. has been shown to be more sensitive than correlational methods to group differences in score profile utility. However, it is a group average, representing the expected value over a population of examinees. Just as score reliability varies across individuals and subgroups, one can expect that the reliability of score profiles will vary across examinees. This article proposes two conditional indices of score profile utility grounded in multivariate generalizability theory. The first is based on the ratio of observed profile variance to the profile variance that can be attributed to random error. The second quantifies the proportion of observed variability in a score profile that can be attributed to true score profile variance. The article describes the indices, illustrates their use with two empirical examples, and evaluates their properties with simulated data. The results suggest that the proposed estimators of profile error variance are consistent with the known error in simulated score profiles and that they provide information beyond that provided by traditional measures of subscore utility. The simulation study suggests that artificially large values of the indices could occur for about 5% to 8% of examinees. The article concludes by suggesting possible applications of the indices and discusses avenues for further research.

Extreme response style bias in burn survivors.

This paper explores extreme response style to the Life Impact Burn Recovery Evaluation (LIBRE) Profile, a measure of social participation in burn survivors. We fit a Multidimensional Generalized Partial Credit Model (MGPCM) with a positive extreme response style (PERS) factor and compared this model with the original MGPCM, estimated the impact that PERS has on scores, and examined the personal characteristics that may result in an individual more likely to respond in a fashion that would inflate their true low scores. The average impact of the PERS, based upon the root mean squared bias, ranged from 0.27 to 0.50 of a standard deviation of the scale. Individuals who were older, had participated in a burn survivor support group, and had selected to self-administer the measure were less likely to have a high PERS bias that masks low scores. Future work can consider PERS when measuring the psychosocial impacts of burn injuries and other health conditions.

Standard error of measurement and smallest detectable change of the Sarcopenia Quality of Life (SarQoL) questionnaire: An analysis of subjects from 9 validation studies.

ObjectivesThe Sarcopenia Quality of Life (SarQoL) questionnaire, a sarcopenia-specific patient-reported outcome measure, evaluates quality of life with 55 items. It produces 7 domain scores and 1 overall quality of life score, all between 0 and 100 points. This study aims to contribute to the interpretation of the SarQoL scores by calculating the standard error of measurement (SEM) and smallest detectable change (SDC) in a sample of subjects from 9 studies.MethodsSubjects from 9 studies (conducted in Belgium, Brazil, Czech Republic, England, Greece, Lithuania, Poland and Spain) were included. The SEM, a measure of the error in the scores that is not due to true changes, was calculated by dividing the standard deviation of the difference between test and retest scores (SDdiff) by √2. The SDC, defined as change beyond measurement error, was calculated by multiplying SDdiff by 1.96. Bland-Altman plots were assessed for the presence of systematic errors.ResultsA total of 278 sarcopenic subjects, aged 77.67 ± 7.64 years and 61.5% women, were included. The SEM for the overall SarQoL score ranged from 0.18 to 4.20 points for the individual studies, and was 2.65 points when all subjects were analyzed together. The SDC for the overall score ranged from 0.49 to 11.65 points for the individual studies, and was 7.35 points for all subjects. The Bland-Altman plots revealed no systematic errors in the questionnaire.ConclusionThis study shows that, for individual subjects, a change in overall quality of life of at least 7.35 points (on a scale from 0 to 100) would have to be observed to confirm that a true change, beyond measurement error, has occurred. It also demonstrated that the SarQoL questionnaire is a precise instrument, with the observed scores within less than 3 points of the theoretical “true score”.

Exploratory Factor Analyses of the Intelligence and Development Scales-2: Implications for Theory and Practice.

The factor structure of the intelligence and scholastic skills domains of the Intelligence and Development Scales-2 was examined using exploratory factor analyses with the standardization and validation sample (N = 2,030, aged 5 to 20 years). Results partly supported the seven proposed intelligence group factors. However, the theoretical factors Visual Processing and Abstract Reasoning as well as Verbal Reasoning and Long-Term Memory collapsed, resulting in a five-factor structure for intelligence. Adding the three scholastic skills subtests resulted in an additional factor Reading/Writing and in Logical-Mathematical Reasoning showing a loading on abstract Visual Reasoning and the highest general factor loading. A data-driven separation of intelligence and scholastic skills is not evident. Omega reliability estimates based on Schmid-Leiman transformations revealed a strong general factor that accounted for most of the true score variance both overall and at the group factor level. The possible usefulness of factor scores is discussed.

Calibration of Measurements

Traditional notions of measurement error typically rely on a strong mean-zero assumption on the expectation of the errors conditional on an unobservable “true score” (classical measurement error) or on the data themselves (Berkson measurement error). Weakly calibrated measurements for an unobservable true quantity are defined based on a weaker mean-zero assumption, giving rise to a measurement model of differential error. Applications show it retains many attractive features of estimation and inference when performing a naive data analysis (i.e. when performing an analysis on the error-prone measurements themselves), and other interesting properties not present in the classical or Berkson cases. Applied researchers concerned with measurement error should consider weakly calibrated errors and rely on the stronger formulations only when both a stronger model's assumptions are justifiable and would result in appreciable inferential gains.

Prediction of Writing True Scores in Automated Scoring of Essays by Best Linear Predictors and Penalized Best Linear Predictors

AbstractMany assessments of writing proficiency that aid in making high‐stakes decisions consist of several essay tasks evaluated by a combination of human holistic scores and computer‐generated scores for essay features such as the rate of grammatical errors per word. Under typical conditions, a summary writing score is provided by a linear combination of the holistic scores and the feature scores. The best linear predictor (BLP) is used to approximate the true composite writing score by a linear combination of holistic scores and scores of essay features. However, because the relationship between computer‐generated feature score and human scores may depend on subgroup membership and the same scoring rules must normally be applied to all test takers, Yao, Haberman, and Zhang proposed a modified methodology of the penalized best linear predictor (PBLP) by incorporating a quadratic penalty function into the conventional BLP method. This research report contains full accounts of the BLP results as well as supplementary PBLP results to Yao et al. for three assessments of writing that aid in making high‐stakes decisions: the TOEFL iBT® Writing test, the GRE® General Analytical Writing subject test, and the Praxis® Core Academic Skills for Educators: Writing assessment. Results obtained indicate the added value in using machine features for prediction of composite true scores of essay writings and effectiveness of the penalty function in suppressing the lack of population invariance.

Job security matters: A systematic review and meta-analysis of the relationship between job security and work attitudes

AbstractThis article synthesizes public and private sector accumulated research regarding the relationship between job security and employee work attitudes (i.e., job satisfaction and organizational commitment). The present meta-analysis of 37 studies (including 45 independent samples) shows that the medium-sized associations between job security and each work attitude variables (i.e., job satisfaction and organizational commitment) were found, with true score correlations (ρ) of .327 for job satisfaction, and .253 for organizational commitment. These results highlight the significance of job security at the workplace, in shaping and enhancing attitudes of employee and job security is worth retaining in some form in the public sector, contrary to the logic of at-will employment. This meta-analysis findings also call attention to several important considerations for developing effective public job security policy.

Meta-analysis in Physical Therapy Education Research: Using the Pearson's Product–Moment Correlation Coefficient as a Measure of Effect Size

Background and Purpose. Practicing physical therapists seek the highest levels of evidence (ie, systematic review and meta-analysis) to inform clinical decision making. Performing a meta-analysis with assimilated data from a systematic review is important for comprehensive decision making in clinical and educational practice. In educational practice, the Pearson's product–moment correlation coefficient (r) is often used as the measure of effect size to examine causal relationships between variables. Our purpose was to present the best model for conducting a meta-analysis using r. Model Description and Evaluation. A meta-analysis is prescribed through five steps: (1) calculation of the effect sizes, (2) selecting the appropriate effects model and computational method, (3) applying the computational method, (4) performing heterogeneity analysis, and (5) assessing for publication bias. Outcomes. Key statistical outcomes include: (1) total sample size, (2) number of correlations, (3) mean true score correlation, (4) variance of true score correlations, (5) 80% credibility interval, and (6) 95% confidence interval. To help with the interpretation of these results, figures (ie, forest plots and funnel plots) are used. Discussion and Conclusion. Central to achieving evidence-based practice is the selection and application of high-quality research. A systematic review with random-effects meta-analysis using high-quality cohort studies can provide high-quality evidence to physical therapy educators to answer their pertinent education questions.

Observed Scores as Matching Variables in Differential Item Functioning Under the One‐ and Two‐Parameter Logistic Models: Population Results

We derive formulas for the differential item functioning (DIF) measures that two routinely used DIF statistics are designed to estimate. The DIF measures that match on observed scores are compared to DIF measures based on an unobserved ability (theta or true score) for items that are described by either the one‐parameter logistic (1PL) or two‐parameter logistic (2PL) item response theory (IRT) model. We use two different weighting schemes (uniform weights and item discrimination weights) to construct the observed score matching variable. Our results show that (a) under the 1PL item response model, the observed score‐based DIF measures always approximate the true score‐based DIF measures very closely; (b) under the 2PL model, when the observed score is the simple sum score, the observed score‐based DIF measures underestimate or overestimate the true score‐based DIF measures under the null hypothesis of no DIF when the groups are different in ability, and this bias is related to the degree to which the average discrimination parameter underestimates or overestimates the studied discrimination parameter; and (c) under the 2PL model, when the item discrimination weights are used to define the observed score, the observed score‐based DIF measures always approximate the true score‐based DIF measures very closely. These results will hold for any sets of item responses that are described by either the 1PL or 2PL IRT model.

Test Eşitlemede Çok Boyutluluğun Eş Zamanlı ve Ayrı Kalibrasyona Etkisi

The aim of this study is to investigate the effects of multidimensionality on equating results which obtained from separate and concurrent calibration methods. The study was conducted with using simulated data. In the scope of research, totally 40 simulation conditions [5 (degree of multidimensionality: 0.90, 0.75, 0.50, 0.25, and 0.00) x 2 calibration methods (separate and concurrent) x 2 (scale transformation methods: Stocking-Lord and Haebara) x 2 (test equating methods: IRT true score equating and observed score equating)] were examined. Multidimensionality was constructed as assuming the two test forms measuring Ɵ1 and Ɵ2 abilities. While the simulation condition which has correlation between abilities 0.90 represents weak multidimensional case, the correlation between abilities 0.00 represents the severe multidimensional case. Tests were equated under common-item non-equivalent groups design. Equating results were evaluated by using bias, standard deviation and RMSE evaluation criteria. The results showed that, generally under all conditions equating results provided from concurrent calibration more biased and had higher RMSE values than equating results provided by separate calibration. Based on standard deviation criteria, when the degree of multidimensionality was low, equating results which got from concurrent calibration and separate calibration with Stocking-Lord or Haebara scale transformation methods showed similar performance but when the degree of multidimensionality was severe equating results which had lowest random error were provided by concurrent calibration.

Solar Flare Forecasting from Magnetic Feature Properties Generated by the Solar Monitor Active Region Tracker

We study the predictive capabilities of magnetic-feature properties (MF) generated by the Solar Monitor Active Region Tracker (SMART: Higgins et al. in Adv. Space Res. 47, 2105, 2011) for solar-flare forecasting from two datasets: the full dataset of SMART detections from 1996 to 2010 which has been previously studied by Ahmed et al. (Solar Phys. 283, 157, 2013) and a subset of that dataset that only includes detections that are NOAA active regions (ARs). The main contributions of this work are: we use marginal relevance as a filter feature selection method to identify the most useful SMART MF properties for separating flaring from non-flaring detections and logistic regression to derive classification rules to predict future observations. For comparison, we employ a Random Forest, Support Vector Machine, and a set of Deep Neural Network models, as well as lasso for feature selection. Using the linear model with three features we obtain significantly better results (True Skill Score: TSS = 0.84) than those reported by Ahmed et al. (Solar Phys. 283, 157, 2013) for the full dataset of SMART detections. The same model produced competitive results (TSS = 0.67) for the dataset of SMART detections that are NOAA ARs, which can be compared to a broader section of flare-forecasting literature. We show that more complex models are not required for this data.

TensorFlow-Based Automatic Personality Recognition Used in Asynchronous Video Interviews

With the development of artificial intelligence (AI), the automatic analysis of video interviews to recognize individual personality traits has become an active area of research and has applications in personality computing, human-computer interaction, and psychological assessment. Advances in computer vision and pattern recognition based on deep learning (DL) techniques have led to the establishment of convolutional neural network models that can successfully recognize human nonverbal cues and attribute their personality traits with the use of a camera. In this paper, an end-to-end AI interviewing system was developed using asynchronous video interview (AVI) processing and a TensorFlow AI engine to perform automatic personality recognition (APR) based on the features extracted from the AVIs and the true personality scores from the facial expressions and self-reported questionnaires of 120 real job applicants. The experimental results show that our AI-based interview agent can successfully recognize the “big five” traits of an interviewee at an accuracy between 90.9% and 97.4%. Our experiment also indicates that although the machine learning was conducted without large-scale data, the semisupervised DL approach performed surprisingly well with regard to APR despite the lack of labor-intensive manual annotation and labeling. The AI-based interview agent can supplement or replace existing self-reported personality assessment methods that job applicants may distort to achieve socially desirable effects.

Clutter negotiating ability in an ensemble of forest interior bats is driven by body mass.

Differences in wing morphology are predicted to reflect differences in bat foraging strategies. Experimental tests of this prediction typically assess the relationship between wing morphology and a measures of flight performance on an obstacle course. However, studies have lacked measures of obstacle avoidance ability true scores, which may confound interpretation of ability across the range of presented tasks. Here, we used Rasch analysis of performance in a collision-avoidance experiment to estimate the ability of bat species to fly through vegetative clutter. We refer to this latent trait as 'clutter negotiating ability' and determined the relationships between clutter negotiating ability and wing morphology in 15 forest insectivorous bat species that forage in the densely cluttered rainforests of Malaysia. The clutter negotiating ability scores were quantified based on individual responses of each species to 11 different obstacle arrangements (four banks of vertical strings 10-60 cm apart). The tasks employed for the collision-avoidance experiment were reliable and valid, although Rasch analysis suggested that the experiment was too easy to discriminate completely among the 15 species. We found significant negative correlations between clutter negotiating ability and body mass, wingspan, wing loading and wing area but a positive significant correlation with wingtip area ratio. However, in stepwise multiple regression analyses, only body mass and wing loading were significant predictors of clutter negotiating ability. Species fell into clusters of different clutter negotiating ability, suggesting a potential mechanism for resource partitioning within the forest interior insectivorous ensemble.