Interscorer Reliabilities Research Articles

Obstructive sleep apnea predicts 10-year cardiovascular disease-related mortality in the Sleep Heart Health Study: a machine learning approach.

Obstructive sleep apnea (OSA) is considered to be an important risk factor for the development of cardiovascular disease (CVD). This study aimed to develop and evaluate a machine learning approach with a set of features for assessing the 10-year CVD mortality risk of the OSA population. This study included 2,464 patients with OSA who met study inclusion criteria and were selected from the Sleep Heart Health Study. We evaluated the importance of potential features by mutual information. The top 9 features were selected to develop a random forest model. We evaluated the model performance on a test set (n = 493) using the area under the receiver operating curve with 95% confidence interval and confusion matrix. A random forest model awarded the highest area under the receiver operating curve of 0.84 (95% confidence interval: 0.78-0.89). The specificity and sensitivity were 73.94% and 81.82%, respectively. Sixty-three years old was a threshold for increased risk of 10-year CVD mortality. Persons with severe OSA had higher risk than those with mild OSA. This study demonstrated that a random forest model can provide a quick assessment of the risk of 10-year CVD mortality. Our model may be more informative for patients with OSA in determining their future CVD mortality risk. Li A, Roveda JM, Powers LS, Quan SF. Obstructive sleep apnea predicts 10-year cardiovascular disease-related mortality in the Sleep Heart Health Study: a machine learning approach. J Clin Sleep Med. 2022;18(2):497-504.

A comparison of automated and manual sleep staging and respiratory event recognition in a portable sleep diagnostic device with in-lab sleep study.

The objectives were to develop and validate an algorithm for editing WatchPAT scoring and assess the accuracy in an unselected clinical population as well as age and sex substrata. Two hundred sixty-two participants were enrolled to undergo WatchPAT simultaneously with in-lab polysomnography (PSG) recordings for developing (n = 30), optimizing (n = 62), and validating (n = 170) an algorithm to review and edit respiratory events and sleep architecture of WatchPAT recordings, which was based on visual inspection of WatchPAT signals. Apnea-hypopnea index (AHI) and sleep indices were compared with PSG-derived and automated WatchPAT indices. Although estimation of total sleep time (TST) was comparable between automated and manual algorithm, estimation of rapid eye movement (REM) sleep time was markedly improved with manual editing from 0.48, 23.0 min (-43.9 to 89.8) to 0.64, 18.3 min (-32.6 to 69.1) (correlation with PSG, mean difference [reference range] from PSG, respectively). Manual scoring also improved correlation and agreement with PSG AHI from 0.65, 2.5 events/h (-24.0 to 28.9) to 0.81, -4.5 events/h (-22.5 to 13.6) as well as concordance for categorical agreement of sleep-disordered breathing severity and concordance for detecting severe REM-related sleep-disordered breathing. Interscorer reliabilities were excellent for TST and AHI, while good for REM sleep time. The automated algorithm performed better in younger than in older patients, while performed similarly between men and women with respect to concordance statistics. The manual algorithm markedly improved concordances more in older patients and women than in their counterparts. Our manual editing algorithm improves correlation and agreement with PSG-derived sleep and breathing indices. Sex and age influence the accuracy of automated analysis and the performance of manual editing on AHI concordance.

Psychometric Properties of the Bender Gestalt Test Using Lacks' Version of the Hutt-Briskin Scoring System

Criterion-referenced (Livingston r) and norm-referenced (Gilmer-Feldt r and Coefficient Alpha) techniques were used to calculate the internal consistency reliability of the Bender-Gestalt Test (BGT) Total Score using the 12-item Lacks system of scoring. Livingston's r was found to be .825 for the Lacks BGT cutoff score of 5. The Gilmer-Feldt and alpha coefficients for the Lacks Total Score was found to be .644 and .626, respectively. An item analysis showed that most of the BGT items (9 out of 12) were within established criteria for item difficulty, however, 7 items were found to be poor discriminators. The interscorer reliabilities based on three scorers, two scorers, and a single scorer was found to be .895, .852, and .740, respectively. Due to the low reliabilities and several inherent flaws that were identified with the Lacks scoring system, the authors recommend that users of the BGT consider alternative objective scoring systems.

Interscorer reliability of the MMPI-2: should TRIN and VRIN be computer scored?

Some clinicians and researchers hand score their MMPI-2 protocols. Graham (1990) noted that TRIN and VRIN involve complex scoring that is best done by a computer. We examined the interscorer reliabilities of TRIN and VRIN and compared these results to the reliabilities of the other validity and clinical scales. Despite the high reliabilities (.97 to 1.00), clerical errors were common (i.e., 7% of the scales were scored inconsistently). Graham's caution about the inherent difficulty of scoring VRIN, but not TRIN, was supported. There were three times as many scoring disagreements for VRIN than any other scale (i.e., 27% of the protocols were in disagreement).

Interscorer Reliabilities for Memory and Localization Scores of the Tactual Performance Test

The interscorer reliabilities were computed for the Memory and Localization scores from the Tactual Performance Test for a sample (N = 309) consisting of normal volunteers (n = 27), normal patients (n = 78), patients with diffuse brain-damage (n = 69); patients with Korsakoff's syndrome (n = 15), and undiagnosed patients (n = 120). The interscorer reliabilities were .9810 for Memory and .9773 for Localization. Estimates of the three individual scorer reliabilities were higher than .89.

Narrative and visual spatial recall: Assessment incorporating learning and delayed retention

Abstract We determined whether memory for a complex figure improves (learning occurs) after re-presentation of the stimulus despite a delay, as it does for verbal narrative information in the Babcock-Levy recall procedure. In addition, we compared interdigitated administrations of the verbal and nonverbal tests, which reduce delays to 5 minutes, to separate administrations of each test with delays of 20 minutes. Scoring manuals were written, and interscorer reliabilities in the .90's were obtained. Results indicated that visual-spatial recall improves, that the two delays generated equivalent results and that these measures of visual-spatial and verbal memory are not correlated. The interdigitated sequence may be an efficient battery for measuring deliberate recall and learning of verbal and nonverbal complex information.

An examination of three tests of visual-motor integration.

This study examines three tests of visual-motor integration which might be used as predictors of learning disabilities in a kindergarten population: the Bender Gestalt Test, the Beery Developmental Test of Visual Motor Integration, and the Geometric Design subtest of the WPPSI. Issues discussed include the relative interscorer reliabilities of the three tests, correlations among scores of the three tests, correlations between visual-motor integration scores and IQ, and sex differences in test performance. Suggestions are provided regarding the selection of a test of visual-motor integration for the kindergarten population.

THE RELIABILITY OF AUDITORY BRAINSTEM EVOKED RESPONSES IN HUMANS

Recently researchers have reported a lack of prediction of newborn auditory brainstem evoked responses (ABER's) assessment to later hearing performance which may be traced, in part, to lack of reliability of the ABER in the newborn period. We recorded 2 ABER's to each of 3 stimuli in 4 groups of subjects (n=10 in each group). One group consisted of adults; the other 3 were newborns - 32,36, and 40 weeks conceptional age at the time of testing. Each record was scored independently and blindly by 2 individuals experienced in ABER recording. The reliability of the waveforms differed as a function of the wave (Wave V was most reliably scored, I the least), age of the subject (especially for the softest stimulus, the younger the subject the less reliable the waveforms), and the stimulus (the slow, loud stimulus elicited the most reliable waveforms; the soft stimulus, the least). Poor intra-and inter-scorer and test re-test reliabilities for waveforms elicited by the softest stimulus suggest difficulty in wave identification. For the loud stimuli, intra-and inter-scorer reliabilities were acceptably high suggesting that true variability in the subject's response accounted for the lack of reproducibility.

The Competency Screening Test

The Competency Screening Test was administered to a group of patient/defendants to study its interscorer reliability and its predictive validity of fitness to stand trial based on psychiatric opinion. The interscorer reliabilities were high. The test also significantly differentiated between patients categorized by the participating psychiatrist as fit and those categorized as not fit. The results compared favorably in terms of overall accuracy with those published earlier. An analysis of false positives and false negatives suggested some possible differences in the kinds of predictive errors inherent in the two sets of data. The desirability of the latter approach was pointed out. The data support the conclusion that this instrument may be useful as a screening tool for referral of patient/defendants by a nonpsychiatric member of the treatment team for competency evaluation by a psychiatric member of that team.

Guidelines for Use in Scoring Tat Stories for Time-Span

Application of Epley and Ricks' (1963) system to score the TAT for time perspective did not reproduce the high interscorer reliabilities reported by users of the system. A set of guidelines is presented which were instituted to decrease the subjective nature of the system. When guidelines and system were used together, correlations comparable to those previously reported were found.

Interscorer Reliability of the Hand Test with Normal Subjects

The Hand Test (Wagner, 1962) was administered to 48 subjects. Interscorer reliabilities were computed based on the agreement in scoring between a well trained graduate student and the author of the test. The reliabilities were quite high for a projective technique (median rho = .910).

Some Indian Data on Reliability of the Hindi Translation of the Uses Test

Interscorer, test-retest and split-half reliabilities of the Hindi translation of Dennis' Uses Test were estimated with 180 Indian subjects, varying in academic placement from nursery to college and in age from 5 to 25 yr. The three reliabilities ranged from .97 down to .85, .89 down to .24 and .82 down to .25 respectively. On the basis of these results, the Hindi translation seems useful for group comparisons of children, adolescents and adults in India, except scores on religious value which are of doubtful utility.

On the measurement of hostility, aggression anxiety, projection and dependency.

Summary A 114-item sentence completion test of hostility, aggression anxiety, projection of hostility, and dependency was scored with an explicit scoring standard, correlated with extrapunitive and intropunitive peer ratings of 101 subjects, and cross-validated against a second group of 101 subjects. The reliability of peer ratings ranged from +.85 to +.91. Interscorer reliabilities for the sentence completion scales ranged from +.90 to +.98, while odd-even reliability was +.78. Significant correlation ratios between peer ratings and the hostility and aggression anxiety scales of the test were obtained.

A normative study of an auditory projective technique.

Abstract The purpose of this study is to provide normative data for the sound effect sequences of the Braverman-Chevigny Auditory Projective Test. The test was administered to 149 college undergraduates in a group setting, who were asked to compose stories to the seven sound sequences. Stories were coded in terms of tone, outcome, number of characters, theme, and identification of “hero.” The stories were predominantly negative in tone, with outcomes generally neutral or unsuccessful. Interscorer reliabilities were all significant beyond the .001 level.

The Mayman form-level scoring method: scorer reliability and correlates of form level.

Abstract A method of scoring the form level of Rorschach responses developed by Mayman (1962) was employed in a study of 100 twins and 95 of their parents. A method of training is described that led to generally adequate inter-scorer reliabilities. The relationship of mean form level to age, sex, and intellectual abilities is discussed, as are results relevant to the “heritability” of mean form level. It is concluded that mean form level is of complex determination and should be interpreted cautiously in developmental studies. It is argued that Mayman's highly differentiated approach to form level has unique value for qualitative analyses of Rorschach protocols.

The adjustment of drug addicts as measured by the sentence completion test.

Summary The Rotter Incomplete Sentences Blank was administered to fifty out-patients at a rehabilitation center for narcotic addicts. There were twenty male heroin users, twenty male pill users, and ten female heroin users. Interscorer, intrascorer, and split-half reliabilities were all high. It was found that a cutting score of adjustment of 135 correctly identified 80 per cent of the male heroin users, 90 per cent of the male pill users, and 100 per cent of the female heroin users.

Thematic analysis of dreams: inter-scorer reliabilities.

Psychoanalytically-based scales for the analysis of emotional and cognitive contents of dreams are described. Data for inter-scorer reliability and agreement are presented for each of three series of dreams. Prior to scoring each set, the raters underwent extensive training with the Dream Scales. The data indicate that inter-scorer reliability may decrease over time, but that with training, reliabilities and coefficients of agreement are sufficiently high to warrant further exploration of the utility of the scales.

VICARIOUS REINFORCEMENT AND IMITATIVE LEARNING.

VioLit summary: OBJECTIVE: The intent of this article by Bandura et al. was to explore the effects of vicarious reinforcement on conditioned emotional responses among nursery school children with a particular focus on modeling aggression. METHODOLOGY: The authors employed an experimental design by randomly assigning 40 boys and 40 girls from the Stanford University Nursery School into one of 2 experimental groups or one of 2 control groups. The mean age of the children was 51 months, with a range between 38 and 63 months. The children witnessed a 5 minute televised program which was either: 1) The Aggressive Model-Rewarded condition, in which a man's aggressive verbal and physical behavior in a play room were accompanied by a sense of victory and mastery over another man and his environment. 2) The Aggressive Model-Punished condition, in which a man's aggressive verbal and physical behavior in a play room (identical to the scenario in the previous model) were accompanied by severe punishment. 3) The Nonaggressive Model-Control group, in which the objects in the play environment were unchanged, but they were utilized by both characters in vigorous but nonaggressive play. 4) The Nonexposure Control group received no exposure to any of the models. The children were then tested for the incidence of postexposure imitative and nonimitative aggressive responses, while observed in an experimental play room. Several objects were available in the play room that could be used to reenact the aggressive behavior (baton, balls, Bobo dolls, hoola hoop, lasso, dart guns, cars, plastic farm animals) as well as several objects that tended to elicit predominantly nonaggressive responses (blackboard, doll house, cotton-stuffed dolls, building blocks). Each child spent 20 minutes in the play room, with only a noninteractive adult present (to ensure the child's comfort in a strange environment), and was rated by a hidden evaluator. Scores were obtained in predetermined categories of aggression which matched either those in the experimental aggressive models (kicking, lassoing or striking the Bobo doll with a ball or baton, shooting darts at cars or plastic farm animals, or repeating the verbal aggression which was displayed in the models) or those in predetermined categories of nonmatching aggression (punching or slapping the Bobo dolls, crashing the automobiles, acting out physical attacks toward the dolls or animals, etc.). A second rater witnessed 11 of the 80 children to provide an estimate of interrater reliability, and neither rater had knowledge of the treatment conditions to which the children had been exposed. The interscorer reliabilities were high, with product-moment coefficients in the .90s. After the rating each child was asked to discuss the behavior of the characters in the program and to identify the characters he or she preferred to emulate. FINDINGS/DISCUSSION: Many of the children, especially the girls, received low imitation scores and therefore nonparametric techniques were employed to evaluate the significance of the differences which were obtained. Results from the Kruskal-Wallis analysis of variance indicated that imitation was significantly influenced by response-consequences to the model (rewarded versus punished p AUTHORS' RECOMMENDATIONS: Additional research, argued the authors, should be implemented in which the difference between performance and learning can be better assessed, in order to identify the depth of impact from the vicarious transmission, rather than the just behavioral manifestation of the vicarious transmission. (CSPV Abstract - Copyright © 1992-2007 by the Center for the Study and Prevention of Violence, Institute of Behavioral Science, Regents of the University of Colorado) KW - California KW - Early Childhood KW - Child Aggression KW - Preschool Student KW - Reward KW - Punishment KW - Social Learning KW - Aggression Causes KW - Imitation Language: en