Scoring System For Use Research Articles

Accuracy in the scoring of writing: Studies of reliability and validity using a New Zealand writing assessment system

Accuracy in the scoring of writing is critical if standardized tasks are to be used in a national assessment scheme. Three approaches to establishing accuracy (i.e., consensus, consistency, and measurement) exist and commonly large-scale assessment programs of primary school writing demonstrate adjacent agreement consensus rates of between 80% and 100%, and consistency and measurement coefficients ranging between .70 and .80, .60 and .80, respectively. A New Zealand educational assessment project has developed a set of writing assessment rubrics that contain curriculum based rating scales for six purposes of writing each with its own bank of writing prompts for use by classroom teachers. Standardization of the rating scales and prompts was conducted with representative samples of students, aged 10–13. This article describes two studies that established the validity of the scoring system for use in New Zealand classrooms. Adjacent agreement consensus fell between 70% and 90%, while consistency and measurement correlations fell in the range .70–.80 in both studies. This consistency with international standards was sufficiently robust to provide confidence in the underlying norms provided by the asTTle assessment tool. Relatively low levels of training were required by teachers to reach this degree of accuracy. The accuracy of scoring gives government and teachers confidence in the validity of the project's rating scales and suggests that classroom teachers will be able to generate accurate scores upon which instructional decisions can be based.

Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project.

The SCORE project was initiated to develop a risk scoring system for use in the clinical management of cardiovascular risk in European clinical practice. The project assembled a pool of datasets from 12 European cohort studies, mainly carried out in general population settings. There were 20,5178 persons (88,080 women and 11,7098 men) representing 2.7 million person years of follow-up. There were 7934 cardiovascular deaths, of which 5652 were deaths from coronary heart disease. Ten-year risk of fatal cardiovascular disease was calculated using a Weibull model in which age was used as a measure of exposure time to risk rather than as a risk factor. Separate estimation equations were calculated for coronary heart disease and for non-coronary cardiovascular disease. These were calculated for high-risk and low-risk regions of Europe. Two parallel estimation models were developed, one based on total cholesterol and the other on total cholesterol/HDL cholesterol ratio. The risk estimations are displayed graphically in simple risk charts. Predictive value of the risk charts was examined by applying them to persons aged 45-64; areas under ROC curves ranged from 0.71 to 0.84. The SCORE risk estimation system offers direct estimation of total fatal cardiovascular risk in a format suited to the constraints of clinical practice.

The development of a scoring system for the Gerontological Apperception Test

To explore the viability of a revised and more differentiated scoring system for use with the Gerontological Apperception Test (GAT; Wolk & Wolk, 1971), 102 older adults (mean age = 68.3 years) were administered the GAT. Scoring criteria were developed to reflect a variety of constructs speaking to the interpersonal, health-related, and intrapsychic dimensions of the experience of later life. For each of these 20 GAT variables, indices of interscorer agreement exceeded 80%. Bivariate correlations suggested that 12 of 20 GAT dimension scores were related to chronological age as well as to scores derived from measures of crystallized and fluid ability. This also was true for 2 of 5 GAT factor scores. Not only do these data suggest that newly developed GAT variables can be reliably scored, but they also indicate that these more carefully differentiated variables relate to measures of intellectual functioning in theoretically meaningful ways. These findings therefore speak to the utility of the GAT using more carefully defined scoring criteria.

The Reliability and Repeatability of a Lameness Scoring System for Use as an Indicator of Welfare in Dairy Cattle

In 16 dairy herds, a lameness scoring system which was reasonably quick and easy to use was evaluated with regard to reliability and repeatability. The ultimate aim was to include it into an on-farm welfare assessment protocol. Locomotion scores in individual cows (n = 206) were significantly correlated with the heel and sole disorder indices obtained at routine claw trimmings (e.g. index for hind limb lesions: r s = 0.40, p = 0.0001; Spearman Rank Correlation Coefficients). The inclusion of lesions caused by digital dermatitis increased correlation coefficients when hind limbs and all four feet were considered (r s = 0.47/0.50). The inter-observer repeatability of locomotion scores in individual cows was high, and the variation between observers was low when calculated as a percentage of animals within each scoring class. Thus the locomotion scoring system proved to be a useful tool for assessing lameness, and as such a valuable part of on-farm welfare assessment in dairy cattle. It is recommended that scoring be performed at least twice during the housing period.

Chest radiograph scoring system for use in preterm infants

Papers dealing with the chest radiograph in pre-term infants are sufficiently rare in the British radiological literature to merit enhanced attention. Yuksel et al (1991) are to be congratulated in helping to refine those features which have a bearing on the outcome of chronic lung disease in these infants. Although not germane to the theme of the paper, I was struck by the fact that two patients had lobectomies because of expanding air cysts.

Chest radiograph scoring system for use in preterm infants

We thank I. Hyde for this interest in our paper (Yuksel et al. 1991). Two of our patients had undergone lobectomy for gross cystic degeneration following localized severe pulmonary interstitial emphysema. Both infants were ventilator-dependent at the time of lobectomy with deteriorating blood gases. As a consequence of this deterioration in respiratory status it was decided that they should undergo lobectomy following which their respiratory status improved and it was possible to extubate the infants.

Evaluation of a QRS scoring system for estimating myocardial infarct size: VIII. Specificity in a control group with left ventricular hypertrophy and proposal of a new scoring system for use with this confounding factor

Electrocardiographic differentiation between left ventricular hypertrophy (LVH) and myocardial infarction (MI) is often difficult because both diagnoses are based primarily on QRS changes on the electrocardiogram (ECG). The specific goal of this study was the development of ECG criteria that could be used with the complete Selvester QRS scoring system for MI size in patients with LVH. A study population of 127 patients had significant aortic valve disease verified by cardiac catheterization. Inclusion in the study required no significant coronary artery disease, no focal contraction abnormality on the left ventriculogram, and no documented MI. Quantitative criteria for LVH developed by Bonner (IBM) and also those developed by the Cornell group were used to determine the ECG evidence for LVH in each patient. One or both sets of criteria were met in 110 (87%) of the 127 patients. This group was compared to a previously evaluated control population of 500 normal subjects. The complete 54-criteria, 32-point QRS MI size scoring system was applied to the 12-lead ECG of both groups. The score was 98% specific in the normal controls and 73% specific in the LVH group using a score of > 3 points as diagnostic for MI. Of the 54 individual QRS criteria, 16 failed to achieve 95% specificity in the LVH population: 13 were for anterior (and apical), 2 for inferior, and 1 for posterior locations. Of these 16, minor modifications to 11 were sufficient to achieve the 95% specificity standard. The resultant 49-criteria, 31-point QRS score for estimating MI size in patients with LVH was 95% specific using a level of > 3 points as diagnostic for MI. The sensitivity of this score and its ability to predict MI size in the presence of LVH must now be tested in prospective studies of patients with documented LVH, with and without MI.

Chest radiograph scoring system for use in pre-term infants

A scoring system to characterize the pre-term infant's chest radiograph appearance has been evaluated. Chest radiographs taken at 1 month of age in 20 infants (median gestational age 27 weeks) were scored according to abnormalities of lung volume, presence of opacification, interstitial changes and cystic elements (maximum score 17). The median score was 4 (range 1-14); the highest scores were for the only infants with cystic elements and/or interstitial changes on their chest radiographs. The scoring system documented accurately the severity of neonatal respiratory distress; the highest scores occurred in those infants requiring the most prolonged respiratory support, i.e. ventilated beyond 1 week of age (p less than 0.01) and oxygen-dependent beyond 1 month (p less than 0.01). It was also a useful predictor of chronic respiratory problems, with an 88% sensitivity in the detection of infants with the most severe lung function abnormalities at 6 months of age.

A computerized scoring system for use with content analysis scales

1. 1. This is a report of the development of a computerized scoring system for use with content analysis scales of speech. The goal has been to surpass the content analysis of single, isolated words in natural language and to focus the analysis on the quality and quantity of meaning carried in clauses or sentences and, eventually, in a broader contextual framework. 2. 2. Using the Gottschalk-Gleser content analysis Hostility scales, the first step taken was the use of an efficient parser that contained a sufficiently powerful grammar to characterize the subset of English that is often encountered in psychiatric interviews. Wood's Augmented Transition Network Parser was modified to run on a PDP-10 computer, and its grammar was enlarged to cover certain linguistic constructions that frequently occur in spoken discourse. In addition, a small dictionary of several hundred entries was created, permitting the maintenance of this dictionary in core. 3. 3. Since the Gottschalk-Gleser content analysis method derives a score on the basis of the action verb in a clause in conjunction with the noun-phrases that function as actors and recipients of this action, a technique was developed for assigning “meaning” to each of these constituents. Accordingly, verbs were assigned semantic features called verb-types based on the thematic categories and weights of the Gottschalk-Gleser Hostility Outward scale. 4. 4. The infinitive form of the verb in each spoken clause was retrieved from the parse and looked up in the semantic-features dictionary. The parser performed the required morphological analysis and stripped the inflectional ending of the verb, yielding its infinitive or root form. 5. 5. The computer next assigned semantic features to the noun-phrases that function as actors and recipients of the action verbs. In the simplest instances, the noun-phrase received the semantic features of its head noun. In more complex instances, for example in prepositional-phrases, noun-phrases were located as semantic markers for the initial noun-phrase. 6. 6. The scoring algorithm for arriving at the score for each clause was obtained after examining the set of semantic features assigned to the above-mentioned noun-phrases and head verb. An information structure (scoretree) was used for advice on how to interpret the combinations of noun-phrases and head verbs, and it was indexed by the verb-type. At present, each scoretree has only two levels; the first level is made up of tests for the features of the subject of the clause in question and the second level is composed of tests for the object. Which scoretree to access is determined by locating the verb-type of the head verb in the clause; the first-level node is obtained by overlapping the features of the subject noun-phrase with the features of successive first-level nodes until some partial match is achieved. The second-level node is located from the first-level node overlapping the features of the object noun-phrase with successive nodes until another partial match is achieved. Associated with the second-level node is the score that the total clause is to receive. 7. 7. In testing this automated method on 100 sentences taken at random from the manual of instructions for using the Gottschalk-Gleser content analysis scales, 60% of them were correctly parsed and scored. As another preliminary test of the information retrieved by this computerized method, six 5-min speech samples were scored by human content analysis technicians, and their scores were compared to the computerized scoring system at its present level of development. A Spearman rank difference correlation of 0.80 was obtained between the ranked scores using the two methods. 8. 8. The shortcomings and further steps to take in the development of this content analysis method are outlined and discussed.