Free-response Format Research Articles

Are multiple-choice tests unfair to girls?

Gender differences in English proficiency as a function of testing method were examined by comparing the results of a 10% representative sample ( n varied between 3,258 and 4,131) of boys and girls in the Swedish upper-secondary school on a subtest of the national test in English for the years 1986–1993. The tasks were similar, but ca half of them were of the multiple-choice and half of the free-response format. A hypothesis that boys would be more successful than girls on the multiple-choice tasks was confirmed. Each year the boys performed better than the girls on the multiple-choice part and the girls better on the free-response part.

Cognitive Models Underlying Students' Formulation of Simple Linear Equations

Data are presented to show that errors in formulating algebraic equations are not primarily due to syntactic translation, as has been assumed in the literature. Furthermore, it is shown that the reversal error is common even when none of the previously published causes of the error is applicable. A new explanation is required and is proposed in this paper. An examination of students' errors leads us to suggest that students generally construct from the natural language statement a cognitive model of compared unequal quantities. They formulate equations by trying to represent the model directly or by drawing information from it. This hypothesis is supported by research on the comprehension of relationships by linguists, pyscholinguists and psychologists. Data were collected from 281 students in grade 9 in free response format and from 1048 students in grades 8, 9, and 10 who completed a multiple-choice item.

Assessment of the Full Content of Physique Stereotypes with a Free-Response Format

A major limitation of physique stereotyping research is that American subjects have used a restricted set of traits provided by the experimenter to evaluate endomorphs, mesomorphs, and ectomorphs. A free-response procedure was used in Study 1 to identify the full domain of traits associated with each physique. Although many of the traits identified were similar to those used in previous research, a number of new trait dimensions were uncovered. In Study 2, the large number of traits generated with the free-response format was reduced by combining synonyms. This new set of traits was placed in semantic differential format and given to a new sample of subjects to evaluate. The results paralleled those of the first study with the exception that ectomorphs were seen more favorably. This discrepancy, as well as the fact that raters listed trait antonyms within each of the three major physiques in Study 1, suggests the possibility of unique subtypes within the global stereotypes. Examples of these subtypes are proposed.

Social Influence Strategies Among Japanese and American College Women

Social influence strategies of 40 Japanese and 41 American college women were compared. With the use of a free-response format, respondents were asked to describe how they get their way with their mother, father, male teacher/boss, female teacher/boss, male friends, and female friends. Contrary to expectations, content analysis indicated that Japanese women reported using strong and neutral strategies more frequently and weak strategies less frequently than American women. American women used manipulation (especially sexual manipulation) more frequently and reasoning less frequently than Japanese women. Analyses by target of influence indicated that these differences were not found when the target was a female friend but were demonstrated across most of the other targets.

Causes of Drug Use

The perceived causes of drug use and their interrelationship were examined by network analysis. Subjects were asked in a free-response format to nominate causes of drug use. The 10 most frequently cited causes were used as labels on a matrix, and subjects were asked to determine whether there was a cause-and-effect link between each pair of nominated causes. Individual matrices were combined, and the composite matrix was subjected to network analysis. The results indicate that subjects view personal problems (stress and depression) and peer-related societal acceptance as important causal factors in drug use. These findings are discussed with respect to current approaches to the drug problem.

A COMPARISON OF OPEN-ENDED AND MULTIPLE-CHOICE QUESTION FORMATS FOR THE QUANTITATIVE SECTION OF THE GRADUATE RECORD EXAMINATIONS GENERAL TEST

Free-response counterparts to a set of items from the quantitative section of the Graduate Record Examinations General Test were developed. Examinees responded to these items by gridding numerical answers on a machine-readable answer sheet in essentially the same way social security numbers are now gridded. In addition, the answer grid included special symbols for a negative sign (-), a decimal point (.), a division sign (/), and a variable (k). The test section with the special answer sheet was administered at the end of a regular GRE administration. Test forms were spiraled so that random groups received either the grid-in questions or the same questions in a multiple-choice format. Because the amount of extra time required to grid in responses was unknown, both a long and a short form were developed, with students randomly receiving one or the other. Sample size for these comparisons was 12,465. In a separate data collection effort, 364 paid volunteers who had recently taken the General Test used a computer keyboard to enter answers to the same set of questions. Correct and incorrect answers chosen in the multiple-choice format were compared with the answers generated in the free-response format. Three-parameter item characteristic curves were compared, as were correlational patterns with other test scores and undergraduate grade point average. Format by gender and format by ethnicity (Asian American, Black, Hispanic, and White) interactions were explored with analyses of covariance. Despite the format differences noted for individual items, total scores for the multiple-choice and free-response tests demonstrated remarkably similar correlational patterns, and there were no significant interactions of test format with either gender or ethnicity.

Patient and staff perceptions of caring: review and replication

The present study identified patient (n = 86) and nursing staff (n = 73) perceptions of most and least important caring behaviours. Using a Swedish version of the CARE-Q instrument or a free rating scale, patients ranked items concerned with giving honest and clear information and competent clinical expertise as most important. The nursing staff ranked expressive/affective behaviours as most important. There were significant differences between the two groups in the ranking of 14-30 out of 50 specific behaviours and in 3-5 out of 6 subscales. Patient and staff ratings did not differ appreciably between the methods used, with the exception that staff gave much higher ratings to most items in the free response format. Patients were more discriminating in the importance they assigned to the various items.

The Relationship Among Conductors' Rankings of Three Unfamiliar Wind Band Scores

The twofold purpose of this study was to examine college and university wind band conductors' agreement regarding the relative musical quality of three unfamiliar wind band scores and to collect and synthesize conductor's reported criteria for their rankings. Thirty-three conductors responded to a request to examine, compare, and rank the scores on the basis of relative musical quality, and then list their criteria for their decisions in a free-response format. There was no statistically significant agreement among the conductors' expressed judgments of the relative musical quality of the three unfamiliar scores used in this study. There seemed to be agreement, however, regarding the criteria used as bases for their judgments. Results suggest that individual perception and differing emphases among the criteria may have influenced the conductors' rankings of the three scores.

Intergroup violence and intergroup attributions

Pettigrew's (1979) prediction that relative to in-group behaviour, negative out-group behaviour would be attributed to internal characteristics, was tested in the context of Northern Ireland's continued conflict. Catholic and Protestant respondents were presented with newsreel footage depicting scenes of in- and out-group violence. One showed a Protestant attack on mourners at a Catholic funeral. The other showed a Catholic attack on a car containing two plain clothes soldiers at a Catholic funeral. Using a free response format, subjects' explanations were classified into internal and external attributions. The results showed strong support for Pettigrew's hypothesis. The implications of these findings with regard to the maintenance of intergroup conflict are discussed.

“Paying Your Dues” in the Hospitality Industry: How Functional is the Belief?

One of the central beliefs about career success within the hospitality industry is that new employees must “pay their dues.” In Study 1, free-response format interviews with 40 hospitality program graduates generated functional and dysfunctional connotations of this belief statement for both the individual and the organization. In Study 2, 35 graduating seniors were asked to describe the extent to which the “pay your dues” message was present during job interviews. Future research and implications for surfacing and rethinking this belief are discussed.

Language use in intergroup contexts: the linguistic intergroup bias.

Three experiments examine how the type of language used to describe in-group and out-group behaviors contributes to the transmission and persistence of social stereotypes. Two experiments tested the hypothesis that people encode and communicate desirable in-group and undesirable out-group behaviors more abstractly than undesirable in-group and desirable out-group behaviors. Experiment 1 provided strong support for this hypothesis using a fixed-response scale format controlling for the level of abstractness developed from Semin and Fiedler's (1988a) linguistic category model. Experiment 2 yielded the same results with a free-response format. Experiment 3 demonstrated the important role that abstract versus concrete communication plays in the perpetuation of stereotypes. The implications of these findings and the use of the linguistic category model are discussed for the examination of the self-perpetuating cycle of stereotypes in communication processes.

MULTIPLE‐CHOICE VERSUS FREE‐RESPONSE: A SIMULATION STUDY

Responses to a 40‐item test were simulated for 150 examinees under free‐response and multiple‐choice formats. The simulation was replicated three times for each of 30 variations reflecting format and the extent to which examinees were (a) misinformed, (b) successful in guessing free‐response answers, and (c) able to recognize with assurance correct multiple‐choice options that they could not produce under free‐response testing. Internal consistency reliability (KR20) estimates were consistently higher for the free‐response score sets, even when the free‐response item difficulty indices were augmented to yield mean scores comparable to those from multiple‐choice testing. In addition, all test score sets were correlated with four randomly generated sets of unit‐normal measures, whose intercorrelations ranged from moderate to strong. These measures served as criteria because one of them had been used as the basic ability measure in the simulation of the test score sets. Again, the free‐response score sets yielded superior results even when tests of equal difficulty were compared. The guessing and recognition factors had little or no effect on reliability estimates or correlations with the criteria. The extent of misinformation affected only multiple‐choice score KR20′s (more misinformation—higher KR20′s). Although free‐response tests were found to be generally superior, the extent of their advantage over multiple‐choice was judged sufficiently small that other considerations might justifiably dictate format choice.

Factors related to grade retention in children with short stature.

A recent study by Holmes et al. (1982) revealed a high degree (23%) of grade retention in a group of children with significant short stature secondary to growth hormone deficiency (GHD), constitutional delay (CD); or Turner's Syndrome (TS). Parents indicated in a free response format that 'immaturity' and 'small size' were the reasons for retention. The present follow-up study obtained academic achievement data on 47 of the 56 short children from the Holmes et al. (1982) study to assess what role academic factors, not spontaneously mentioned by parents, may have had in the retention decision. Results indicated that despite repeating a year in the primary grades, the group of retained children were still functioning 6 months below grade expectation, compared to grade appropriate achievement in the never-retained group. Parents and teachers were both accurate in their perceptions of children's academic achievement. Behavioural ratings by teachers indicated more adjustment difficulties for retained children, while parent ratings of behaviour showed a similar high level of problems for short children regardless of retention status. Although both groups of children possessed average intellectual abilities, the retained children obtained lower IQ scores than the never retained children. In summary, a majority of short children were achieving appropriate grade levels, but a substantial minority were experiencing under-achievement, behaviour problems, and grade retention, despite average intelligence.

The cognitive content of anxiety: Naturalistic evidence for the predominance of threat-related thoughts

Beck's (1976) hypothesis that the experience of anxiety is preceded by thoughts with a content of personal threat or danger was evaluated in a study that controlled for effects due to experimenter expectancy. Seventy introductory psychology students were exposed to one of three types of expectancy “training”: One group was trained to expect threat-related thoughts prior to experiencing anxiety, another group was trained to expect loss-related thoughts, and the final group received a neutral training. Subjects were then asked to record their thoughts preceding anxiety experiences in a free-response journal format. Each response was categorized as either a thought of threat, thought of loss, or other thought, and the number of thoughts falling into each category was the dependent variable. Threat was found to be the predominant cognitive content across groups (p < .001), while training had a small but significant (p < .05)effect on subjects' responses. These results are seen as offering qualified support to Beck's proposed threat-anxiety connection.

Some comparisons of intelligibility of synthetic and natural speech at different speech‐to‐noise ratios

Despite the extensive literature on the intelligibility of natural speech in noise, relatively little is currently known about the deleterious effects of noise on the perception (i.e., recognition, identification, and understanding) of synthetic speech. In this paper, we wish to report the results of a study that examined the effects of noise on the intelligibility of synthetically produced monosyllabic words using the traditional Modified Rhyme Test to measure intelligibility. Over a range of S/N ratios, we found that synthetic speech suffers a greater, decrement in performance than naturally produced speech. Moreover, the decrement in performance was even greater when the MRT response format was changed from a closed six‐alternative forced choice procedure to an open free response format. The increase in uncertainty affected recognition of the synthetic items more than the natural ones. These findings have implications for the use of voice response systems that employ synthetic speech in noisy listening environments where intelligibility of the linguistic message may be at a premium. [Supported by a research grant from NINCDS to Indiana University.]

A Free Response Test of Interpersonal Effectiveness

Development of the Interpersonal Problem Solving Assessment Technique (IPSAT), College form, is described. Guided by Rotter's Social Learning Theory, problem-solving, and assertiveness research, a semi-structured free response format was designed to assess components of interpersonal effectiveness, The instrument yields patterns of self-reported behaviors in six classes of problematic social situations. A detailed manual enabled reliable scoring of the following response categories: Effectiveness, avoidance, appropriateness, dependency and solution productivity. Scores were not materially affected by sex, verbal ability, or social desirability response sets. Correlations with the College Self-Expression Scale, the Edwards Personal Preference Schedule and the Lanyon Psychological Screening Inventory provided initial evidence of validity. Comparison of mean IPSAT scores of 23 psychotherapy clients with those of 78 normative subjects showed that clients report less interpersonal effectiveness and more avoidance than controls. Implications for utility of the IPSAT are discussed.

MEASURING PROBLEM SOLVING ABILITY IN MATHEMATICS WITH MULTIPLE‐CHOICE ITEMS: THE EFFECT OF ITEM FORMAT ON SELECTED ITEM AND TEST CHARACTERISTICS

The development of problem solving skills is a major goal of most, if not all, mathematics curricula (Taylor, 1977; Robinson, 1977). Given this goal, there is the obvious need to assess the development of these skills. If a paper and pencil test is used for assessment, the best test is probably one that uses a free response format because relatively specific diagnostic information can be obtained. For example, the scorer may be able to determine which of the following three errors caused a student to answer a problem incorrectly: (a) a computational error; (b) an error in selection of the mathematical process; (c) a reading comprehension error. However, if a paper and pencil test uses a multiple-choice format, then there is usually a minimum amount of diagnostic information available. The most commonly used type of multiple-choice item is illustrated below:

A comparison of multiple-choice tests and free-response tests in examinations of clinical competence.

This paper reports a study which compared the performance of different groups of students and doctors on identical and equivalent tests set in an objective-type format and in a free-response format. The tests were designed to ensure that the content was relevant to clinical practice at the hospital intern level. In all test situations candidates' scores were significantly higher in the objective tests than in the free-response tests. This difference was greater for the more junior and less competent students than for the more competent doctors. The cueing effect of the options was thought to be the main factor responsible for the difference in performance. The results of a questionnaire survey demonstrated that students were aware of the deficiencies in multiple-choice tests. A large majority of the students believed that the free-response tests gave a more accurate assessment of their clinical ability. It was found that in these tests, aimed at measuring aspects of clinical competence, multiple-choice questions appeared to overestimate the candidate's ability to an extent that made them less suitable than free-response questions for this purpose. It was also found that free-response tests, of the type used in this study, provide a suitable alternative to multiple-choice tests for use in the written section of clinical examinations. It was concluded that the written component of the final examination in the medical course should have a preponderance of free-response items over multiple-choice items.

Irish Students' Stereotypes About Some National and Subnational Groups Within Ireland and Great Britain

A sample of Irish undergraduates was asked to characterize the English, Irish, Welsh, Scots, Northern Irish Catholics, Northern Irish Protestants, Southern Irish Catholics, and Southern Irish Protestants using both a check list and a free response format. Analyses of the social and personal stereotypes indicated that the English were described in different and significantly less favorable terms than the Irish. The enmities and strife in Northern Ireland were reflected in the stereotypes about the subnational Irish groups. While Catholics and Protestants, on both sides of the political border, were seen to be different from each other, these differences were overshadowed by the distinctions between the Northern and Southern groups. Both of the Northern irish targets were characterized in much less favorable and much less differentiated terms than their Southern counterparts. The implications of these findings were discussed.

Computers in behavioral science: Computer scoring of sentence completion data

The development of techniques for computer-based personality assessment from sentence completions is outlined. The One-Word Sentence Completion (OWSC) instrument was designed to elicit data suitable for machine processing, while retaining most of the advantages of a free-response format. Two operative scoring systems are described. The first employs a “dictionary” of 4366 weighted response words to yield 25 scores from a 90-item OWSC form. The second system utilized a complex word-root data reduction procedure and a bank of 892 generic roots to produce scores for 40 variables. Initial reliability data and normative sex differences are reported, and future development of the technique is discussed.