Group Oral Test Research Articles

Prompt effects on assessment of interactional competence in a group oral test

AbstractThis study serves as an attempt to validate an in‐house oral proficiency test (hereafter referred to as OPT) with a particular focus on prompt effects on assessment of interactional competence, an issue that has attracted burgeoning interest but not been adequately investigated yet. Prompts employed in the parallel versions of the group discussion task, which were supposed to be comparable, were found to differ in two dimensions, that is, topical domain and syntactic structure. The transcribed interactions from 136 sophomore test takers (34 groups) sitting in the first administration of the OPT were coded through a rigorous procedure and examined using conversation analysis. Five patterns of interaction were identified with distinct features based on two dimensions, mutuality and equality. Marked differences in distribution of patterns of interaction were detected in discourse elicited by different prompts, which should be mainly attributed to the syntactic structure of the prompts. The open‐question prompts were found more likely to elicit “conversation‐like” interaction with higher mutuality, characterized by the collaborative and unbalanced‐collaborative patterns, while the closed‐question prompts tended to elicit more “solo versus solo” interaction with lower mutuality, featured by the parallel and unbalanced‐parallel patterns. The influence of topic domain, however, was minimal. The results thereby constitute rebuttals that weaken meaningfulness and impartiality of the claim of interpretations about the construct within Bachman and Palmer's assessment use argument framework and suggest close attention from the test developers. The implications for prompt design, and development and validation of the group oral test are further provided.

Aligning the Language Criteria of a Group Oral Test to the CEFR: The Case of a Formal Meeting Assessment in an English for Occupational Purposes Classroom

The Malaysian Education Blueprint (MEB) 2015-2025 has set in motion efforts from all stages of education to align programs, courses, and syllabuses to the Common European Framework of Reference (CEFR) benchmark. This exercise has brought on major revamps in all aspects of English language education in the nation. This study will present such an undertaking in a public university in Malaysia and detail how the language criteria for an oral group test of an English for Occupational Purposes course have been aligned to the stipulated CEFR level. The actual assessment task involved groups of four or five students conducting a meeting of their established company. Data for the study came from an analysis of the audio recordings of nine group meetings, along with post-assessment interviews and focus group discussions involving three EOP instructors. Based on the data analysis, this study recommends a revised set of language criteria for the assessment. Furthermore, it demonstrates how an alignment of the scoring criteria with the descriptors of the targeted CEFR scale can be achieved through a systematic comparison of the language functions (LFs) produced in the meeting task to the targeted CEFR descriptor scales. The revised language component for the meeting assessment could help ease instructors’ assessment of students interactional skills and allow them to gauge better their students’ attainment of the skills required in a formal meeting context.

The use of interactive functions through an online group oral discussion test

This study attempted to explore the interactive functions of initiating, responding, developing, and collaborating that occurred during an online group oral discussion test. A total of nine university students were assigned to low, intermediate, and advanced level groups and participated in a 30-minute discussion without given topics. Transcripts of the discussion were used to code interactive functions, and individual interviews were conducted to take a closer look at the interactive functions that took place during the discussion. This study also analyzed how much each group differed in their use of interactive functions. Results were as follows: (i) the number of interactive functions used was not related to English proficiency level. Rather, particular interactive functions were frequently used by groups of each level, (ii) the factors that influenced the discussion seemed to be personalities and common interests in topics, and (iii) a majority of participants showed positive perspectives in terms of their view on taking an online group oral discussion test. Also reported in this article are participants’ individual perspectives on the use of online group oral discussion tests. Findings have implications for the development and implementation of appropriate online group oral discussion tests and discussions in the EFL environment.

Bias Sequences and Turn-Taking in the Validation of the Small Group Oral Examination

This research originated in an interest in the characteristics of communication in a group oral test. The researcher’s attention was directed towards its turn-taking mechanisms and how communication is shaped in groups of three. The investigation of the test discourse, with reference to insights and predictions from Conversation Analysis, yielded external validity evidence for the conversational nature of the interaction in this test. The findings support the claim that in Phase two of the group oral test the interaction is sufficiently conversational. True to the nature of conversation, each of the recorded exams contained bias sequences, in which interaction involves only two of the speakers from time to time. The findings also suggest a research-based rationale for the number of speakers desirable in a group oral test in that it may be a happy choice to examine students in threes, rather than twos or larger numbers.

Assessing ESL undergraduate performance in a group oral test: Rater orientations

Group oral testing has attracted increasing interests in performance-based assessment, mainly due to its ability to measure interactional competence. Increasing attention has consequently been paid to the performance of raters, whose orientations have an impact on the scores in view of their role as mediators between performances and scores. Studies have shown that despite training, raters’ personal constructs can lead to different assessments. This study examines rater orientations before and after viewing student performance in an oral test, to discover whether raters subscribe to the view that interaction is individual-focused and is mainly a representation of cognitive or within-language user construct, or whether they believe in a social perspective of interaction. Fourteen participants were interviewed to ascertain their personal constructs for assessing group oral interaction, as well as their justifications for rating a videotaped group oral performance using these personal constructs. The findings show that while raters valued a number of qualities including linguistic abilities, their focus was on interaction. This suggests an inclination towards the social interactional perspective, as they seem well aware that successful interaction is co-constructed, and cannot be achieved through individual language ability alone.

The Influence of Interlocutor Personality on Discourse in Pair and Group Oral Tests

The Influence of Interlocutor Personality on Discourse in Pair and Group Oral Tests

Detecting development of speaking proficiency with a group oral test: A quantitative analysis

This paper is an investigation into the use of the group oral discussion test (GOT) to detect changes in speaking proficiency over a two-year period. In this test, three or four test-takers discuss a topic for up to 10 minutes without outside intervention. The performance of 53 Japanese university English major students on this test was videoed before their classes started and at the end of their first and second years of study. Indices of complexity, accuracy, and fluency were calculated and interactive function was analysed to create indices that tracked initiating, responding, developing, and collaborating functions. Improvements were detected in most of the indices over the three administrations, with varying patterns of development. However, the test-takers’ scores in five rated scales only improved significantly in the second administration. Possible reasons for this discrepancy are discussed, as are the implications this study has for the GOT format and its administration.

The effects of test-taker characteristics on performances in pair and group oral tests

The effects of test-taker characteristics on performances in pair and group oral tests

Assessing English Language Learners’ Oral Performance: A Comparison of Monologue, Interview, and Group Oral Test

ABSTRACTIn the present study we investigated the effect of test format on oral performance in terms of test scores and discourse features (accuracy, fluency, and complexity). Moreover, we explored how the scores obtained on different test formats relate to such features. To this end, 23 Iranian EFL learners participated in three test formats of monologue, interview, and group oral test. Four raters rated the recorded productions holistically and analytically. The results of Friedman and Wilcoxon Signed Rank Tests indicated that the participants obtained the highest scores on the group oral test followed by the monologue and interview, although the differences between the group oral test and monologue were not statistically significant. Analysis of the produced discourse also indicated significant differences among the three test formats. The most accurate production was found on the group oral test and the most complex production occurred in monologue. Further analysis indicated that accuracy features were significantly related to both analytic and holistic ratings in all the test formats. The study highlights the complex relationship that exists between the features of the discourse produced on a test task and the factors raters consider in rating.

How Does Form-Focused Instruction Affect Korean EFL Students in a Communicative Language Teaching Classroom?

This study aims to explore how Form-Focused Instruction (FFI) in a CLT context would affect Korean EFL university students’ oral communicative competence as well as their perceptions. Fifty two students participated in this study, and 26 in the FFI class were instructed to learn specific grammatical features with communication skills; whereas the rest of 26 in the non-FFI class were never given any grammar instruction or feedback. The two classes’ performance on the group oral conversation test and their responses to the survey questionnaire were, respectively, compared. The test results showed that the FFI class students outperformed those in the non-FFI class in all areas including grammar, vocabulary, content, fluency, turn-taking, and non-verbal behaviors. In addition, the survey results demonstrated that the FFI class students thought they had improved particularly grammar and fluency, and thus gained greater confidence and motivation; whereas the non-FFI class students were concerned much about their grammatical correctness which made them difficult to continue conversation. It can be thus confirmed that FFI in a CLT classroom would help students enhance their communicative competence and confidence.

Book Review: The Co-construction of Conversation in Group Oral Tests

Book Review: The Co-construction of Conversation in Group Oral Tests

Nakatsuhara, Fumiyo: The Co-construction of Conversation in Group Oral Tests

Nakatsuhara, Fumiyo: The Co-construction of Conversation in Group Oral Tests

The influence of prompt on group oral tests

This paper reports an investigation into how the prompt may influence the discourse of group oral tests. The group oral test, in which three or four participants are rated on their ability to discuss a prompt, is a format for assessing the spoken ability of language learners. In this study, 141 Japanese university students were videoed in 41 group orals of three or four test-takers. Although the four different prompts written for the test were supposed to be of equal difficulty, they were found to be substantially different in the type and number of questions that comprised them. Analysis of the transcribed interactions revealed significant differences in turns taken, syntactic complexity and fluency of the interactions they elicited. A qualitative examination revealed that the two prompts that elicited longer, more complex turns did so by encouraging test-takers to explain their family circumstances or speculate about their future. Prompts with more factual content elicited shorter, less complex turns and the prompt that test-takers responded to with the least fluency required the test-takers to talk about a more personal subject. The implications for rating and creating prompts are discussed, and the need to tailor them to the purpose of the test.

Stakeholder Input and Test Design: A Case Study on Changing the Interlocutor Familiarity Facet of the Group Oral Discussion Test

Test takers should have a voice in testing practices (Mathew, 2004). However, when incorporating their input, systematic processes to ensure the validity of testing practices must be followed. Such processes allow for test development to be a more democratic process (Shohamy, 2001), without sacrificing the value of the resulting inferences made from the test scores. This article describes a case study that incorporated the desires of test takers to change the procedures of a group oral discussion test in a university English as a Foreign Language program. A study was designed to determine the extent to which the proposed changes would threaten the validity of the testing process. Specifically, the procedures for the group oral were altered to investigate the effect of interlocutor familiarity. Students were randomly assigned to class-familiar (n = 146) and class-unfamiliar (n = 159) groups to identify to what extent group familiarity affected test takers' scores in the four assessed categories: pronunciation, fluency, lexis and grammar, and communication skills. For the two groups, no statistically significant difference in scores was found, and score reliability estimates were similar. The implications of the findings are addressed in terms of recommendations for using stakeholder input in the assessment design process.

Effects of test-taker characteristics and the number of participants in group oral tests

This study explores the nature of co-constructed interaction in group oral tests by examining whether a test-taker’s own and his or her group members’ extraversion levels and oral proficiency levels have different influences on conversational styles between two group sizes: groups of three and groups of four. Data were collected from 269 Japanese upper-secondary school students, who took group oral tests either in groups of three or four. All sessions were video-taped and transcribed following Conversation Analysis (CA) conventions. The data were quantitatively analysed in terms of goal-orientation, interactional contingency and quantitative dominance. Then, CA methodology was used to interpret and elaborate the statistical results. The findings have implications for our understanding of the group oral test construct and for appropriate choices of group size in group oral testing.

Self‐consciousness and Assertiveness as Explanatory Variables of L2 Oral Ability: A Latent Variable Approach

Drawing on current theories in personality, second‐language (L2) oral ability, and psychometrics, this study investigates the extent to which self‐consciousness and assertiveness are explanatory variables of L2 oral ability. Three hundred sixty first‐year Japanese university students who were studying English as a foreign language participated in the study. Personality was measured with the Japanese version of the NEO‐PI‐R (Shimonaka, Nakazato, Gondo, & Takayama, 2002), and L2 oral ability was assessed with an institutionalized group oral discussion test in which performances were judged for pronunciation, fluency, grammar, vocabulary, and communication skills. An analysis of covariance structures technique indicated that assertiveness is a significant explanatory variable of L2 oral ability and no effect was found for self‐consciousness. Communication skills and fluency were slightly better explained by assertiveness than the other subconstructs of oral ability.

The effects of group members' personalities on a test taker's L2 group oral discussion test scores

The second language group oral is a test of second language speaking proficiency, in which a group of three or more English language learners discuss an assigned topic without interaction with interlocutors. Concerns expressed about the extent to which test takers' personal characteristics affect the scores of others in the group have limited its attractiveness. This study investigates the degree to which assertive and non-assertive test takers' scores are affected by the levels of assertiveness of their group members. The sample of test takers was Japanese first year university students who were studying English in Japan. The students took the revised NEO-PI-R (Costa & McCrae, 1992; Shimanoka et al., 2002), a group oral test, and PhonePass SET-10 (Ordinate, 2004). Two separate MANCOVA analyses were conducted, one designed to determine the extent to which assertive test takers' scores are affected by the levels of assertiveness of group members (N = 112), and one designed to determine the extent to which non-assertive test takers' scores are affected by the levels of assertiveness of group members (N = 113). The analyses indicated that assertive test takers were assigned higher scores than expected when grouped with only non-assertive test takers and lower scores than expected when grouped with only assertive test takers, while the study failed to find an effect for grouping based on assertiveness for non-assertive test takers' scores. The findings of the study suggest that when the group oral is used, rater-training sessions should include guidance on how to evaluate a test taker in the context of the group in which the test taker is assessed and assign scores that are not based on a comparison of proficiencies of group members.

Washback of group oral tests for Japanese language in a classroom environment

Washback of group oral tests for Japanese language in a classroom environment

Language testing

07–449Barber, Richard (Dubai Women's College, UAE), A practical model for creating efficient in-house placement tests. The Language Teacher (Japan Association for Language Teaching) 31.2 (2007), 3–7.07–450Chang, Yuh-Fang (National Chung Hsing U, Taiwan), On the use of the immediate recall task as a measure of second language reading comprehension. Language Testing (Hodder Arnold) 23.4 (2006), 520–543.07–451Hyun-Ju, Kim (U Seoul, Korea), World Englishes in language testing: A call for research. English Today (Cambridge University Press) 22.4 (2006), 32–39.07–452Mahon, Elizabeth A. (Durham Public Schools, North Carolina, USA), High-stakes testing and English language learners: Questions of validity. Bilingual Research Journal (National Association for Bilingual Education) 30.2 (2006), 479–497.07–453McCoy, Damien (Australian Centre for Education and Training, Vietnam), Utilizing students' preferred language learning strategies for IELTS test preparation. EA Journal (English Australia) 23.1 (2006), 3–13.07–454Menken, Kate (City U New York, USA), Teaching to the test: How no child left behind impacts language policy, curriculum, and instruction for English language learners. Bilingual Research Journal (National Association for Bilingual Education) 30.2 (2006), 521–547.07–455Pae, Tae-Il (Yeungnam U, China) & Gi-Pyo Park, Examining the relationship between differential item functioning and differential test functioning.Language Testing (Hodder Arnold) 23.4 (2006), 475–496.07–456Rimmer, Wayne (U Reading, UK), Measuring grammatical complexity: The Gordian knot. Language Testing (Hodder Arnold) 23.4 (2006), 497–519.07–457Rupp, André A. (Humboldt U, Berlin, Germany) Tracy Ferne & Hyeran Choi, How assessing reading comprehension with multiple-choice questions shapes the construct: A cognitive processing perspective. Language Testing (Hodder Arnold) 23.4 (2006), 441–474.07–458Vanderveen, Terry (Kangawa U, Japan), The effect of EFL students' self-monitoring on class achievement test scores. JALT Journal (Japan Association for Language Teaching) 28.2 (2006), 197–206.07–459Van Moere, Alistair (Lancaster U, UK), Validity evidence in a university group oral test. Language Testing (Hodder Arnold) 23.4 (2006), 411–440.

Validity evidence in a university group oral test

This article investigates a group oral test as administered at a university in Japan to find if it is appropriate to use scores for higher stakes decision making. It is one component of an in-house English proficiency test used for placing students, evaluating their progress, and making informed decisions for the development of the English language curriculum. The implementation of a cut-score for students to advance through the university system has recently been proposed, bringing the group oral test component under increased scrutiny. On two successive occasion 113 participants sat the oral test in groups composed of different interlocutors each time. Rasch analysis shows rater fit within acceptable levels considering the length and nature of the test; however, at correlations of .74 inter-rater agreements are lower than has been reported in research on commercially available interview tests. Candidates’ scores on the two different test occasions correlate at .61. A generalizability study shows that the greatest systematic variation in test scores is contributed by the person-by-occasion interaction. Topic, or prompt, was not a significant factor. Candidates’ performances, or how raters perceive an individual candidates’ ability, could be affected to a large degree by the characteristics of interlocutors and interaction dynamics within the group.