Test Of Academic Literacy Levels Research Articles

USING ACADEMC LITERACY LEVEL AS A MEASURE FOR ACADEMIC PREPAREDNESS: A UNIVERSITY OF TECHNOLOGY CASE STUDY

Ponte Academic JournalFeb 2023, Volume 79, Issue 2 USING ACADEMC LITERACY LEVEL AS A MEASURE FOR ACADEMIC PREPAREDNESS: A UNIVERSITY OF TECHNOLOGY CASE STUDYAuthor(s): Jeremiah Madzimure ,Muntuwenkosi ChiliJ. Ponte - Feb 2023 - Volume 79 - Issue 2 doi: 10.21506/j.ponte.2023.2.5 Abstract:National statistics suggest that drop-out rates for first year students entering science, technology engineering and mathematics (STEM) programmes are disproportionately higher across all universities in South Africa. Predicting first-year student success in these programmes has thus become a priority area for researchers in the student academic support and development centres. This paper reports on the results of a pilot study that sought to draw inferences about the predictive value of the Test of Academic Literacy Levels (TALL) at one university of technology in South Africa. The purpose of this test is to inform academic staff of entry profiles of students and to assign students to tutorial groups and other academic support programmes. TALL was meant to be written by all new students taking mathematics and science courses in the years 2015 (n = 1240) and 2016 (n = 2133). The findings indicate poor performance for most students who took the test. The findings indicate that the test is not adequate for predicting future academic performance of first year students at tertiary institutions. The results of the current study have informed the University about the diagnostic value of the tool and the academic profile of students which subsequently lead to improved student academic support initiatives. Download full text:Check if you have access through your login credentials or your institution Username Password

Translating the Test of Academic Literacy Levels into Sesotho

This article reports on an initiative at the Vaal Triangle Campus (VTC) of North-West University (NWU) towards the active encouragement of functional multilingualism in tertiary education. More specifically, in an effort to support the potential use of Sesotho (Southern Sotho) and isiZulu in the teaching-learning environment at this campus, it was decided that an important first step should be an accurate and reliable determination of students’ levels of academic literacy (AL) in these two languages. Such a measurement would provide an indication of student readiness to make productive use of any extra learning support offered in these languages. Since there are no measuring instruments available to assess students’ academic literacy levels in Sesotho and isiZulu at tertiary level, it was decided thattranslating an existing test may be the most productive option in this case. The test selected for translation into Sesotho and isiZulu was the Test of Academic Literacy Levels (TALL), a test that has consistently shown high reliability measures in the assessment of students’academic literacy levels in English. Because Sesotho is the home language used by the majority of students (other than English and Afrikaans) at the VTC, it was decided that Sesotho would be treated as priority in terms of the completion of the project. The article therefore documents the translation process for Sesotho, and discusses primarily positive findings regarding the creation of a conceptually and functionally adequate Sesotho version of the TALL through a process of translation.

The differential predictive validity of a test of academic literacy for students from different English language school backgrounds

In the past decade, testing for academic readiness has become common practice across the South African higher education landscape. The aim of this assessment is to identify students who might not be ready for academic education and to put support programmes in place to help bridge this gap. Given the implied contribution that these programmes are expected to make towards higher graduation rates, it is important that the tests used for identifying those who need the support are valid for all students regardless of background. The aim of this study was to investigate the differential predictive validity of a test of academic literacy for two groups of students; those who took English at Home Language level and those who took it as First Additional Language at school. A regression analysis of a total of 564 scores obtained by the two groups on the Test of Academic Literacy Levels (TALL) as a predictor of their end of first-year average scores was carried out to determine this. The results show that although the Home Language group performed better than their counterpart on the test on average, the test predicted the outcome variable marginally better for the First Additional Language group than it did for their counterpart. This difference was too negligible, however, for one to conclude that the test was predictively biased against any of the two groups.

ACADEMIC LITERACY OF SOUTH AFRICAN HIGHER EDUCATION LEVEL STUDENTS: DOES VOCABULARY SIZE MATTER?

This study explores the extent to which vocabulary size matters in academic literacy. Participants (first-year students at North-West University) were administered the Vocabulary Levels Test (Schmitt, Schmitt and Clapham 2001). Scores from the test were used to estimate students’ vocabulary size and were subsequently mapped onto the levels distinguished by the Test of Academic Literacy Levels (TALL). Estimates show that, on average, the vocabulary size of first-year students at North-West University is approximately 4,500 word families, a size large enough to allow them to follow lectures in English. Furthermore, students with large vocabularies were found to have higher academic literacy proficiency, which establishes a strong relationship between vocabulary size and academic literacy. This relationship was also observed at the different word frequency bands the Vocabulary Levels Test consists of. These results support previous findings which established a relationship between vocabulary size and reading (cf. Nation 2006), and between vocabulary size and overall language proficiency (cf. Beglar 2010, Meara and Buxton 1987, Meara and Jones 1988, Nation and Beglar 2007), which could be extended to academic literacy. Furthermore, a stronger relationship between vocabulary size and academic literacy was found towards more infrequent word bands, indicating that infrequent word bands may best predict academic literacy. On the basis of these findings, we discuss possible strategies to adopt in order to assist some first-years with expanding their vocabularies.

The relationship between self-efficacy and reading proficiency of first-year students: An exploratory study

Self-efficacy, which is the belief about one’s ability to perform a task successfully, has been widely acknowledged as important in learning. This affective factor, though not explicitly evident, has been said to play an important role in academic performance. However, its role in reading development has not been widely investigated. To determine the relationship between self-efficacy and reading proficiency, a study was conducted with first-year students in a South African tertiary institution. Students’ self-efficacy levels were elicited through a questionnaire and their reading proficiency was obtained through the Test of Academic Literacy levels (TALL), which essentially assesses reading proficiency. An analysis of variance showed a robust relationship between reading self-efficacy and reading proficiency for this cohort of students. Regression analysis conducted with other affective factors showed self-efficacy as the best predictor of students’ reading proficiency. Results are discussed as they relate to previous research and recommendations are made to include the development of self-efficacy in reading instruction.

Disparate Impact, Justice and fairness: A case study of the Test of Academic Literacy Levels

The teaching of academic literacy has become critically important at South African universities in the post-apartheid period. The reason for this is that universities that were previously exclusively accessible to white students are currently within reach of non-white students. Most of these students, however, graduated from public schools where they received poor education in English, a medium of instruction at most universities. A result of this has been that the students struggle to handle the demands of university education in the language. This contributes to their failure to complete their studies in scheduled time and to drop out. As a way to deal with this challenge, Higher Education South Africa (HESA) has introduced a National Benchmark Test of academic literacy to assess the reading, writing and thinking abilities of these students to ensure their proper placement at universities. The Inter-institutional Centre for Language Development and Assessment (ICELDA), a partnership of the Universities of Pretoria, Stellenbosch, North-West and Free State, has also developed a test of academic literacy known as the Test of Academic Literacy Levels (TALL) for the same purpose. This paper was a case study of the impact, justice and fairness of this test. The findings were that the test possessed an acceptable degree of justice and fairness and that it aimed for a positive impact on the test-taker.

The relationship between productive knowledge of collocations and academic literacy in tertiary level students

This research explores tertiary level L2 students’ mastery of the collocations pertaining to the Academic Word List (AWL) and the extent to which their knowledge of collocations grows alongside their academic literacy. A collocation test modelled on Laufer and Nation (1999), with target words selected from Coxhead’s (2000) AWL was administered to students from theNorth-West University, Potchefstroom Campus. Results indicate that these collocations pose problems for L2 students. This finding supports and complements Nesselhauf’s (2005) and Laufer and Waldman’s (2011) findings that collocations are challenging for L2 students, even at advanced levels. Furthermore, scores in the collocation test correlate significantly with the Test of Academic Literacy Levels (TALL), indicating that knowledge of collocations corresponds to students’ academic literacy. The pedagogical consequences these results have for the widespread attempt to improve tertiary level students’ use of collocations in writing are discussed1. Key words : productive knowledge of collocations, academic literacy, academic word list, L2 writing

The relationship between socio-affective factors and reading proficiency: Implications for tertiary reading instruction

Research in the field of reading literacy has focussed predominantly on the foundation phase and primary and secondary levels. In addition, these studies in reading literacy are predominantly cognitive-oriented. As a result, information on academic reading at tertiary level is sparse; even more so with regard to socio-affective factors and students' reading proficiency. This paper explores the relationship between tertiary-level students' socio-affective profile and their reading proficiency. The Test of Academic Literacy Levels (TALL) was used to measure students' reading proficiency, and ANOVA tests were applied to analyse and yield data on students' reading background, reading habits, social literacy and several motivational components. The results and the data so obtained indicated a robust relationship between these socio-affective factors and students' reading proficiency. The paper discusses the research findings and their implications for instructing tertiary-level students in a way that would ensure successful academic reading.

Reading ability and academic acculturation: The case of South African students entering higher education

First-year students experience a range of challenges when transferring from secondary to higher education (HE) (cf. Darlaston-Jones et al. 2003, Leki 2006, Brinkworth et al. 2009). This is no different in South Africa, where deviating levels of preparedness for the demands of HE is a recurring theme (Slonimsky and Shalem 2005, Van Schalkwyk 2008, Scott 2009, Yeld 2009, Van Dyk 2010, Van Dyk and Coetzee-Van Rooy 2012). Weideman (2003:56) rightfully points out that the inability to understand and utilise appropriate academic discourse has a detrimental effect on academic success. Young students need to acculturate to the academic environment while adopting the academic community’s currency (Van de Poel and Gasiorek 2012a:294). With this article, we wish to contribute to the discussion by reporting on the academic language ability of one group of first-year students at a South African university, with specific reference to these students’ reading ability, on the basis of the following data: (i) individual differences in terms of learner characteristics (race, first language, gender, Grade 12 results, academic performance); (ii) self-reported reading preparedness, and (iii) reading profiles resulting from a valid and reliable academic literacy test, the Test of Academic Literacy Levels (TALL) and its Afrikaans counterpart, the Toets van Akademiese Geletterdheidsvlakke (TAG). The findings suggest that academic reading ability, as reflected in the test results, is indeed one of the salient contributors to academic success (as confirmed in the literature), regardless of social and individual differences, and that it needs to be supported in order for students to perceive their reading ability in accordance with their reading performance and be able to progress in their academic acculturation. A follow-up study will report on students’ awareness-raising about their own academic reading through the use of the validated scale for Perceived Academic Reading Preparedness (PARP) as a pedagogical tool.

Facing up to literacy: perceptions and performance in a test of academic literacy for postgraduate students

Language proficiency and academic literacy tests such as the National Benchmark Test (NBT) and Test of Academic Literacy Levels (TALL) are already well established assessment instruments that are widely used at universities in South Africa to assess the literacy levels of first-entry students. A more recent initiative has been the institution of language testing at postgraduate level as a means of identifying students at riskof not completing their academic studies at that more advanced level. This article examines the face validity of the Test of Academic Literacy for Postgraduate Students (TALPS) amongst a cohort of postgraduate students at the University of the Free State and the perceptions of these students as to their own levels of academic literacy. A correlation is made with the students’ actual performance in the TALPS as an initial step towards gaining a measure of understanding of the low levels of academic literacy of some postgraduate students. The results of the study show that, although most of the students consider the TALPS to be fair and accurate, there is a major discrepancy between the perceptions of their own academic literacy levels and their actual test performance. Possible reasons for the disparity are gleaned from the responses provided by the students in the survey questionnaire and an analysis of their test scores. It would seem that the academic literacy levels of students may not increase substantially during the undergraduate phase of study, a phenomenon that reflects back on language issues, course electives and undergraduate teaching and assessment practices.Key words: academic literacy, language testing, face validity, test construct, postgraduate assessment

Test and context: The use of the Test of Academic Literacy Levels (TALL) at a tertiary institution in VIÊT NAM

Concern for the academic success of undergraduate students with low literacy levels is by no means unique to South Africa. In response to initiatives by the Ministry of Education and Training to standardise and improve the quality of higher education in the country in accordance with international standards, the University of Đà Nẵng in Việt Nam recently tested out a pilot version of the South African designed Test of Academic Literacy Levels (TALL) at its College of Foreign Languages (CFL). This article examines the appropriateness of the TALL for use at an institution such as the CFL in Việt Nam and compares the literacy levels of students studying at the college with those of a similar group of students at the University of Pretoria. The research findings show that the TALL exceeds the set requirements for international language tests in terms of reliability and that it has high construct validity. In addition hereto the test has proven to be highly feasible to administer, particularly with due consideration to time constraints and practicality. Academic literacy levels were found to be lower amongst the cohort of Vietnamese students than amongst a comparable group of South African students. Recommendations for the continued employment of the test in Việt Nam include doing differential item function analyses to explore gender and cultural differences, the minor refinement of less productive task items, and a slightly shorter test version.

Examining bias in a test of academic literacy: Does the <i>Test of Academic Literacy Levels (TALL)</i> treat students from English and African language backgrounds differently?

Responsible test design relies on close examination of a number of parameters of a test. After finding a clearly argued, rational basis (construct) for the ability being tested, then articulating this in detailed specifications for subtests and item types, and subsequently setting benchmarks for both test reliability and item productivity, there remains, after the results become available, a number of further dimensions of a test that need attention. This article examines one such dimension: that of Differential Item Functioning (DIF), asking whether there is, in the case of the test under consideration, bias towards a certain group of test-takers (testees), so that they are unfairly disadvantaged by some of the items or task types in the test. The test results across four different years (2005-2008) of a large group of first year students, the bulk of the intake at one South African university, are analysed. The fact that there are variations in DIF across the different years and across different task types (subtests) calls for specific explanations. The findings suggest that one would do well to examine test results in depth, in order to avoid conclusions that may be fashionable but inaccurate. However, the argument returns to the defensibility of the test construct, and what should legitimately be included in that, and, by extension, measured.Keywords: test design, subtests, item types, Differential Item Functioning (DIF), bias, test results, defensibility, measurement

Towards providing effective academic literacy intervention

This article focuses on the current situation in South Africa in which a number of factors related to academic literacy deleteriously affect student throughput. This has large financial implications for the country as well as tertiary institutions. Since significant numbers of students with high academic potential are at risk because of their low levels of academic literacy, all first year students at three South African universities write the Test of Academic Literacy Levels (TALL). This test can be used to assign students to appropriate support courses. An explanation illustrated by empirical data is given of the construct and the use of TALL. Preliminary conclusions in terms of the impact of the intervention are drawn as first iteration towards an in-depth longitudinal study to test the effectiveness/success of the academic literacy intervention. Finally suggestions for further research are outlined.

The influence of tutorials on the improvement of tertiary students’ academic literacy

This paper describes a study which explored whether attending tutorials facilitated by senior students improves tertiary students’ marks in an academic literacy course. Data were drawn from the academic literacy test marks and essay assignment marks of students who took a year-long academic literacy course at the University of Pretoria. A statistical analysis was done to determine whether there was any improvement in the test marks and the essay assignments of students who attended tutorials at least 10 times, in addition to regular lectures. Findings suggest that attending extra academic literacy tutorials has little effect on students’ TALL (Test of Academic Literacy Levels) marks, but it does seem to have an influence on students’ writing ability. Because of the number of problems encountered in doing the research, it also illustrates the complex and problematic nature of research into language performance.

Put listening to the test: An aid to decision making in language placement

Universities and other higher education establishments throughout the world, including South Africa, have become concerned about the academic literacy levels of the students they enrol. The problem at most South African tertiary education institutions, which is in line with global trends, is certainly considerable, with almost a third of the students identified as being at risk. A lack of ability in academic discourse is seen as a major cause of students' failure to complete their courses within the given period. In 2006, as part of a nationwide attempt to remedy the academic literacy crisis, Stellenbosch University, along with other academic establishments, officially decided to implement a test of academic literacy in both English and Afrikaans. At Stellenbosch University, the English version of this test is known as TALL (Test of Academic Literacy Levels). It was developed to assess reading and writing abilities in an academic context. The results are used to ‘stream’ students into programmes which assist them in acquiring the various skills deemed necessary for their academic success. Students are sorted according to their TALL results into 'high risk' and 'low to no risk' categories, however, a need has been identified for further screening of the borderline students whose performance in the test falls between these two groups. Administrative and logistical limitations have, thus far, prevented listening skills from being included in the construct of TALL, but there is general consensus that listening is an important skill, particularly at university level. The focus of the research project, reported on in this article, was to design, and put into practice, an academic listening test (ALT) to assist in decision making regarding the placement of first-year students in approved language courses at Stellenbosch University. The qualitative and quantitative results obtained from the various administrations of ALT were analysed to determine the reliability and validity of the test. The final phase of the study involved the correlation of these results with those of TALL to establish whether ALT could assist the TALL administrators in making more informed decisions.

Gender bias and gender differences in two South African tests of academic literacy

How much empirical evidence is there of gender bias in the Test of Academic Literacy Levels (TALL) and its Afrikaans counterpart, Toets van Akademiese Geletterdheidsvlakke (TAG)? This paper reports on meta-analyses of 24 academic tests, containing data of more than 60 000 first-year South African students. TALL appears to be gender-neutral, while for TAG some evidence was found that men slightly outperform women. This outcome can be attributed almost entirely to male students outperforming female students on the subtest requiring the interpretation of graphs and visual information. Women outperform men slightly on the subtest dealing with text types. These outcomes are presented in light of discussions on gender differences versus gender bias.

Constitutive and regulative conditions for the assessment of academic literacy

If we characterise language tests as applied linguistic instruments, we may argue that they therefore need to conform to the conditions that apply to the development of responsible applied linguistic designs. Conventionally, language tests are required to possess both validity and reliability; these are necessary conditions for such tests, and so is their theoretical defensibility (‘construct validity’). The new orthodoxy, however, is that test designers must also seek consequential validity for their instruments, in that they have to consider test impact. Using an emerging framework for a theory of applied linguistics, this paper outlines how a number of constitutive or necessary conditions for test design (their instrumental power, their consistency and theoretical justification) relate to other current concepts. These recently articulated notions include the ideas of test acceptability, utility, and alignment with both the instruction that follows and with the real or perceived language needs of students; as well as their transparency, accountability and care for those taking them. The latter set of ideas may be defined as regulative or sufficient conditions for language tests. These concepts will be illustrated with reference to the design of a test of academic literacy levels that is widely used in South African universities.

Revisiting test stability: further evidence relating to the measurement of difference in performance on a test of academic literacy

In several earlier analyses of two tests of academic literacy—the Test of Academic Literacy Levels (TALL) and its Afrikaans counterpart, the Toets vir Akademiese Geletterdheidsvlakke (TAG)—we have adopted an approach to the problem that tests may be abused (and therefore used to harm people) by discussing various antidotes to this, so as to ensure fairness and consistency in the tests we use, as well as by demonstrating how the process of test development and implementation might be made more transparent. We have pointed out that a true measure of the stability of such tests may only become apparent when we have data stretching across a number of years. This article reports on an investigation of differing measures of performance on the two tests in question across several of their administrations between 2005 and 2008. We point to a number of ways in which empirical measures may be used to achieve benign, enabling effects on those tested.

Transparency and accountability in applied linguistics

The designed solutions to language problems that are the stock-in-trade of applied linguistics affect the lives of growing numbers of people. By calling for these designs to be accountable, applied linguistics has, in its most recent postmodern form, added an ethical dimension that is lacking in earlier work. Postmodern understandings of the field echo many valid concerns that were first raised several decades ago in other fields (cf Stafleu, 2004: 107): the non-neutrality of science, and a critique of progressivism and scientific hubris. The paper shows how this discipline has struggled with issues of continuity, integrity and validation, before analysing the ideas of transparency and accountability as an ethical concern in current applied linguistics. Illustrations are given of attempts at greater transparency and accountability for a typical applied linguistic artefact, the Test of Academic Literacy Levels (TALL) that is used by three South African universities.

Assessing academic literacy: A task-based approach

The Test of Academic Literacy Levels (TALL) used by three South African universities (Pretoria, Stellenbosch and North-West) provides a reliable and affordable alternative means of assessing the academic literacy of new entrants into the higher education sector. A close alignment is sought between the test, the task-based language instruction that follows its administration, and the learning and acquisition aimed for. The article critically examines the construct of the test as well as its task types in light of various current discussions about authenticity. The article is concluded by suggesting a number of possible alternative task types that may achieve a closer alignment with the goals embodied in the construct. Various developmental, contextual, administrative and logistical constraints appear, however, to affect the level of resemblance to academic discourse of the test task types.