Abstract
This study attempted to evaluate the usefulness a CEFR (Common European Framework of Reference) alignment project ‘Matura’ listening test with respects of the item analysis, validity (i.e., content validity) and reliability (i.e., internal consistency reliability, scoring reliably). 93 students randomly selected from six secondary-school of different school-levels across three different regions in Austria completed the listening test. SPSS was employed for the statistical analysis. Facility value, discrimination index together with other descriptive statistics were reported for the item analysis. Content validity by a panel of expert judgments as well as the reliability of the listening test was further examined. Findings showed that: 1) the test paper was of average to high difficulty with its peak score locating around the mid-point, which allow higher education institutions to set an appropriate cut-score for decision making. Relatively widespread test scores further indicated its potential to efficiently discriminate learners of varied listening proficiency; 2) the majority of items in the listening test performed well and were qualitatively reasonably good. However, we did identify several problematic items with possible causes related to construct-irrelevance, too many possible answers, mismatch sign-posted words, fast audio speed, testing simply the background knowledge, and the heavy cognitive load required for “Not Given” option; 3) regarding validity and validity issues, construct-under representation and the less authentic listening materials were then spotted by careful content analysis. Though the statistical output reported relatively high reliability, results should be interpreted with caution. Based on findings, general evaluation of the test as well as implications for further improvement were revealed. Keywords: test evaluation, usefulness, content validity, reliability, Common European Framework of Reference, listening comprehension, item analysis DOI: 10.7176/JEP/11-11-01 Publication date: April 30 th 2020
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.