Curricular Validity Research Articles

Constructing and validating authentic assessments: the case of a new technology-based assessment of economic literacy

BackgroundAuthentic situations are considered a source of learning due to their real world relevance. This can encourage learners to acquire new knowledge. Increasing digitisation and associated resources, such as professional development opportunities for teachers, technology tools, or digital equipment for schools enable the development and implementation of authentic assessments. The basic academic principles for acquiring economic literacy are already provided in lower secondary school. This article examines, using the example of a new authentic technology-based assessment (TBA)—Economic Literacy—Assessing the Status Quo in Grade 8 (ECON 2022) -, the processes involved in constructing a TBA. The purpose is to develop a curricular valid measurement instrument for surveying the current state of economic literacy in the 8th grade of a German federal state. This study explores which economic competencies students—typically between 14 and 15 years of age—possess in Grade 8, and what level of competence can therefore be expected of them at the beginning of a vocational training programme. The assessment is geared toward the curriculum of the subject of economics and is based on a domain model. This article presents the background and construction process for the development of ECON 2022 as a TBA.MethodsTo check the validity of test construction with a focus on the implementation of the authentic assessment and an analysis of difficulty-generating characteristics, the ECON 2022 test items were validated with an expert survey (N = 25). The two-stage data analysis comprised a descriptive quantifying analysis of the rating from the difficulty-generating characteristics specificity, cognitive demand and modelling and the design criterion authenticity. A set of experts rated the criteria. The expert survey was then compared with a previously conducted rating by the research team. The analysis of free-text comments on individual items was carried out discursively and qualitatively by the research team. Both sources of information were used to adapt the test items to measured item difficulties from the field test. For this purpose, items of great difficulty were changed to slightly easier items. In this context, the paper focuses on two central research questions: - How does the authenticity of a test environment relate to difficulty-generating criteria at item level? - Does the authenticity of a test environment have an impact on test results?ResultsResults are as follows. (1) The ECON 2022 assessment offers an example of a test design in which the use of TBAs can support innovative and interactive item development. (2) Using the ECON 2022 assessment enabled the implementation of an assessment of economic literacy using authentic situations and the mapping of different facets of economic competence. (3) The validation study showed that the actual item difficulty did not correlate significantly with the authenticity of the assessment, and authenticity thus did not contribute to item difficulty.ConclusionThe results of the study show that we were successful in developing an authentic TBA in ECON 2022. ECON 2022 allows us to examine economic literacy in schools with a high degree of curricular validity and relevance and to analyse what level of competence and knowledge can be expected of students when they enter a vocational training occupation.

Eureka Role Playing Game: An Instructional Tool In Improving The Proficiency Level In Chemistry 7

The Eureka Role Playing Game is a game-based instructional tool for learning science concepts. The study aimed to improve proficiency level of the learners in Science 7 particularly in Chemistry subject. This study utilized a Quantitative Method to measure the gathered data and test the hypothesis. Quasi-experimental pretest-posttest research design was also used with a validated questionnaire and Eureka Role-Playing Game (RPG) as the primary data gathering instrument to evaluate students’ proficiency level improvement as the intervention was administered to experimental group and traditional teaching method for controlled group. Results showed that in the controlled group, the mean post-test result was higher than the mean pre-test result, but still at the beginning level of proficiency. Meanwhile, the mean post-test result of the experimental group is higher compared to the mean pre-test result. Therefore, the level of proficiency of the experimental group has improved from a beginning level to a developing level after the intervention. A significant difference in the pre-test and post-test between the experimental group and the control group is in favor of the experimental group. Therefore, the alternative hypotheses of this study are accepted. Evaluated by the Science experts, the instructional tool passed the curricular validity criteria. The researchers recommend the use of Eureka Role Playing Game since it significantly increased the proficiency level of the students from beginning to developing level based on the findings.

Teachers’ professional knowledge for teaching early literacy: conceptualization, measurement, and validation

This study suggests a comprehensive conceptualization of teacher knowledge for teaching early literacy in primary schools. Following the discourse on the professional knowledge of teachers, we argue that teachers’ knowledge relevant to support reading and writing at the beginning of primary school education is multidimensional by nature: Teachers need content knowledge (CK), pedagogical content knowledge (PCK), and general pedagogical knowledge (GPK). Although research on teacher knowledge has made remarkable progress over the last decade, and in particular in domains such as mathematics, relevant empirical research using standardized assessment that would allow in-depth analyses of how teacher knowledge is acquired by pre-service teachers during teacher education and how teacher knowledge influences instructional quality and student learning in early literacy is very scarce. The following research questions are focused on: (1) Can teachers’ professional knowledge for teaching early literacy be conceptualized in terms of CK, PCK, and GPK allowing empirical measurement? (2) How do teachers acquire such knowledge during initial teacher education? (3) Is teachers’ professional knowledge a premise for instructional quality in teaching early literacy to students? We present the conceptualization of teacher knowledge for teaching early literacy in primary schools in Germany as the country of our study and specific measurement instruments recently developed by our research group. Assessment data of 386 pre-service teachers at different teacher education stages is used to analyze our research questions. Findings show (1) construct validity of the standardized tests related to the hypothesized structure, (2) curricular validity related to teacher education, and (3) predictive validity related to instructional quality. Implications for teacher education and the professional development of teachers are discussed.

When academic achievement (also) reflects personality: Using the personality-achievement saturation hypothesis (PASH) to explain differential associations between achievement measures and personality traits.

Students’ academic achievement is a key predictor of various life outcomes. The two most prominent measures of academic achievement are grades and standardized test scores. Both measures are commonly used during selection processes as well as for educational monitoring and accountability. Research has suggested that grades and test scores are strongly related to students’ characteristics (e.g., cognitive abilities) but might differentially reflect personality. In order to better explain differential personality-achievement associations, it is important to move beyond the dichotomy of grades versus test scores. To this end, we propose the personality-achievement saturation hypothesis (PASH), which suggests that associations between personality and achievement vary, depending on four main features of the achievement measure used: level of standardization, relevance for the student, curricular validity, and instructional sensitivity. The PASH suggests that conscientiousness should typically be more strongly associated with grade point average, followed by course grades and final examination grades, whereas openness should be more strongly associated with test scores. We used data from three large-scale studies (total N = 14,953) and aggregated our findings. In line with the PASH, conscientiousness was most strongly related to grades, which have lower standardization, moderate to high relevance, high curricular-validity, and high instructional sensitivity, whereas it had substantially weaker associations with more highly standardized, less curriculum-valid, and less instructionally sensitive measures. In addition, openness was most strongly related to highly standardized, less relevant, less curriculum-valid, and less instructionally sensitive measures in English. Implications for the ways in which achievement measures can be used are discussed.

Motor tests for primary school aged children: A systematic review

ABSTRACTMotor testing in education has gained in relevance in light of recent developments in educational systems (Köller & Baumert, 2012). The purpose of this systematic review was to identify studies using motor tests in primary school children aged 4 to 12 years and to synthetize information about the quality and application of the test instruments used. A systematic review of 910 records identified by a search of nine databases yielded 144 papers reporting motor testing in primary school age children, including 25 articles reporting on the quality of 20 test instruments. In the selected articles, tests grounded on the constructs of “motor abilities” and/or “motor skills” are the most frequent and are mostly used for monitoring purposes or for impact research. Tests based on the construct “motor competencies” have become popular only recently, being relevant for educational motor testing due to their curricular validity. The test instruments have been investigated towards their validity and reliability to different extents. However, for several test instruments factorial validity has not been established and more validation studies are needed to improve their psychometric quality especially if used in educational contexts.

Validity of Role Play in Sociocultural Competence Assessment in Year 12 Examination in Latvia

The article examines different facets of the role play task, its application in the Year 12 English language examination and the statistical analysis of its results with the purpose to validate the use of role play for testing the sociocultural competence in Latvia. The research method of the role play here combines the features of external validation (construct validation) with internal validation (contents and response validation). The statistical findings suggest that the role-play task ensures a reliable and valid method of assessment of the student performance; it also provides a means for reliable assessment of a large proportion of the curriculum, as the skills tested by role play differ from the skills tested by other examination tasks. The findings of the curricular validation suggest that removing role-play from the examination would undermine the curricular validity of the Year 12 examination. The contents analysis of the task suggests that the difference in the student performance level in Speaking is created by the impact of the assessment of the sociocultural competence as defined by Celce Murcia, Dörnyei, and Thurrell (1995).

Modellierung und Messung domänenspezifischer Problemlösekompetenz bei angehenden Industriekaufleuten

Daily working routines in offices increasingly require domain specific problem solving competence. According to our understanding, this involves - besides domain specific knowledge - self-regulation and metacognitive skills as well as non-cognitive competence facets (self-concept and interest). In the context of the German project network DomPL-IK (Domänenspezifische Problemlösekompetenz von Industriekaufleuten/Domain specific problem solving competence of industrial management assistants) we developed a multidimensional model of problem solving competence. Based on this model we created scenarios that require industrial management assistants in training to solve typical problems of their trade. We then tested the scenarios in a pilot study with 123 participants (n = 86 industrial management assistants, n = 37 management assistants in informatics). In the article at hand we focus on knowledge application. Problem solutions were analyzed in a fine-grained content analytic approach and then condensed into partial credits. Those were used to estimate competences with item response theory. As expected industrial management assistants score better than management assistants in informatics. This can be cautiously interpreted as an indicator for curricular validity of the scenarios. Moderate internal consistencies (reliabilities) and content-related difficulties in the problem solving process reveal room for improvement for the main study.

Editorial: Mobile Learning. Towards Curricular Validity in the Maelstrom of the Mobile Complex

Editorial: Mobile Learning. Towards Curricular Validity in the Maelstrom of the Mobile Complex

Technical Quality Criteria for Evaluating District Assessment Portfolios Used in the Nebraska STARS

Nebraska districts use different strategies for measuring student performance on the state's content standards. District assessments differ in type and technical quality. Six quality criteria were endorsed by the state. These criteria cover content and curricular validity, fairness, and appropriateness of score interpretations. District assessment portfolios document how well assessments meet these criteria. Districts receive ratings on how well their assessments meet each of the quality criteria and are given a rating from Unacceptable to Exemplary. This article presents these technical quality criteria and explains how they are (a) individually rated and (b) combined for the district's overall quality rating.

New Assessment Forms in Problem-based Learning: The value-added of the students' perspective

To an increasing extent, teachers and students are leaving the well-known, traditional learning and assessment avenues and moving towards new learning environments and modes of assessment. However, the road from theory to practice seems to be a rocky one. This article presents the quantitative and qualitative data gathered from a research project which focused on different quality aspects of two new assessment forms in problem-based learning. A written examination was used to assess the extent to which students are able to define, analyse and solve novel, authentic problems. Peer assessment was introduced for students to report on collaborative work during the tutorial meeting, and during the study period that follows these weekly meetings. Students' perceptions are used as a tool for explaining the strengths and weaknesses of both instruments and the learning environment in which they are embedded. The article suggests that both assessment forms have acceptable qualities. The written examination seems to have an acceptable curricular, instructional and criterion validity. For the peer assessment, generalisability coefficients for both groups were over 0.75. Peer assessment seems to be accurate. Peer marks seem to correlate well with tutor marks and final examination scores. However, the students' learning outcomes, as measured by the examination, are lower than expected. Staff members experience the students' learning activities as disappointing. In the Louvain case, there are indications that the students cannot accurately reflect on their own functioning. In order to highlight plausible explanations, students' perceptions of the learning-assessment environment are analysed. In both cases, students perceive a gap between their working in the tutorial groups and the assessment. These results offer a valuable input for teachers to formulate concrete recommendations for optimising their educational and assessment practices.

The Texas Model for Content and Curricular Validity

The Texas Education Agency (TEA) relied heavily on the test development process used for the Texas Assessment of Academic Skills (TAAS) in its defense presented during the GI Forum v. Texas Education Agency (1999) litigation. In this article, we present the Texas test development process. The process spans 3 years of assessment development that includes the 2 general areas of planning and maintenance activities. Each step of both the planning and maintenance stages is described in detail. In addition, each step of the process is either explicitly or implicitly linked to the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1985). The routine use of this process by TEA and its contractors (as well as the process itself) provided strong evidence of both content and curricular validity of the TAAS.

Diagnostic testing methods for skill assessment in reading, writing, and arithmetic. A critical review

The diagnosis of a specific developmental disorder of reading, writing and arithmetic can be made based upon individually applied standardized methods for testing scholastic achievement and IQ. To make the choice of suitable methods easier for the administrator of the test, a critical survey of German-language methods for assessing skills in reading, writing and arithmetic is presented. Test intention and psychometric properties for scholastic achievement are summarized. The methods are assessed with regard to their utility in the diagnosis of congenital alexia and dyscalculia. Supplementary suggestions for clinical assessment are given. In summary, beyond the primary school area there is a lack of current standardized methods that meet the current standards of quality. Particularly the assessment of arithmetic skills above those of the 4th grade level require resorting to methods of dubious curricular validity. Coverage is once again better for the upper elementary and middle school levels. There is a need for new constructions, respectively a need to update published scholastic achievement tests.

Students’ Perceptions of Coursework in the GCSE: the effects of gender and levels of attainment

Summary Summary Coursework is an integral part of the GCSE framework, valued for its motivational qualities and its curricular validity. It is a common perception, widely reported in the national press and educational media, that coursework can be held at least partly accountable for differential performances at GCSE; coursework, it is argued, advantages girls. This article reports on an analysis of data arising from a project which offered an opportunity to study current and post-GCSE students’ perceptions of coursework. The outcomes indicate that, when categorised by their relative levels of attainment, girls’ and boys’ perceptions show limited evidence of homogeneity. In other words, to suggest that girls’ and boys’ perceptions of coursework are a function of gender is a gross over-simplification. Other factors are at play and further, more specific and tailored research is essential if we are to understand how best to optimise the benefits that are claimed for coursework.

A protocol to assess the curricular validity of cases for PBL

Background: The clinical case is the focal point of problem‐based learning (PBL) groups and learning activities revolve around the content of the case. Purpose: The purpose of this study was to create a protocol to assess the curricular validity of PBL cases in an objective and systematic fashion by observing the overlap between the content and the learning objectives. Methods: A qualitative analysis was conducted to create a quantitative tool with a taxonomy of categories to be used in case analysis. Results: The protocol developed uses a content analysis approach, dividing a PBL paper case into its smallest components of information (unitized) and placed into mutually exclusive categories. These units are then compared to the stated case objectives to determine the degree of overlap. Conclusion: This protocol is presented as a tool to determine the curricular validity of a case or effectiveness at meeting educational objectives.

Contention and Consensus: The Development of the 1992 National Assessment of Educational Progress in Reading

Because of plans for state-by-state reporting of 1992 reading data from the National Assessment of Educational Progress (NAEP), we investigated the adequacy of the process used to develop the assessment, the degree to which it represents a consensus among professionals in the reading field, and its content and curricular validity. To carry out this investigation, we analyzed documents produced by NAEP, convened a 2-day panel of experts, held two public colloquia, conducted 50 interviews, and analyzed responses to a questionnaire completed by 627 leading educators. We found that the planning process did not include enough time to address some major concerns of the field. Despite this, there was widespread agreement that the 1992 NAEP in Reading represents important advances in reading assessment, including more open-ended responses, more authentic texts, and student choice about passages. But these very advances raise problems for test design and the interpretation and scoring of student responses.

Note on the Face Validity of Certification Competencies of Computer Science Teaching

Rating scale data on the face validity of competencies in computer science teaching were compared for three groups (150 practitioners, 67 district specialists, 101 faculty). Through the use of Boxplots certain trends were readily identified. Implications for curricular validity were discussed.

Sensitivity of Item Difficulties to Curricular Validity

The issue of the match of what is taught to what is tested recently has received increased emphasis. Many recent studies have examined the extent of the mismatch between a local set of objectives/instruction and the content of a nationally standardized test. Phillips and Mehrens (1985, 1987; see also Mehrens & Phillips, 1986) have conducted a series of studies that show that, within a district, different textbook series and informal curricula generally have no significant impact on test total, objective, or item scores. The present study explored further the curricular validity differences across textbooks and the relationship between item statistics and measures of curricular validity. In particular, this study sought to determine whether item p‐values appear to be related to measures of curricular validity based on three mathematics textbook series used at a given grade and in a previous grade. The results of this study indicated that the textbooks differed somewhat in content coverage when using a 180‐cell matrix classification. However, these differences were not great, especially when the textbooks in both grades 5 and 6 were considered. Further, all three series covered almost all of the 53 cells in the matrix covered by the Stanford Achievement Test, and the differences that did exist in textbook content coverage had no observable relationship to differences in item p‐values. In addition, the mean difficulty level of the Stanford Achievement Test items classified by cell were similar for students using the different textbooks, despite the differences among textbooks in location, presentation, and organization of content.

Customizing a Norm‐Referenced Achievement Test to Achieve Curricular Validity: A Case Study

Customizing a Norm‐Referenced Achievement Test to Achieve Curricular Validity: A Case Study

A Health Knowledge Test for the Seventh Grade

Abstract A valid and reliable instrument was developed for the purpose of measuring the extent of health knowledge possessed by seventh grade pupils. Appropriate vocabulary burden, level of difficulty, and curricular validity were primary factors considered in item formulation. Experimental test forms were subjected to an extensive refinement process prior to the construction of two final, alternate test forms. The alternate forms were administered in a random sample of public schools in Indiana. Analysis of test results yielded statistical evidence of the equivalence of forms.

Standardization of a Volleyball Knowledge Test for College Men Physical Education Majors

Abstract This study presents the procedures used to construct and standardize a valid test to measure volleyball knowledge of male students in college who are majoring in physical education and who have completed their course of instruction in this sport. These procedures include the establishing of curricular validity, item selection, item difficulty, item validity, test reliability, and establishing national norms. The test is reliable and is further designed for machine scoring. Norms for this test are presented in this article.