Development of a two-tier diagnostic test concerning genetics concepts: the study of validity and reliability

Abstract

Problem Statement: Students’ meaningful understanding of scientific concepts and topics has a major consideration in order for science education programmes’ to attain their goals. The effectiveness of science instruction, which aims to assure students’ complete and accurate understanding of science concepts, can only be effectively evaluated when it is measured with appropriate tools of measurement. Concept maps, multiple-choice tests and interviews, all of which are employed in determining students’ understanding of science concepts, have been supported as successful methods. However, each has its own limitations. In determining students’ understanding of concepts, the use of two-tier diagnostic tests – which evaluate students’ understanding better and which are easy to score and apply – is recommended. The items in two-tier multiple-choice diagnostic instruments are specifically designed to identify alternative conceptions and misunderstandings in a limited and clearly defined content area. Compared with the other approaches, two-tier tests are a more advanced approach as they consider students’ reasons and interpretations for their answers. Moreover, it is stated that, two-tier tests are valid and reliable instruments which evaluate student ideas better, which are easy to score and apply.Genetics is one of the biology topics that students have most difficulty in understanding. Researches demonstrate that most teachers and students hold the view that genetics topics are very difficult both to learn and to teach. Researches into determining students’ understanding of genetics concepts show that students have some lack and confused knowledge on understanding these concepts, their functions and their relations. Purpose of StudyMost genetics concepts have an imaginary and theoretical characteristic. Students’ learning of theoretical genetics concepts meaningfully depends on knowing hypothetical reasons of these concepts. Investigation into students’ knowledge regarding genetics concepts as well as reasons on which they base their knowledge is bound to generate significant and effective consequences in determining their understanding. Therefore, this research aims to develop a two-tier test which determines secondary education students’ understanding of genetics concepts, and to conduct validity and reliability studies of the test. MethodsThe two-tier test directed to determining students’ understanding of fundamental genetics concepts was developed by pursuing a 10-step method containing 3 main phases. First phase is about defining the content; second phase about obtaining information on students’ misunderstandings, and third one is about the development of the two-tier test. In first phase, test content was defined by propositional knowledge statements and a concept map. Afterwards, propositional knowledge statements were associated with concept map; and thus the internal consistent of the content was examined. In second phase, firstly relevant literature was examined so as to determine students’ misunderstandings. And secondly semi-structured interviews were conducted with 21 students in order to gain a deeper perspective of students’ understanding. And later they were given a multiple choice test with free-response answers in which students had to explain reasons for their answers. In third phase, two-tier test items that were composed of multiple choice content questions in first tier and in second tier a set of five justifications for all potentially selected responses to first tier were developed. The two-tier test is intended to determine students’ understanding of the fundamental concepts of genetics and the relations between them as well as the relation of cell divisions with inheritance. This test is composed of two-tier 14 items. It is multiple-choice type in both tiers, and the number of choices is 3 in first tier and 5 in second tier. In order to consider a question as correctly answered, it needs to be answered correctly in both content and reason tiers.Research was conducted with 231 secondary education students. The age range of research group students was between 16 and 19. Of the group, 118 (51.1%) were girls and 113 (48.9%) were boys. Students attended Anatolian high schools, State high schools, and high schools with an intensive foreign language programme, where the same biology curriculum had been applied.The data obtained from two-tier test application were made subject to item analysis and test analysis. Following analyses, difficulty and discrimination of items, functionality of distracters and test reliability were determined. Answers were sought to two basic questions for the purposes of validity determination: first, whether or not the items measured the content that was defined by propositional knowledge statements; and second whether or not the items were suitable to the level of the research group. Expert opinion was consulted for content validity, and was checked with propositional knowledge statements as well as with concept map. A specification grid was designed in order to determine face validity. As to construct validity, the method of group differences was utilized. Findings and ResultsAfter conducting item analysis difficulty and discrimination indices of each item were calculated. Item discrimination indices were found to between 0.34 and 0.74. These values demonstrated that items functioned in a satisfactory way. And item difficulty indices were between 0.25 and 0.67, and had a distribution in a large interval. Test mean was found as 6.02 and test difficulty as 0.43. Test reliability was found to be 0.86. The functionality of distracters analyses showed that distracters functioned well and that there was no need for modifications in choices. In order to determine construct validity, correlation between the data obtained from students of Anatolian high schools and state high schools plus high schools with an intensive foreign language programme was calculated, and was found to be 0.169. The fact that this value was low was an indicator of test construct validity. ConclusionsAll the findings obtained from the item analysis, validity and reliability analyses of two-tier test indicate that this test could be used reliably and in a valid way to determine secondary school students’ understanding of genetics concepts. Findings also show that students performed better when only first part of items was considered, than when both parts of items were considered. This indicates that students have selected the correct content response through rote learning without an understanding of the underlying reasons. RecommendationsThe fact that the test is a multiple-choice type makes it easy to apply and to score for teachers. In addition, thanks to the availability of reason questions, teachers will be able to evaluate students’ understanding of concepts more effectively. When effectively used, these tests are likely to contribute to students’ deeper understanding of science concepts.