Selected-response Questions Research Articles

The interactive reading task: Transformer-based automatic item generation.

Automatic item generation (AIG) has the potential to greatly expand the number of items for educational assessments, while simultaneously allowing for a more construct-driven approach to item development. However, the traditional item modeling approach in AIG is limited in scope to content areas that are relatively easy to model (such as math problems), and depends on highly skilled content experts to create each model. In this paper we describe the interactive reading task, a transformer-based deep language modeling approach for creating reading comprehension assessments. This approach allows a fully automated process for the creation of source passages together with a wide range of comprehension questions about the passages. The format of the questions allows automatic scoring of responses with high fidelity (e.g., selected response questions). We present the results of a large-scale pilot of the interactive reading task, with hundreds of passages and thousands of questions. These passages were administered as part of the practice test of the Duolingo English Test. Human review of the materials and psychometric analyses of test taker results demonstrate the feasibility of this approach for automatic creation of complex educational assessments.

The dynamic relationship between response processes and self-regulation in critical thinking assessments

Our aim was to explore higher education students’ response and self-regulatory processes plus the relationship between these, as evidenced in two types of performance-based critical thinking tasks included in the Collegiate Learning Assessment (CLA+) International instrument. The data collection consisted of 20 cognitive laboratories. The data were analyzed using a qualitative approach. The tasks were found to trigger different response and self-regulatory processes. Overall, the performance task evoked more holistic processes than the selected-response questions, in which students’ processes were more question-oriented. The results also indicated the entanglement of students’ response and self-regulation processes. Three self-regulation groups were identified. Students with versatile self-regulation skills were able to complete the task thoroughly, whereas students with moderate self-regulation skills faced challenges in monitoring and evaluating their performance. Students who were lacking in self-regulation struggled both with the task as a whole and their own progress. Implications for higher education are discussed.

Modern mathematics teaching with the use of computer-based tasks – reality or myth?

In this article, I present partial results of my research, conducted as part of the project entitled Introducing students aged 12-17 to solving computer-based tasks. Research shows that the vast majority of teachers are merely pretending to conduct modern teaching when making use of computer-based tasks during mathematics lessons. In reality, their lessons are typical and conservative, dominated by frontal work and methods based on knowledge transfer. Most often, lessons feature selected-response questions used to verify the student's knowledge, rarely implementing tasks which require the student to experiment by making use of a computer screen. This situation is caused by multiple factors, including: teachers' views on the place and role of using IT resources for teaching mathematics, insufficient number of computers, tablets, and smartphones in schools, not enough computer-based mathematical tasks with a significant didactic value, insufficient competence of teachers, outdated external exam system, incorrect teacher evaluation system.

Assessing the Application of Knowledge in Athletic Training: Extended-Matching Questions

Context The accuracy of summative assessment scoring and discriminating the level of knowledge in subject matter is critical in fairness to learners in health care professional programs and to ensure stakeholders of competent providers. An evidence-based approach to determine examination quality for the assessment of applied knowledge is imperative. Objective To present a written assessment question format that is simple to construct, can be scored electronically, and will accurately assess the application of knowledge in examinations during the classroom curriculum and for high-stakes examinations. Background Traditionally, free-response written assessments have been administered to assess the application of knowledge, for example, short answer or essay questions. Selected-response questions such as a carefully constructed multiple-choice questions or extended-matching questions (EMQ) are reliable and valid options to assess the depth of understanding in subject matter. Description Extended matching is a selected-response question format used to assess the application of knowledge. It consists of items consistent with a common theme and a large list of homogenous options providing distractors and a correct answer. Clinical Advantage Extended-matching questions have demonstrated good psychomotor measures and can assess the application of knowledge over a large number of themes in 1 examination. They are simple to create, minimize cueing, and are scored objectively. Conclusions Both well-written multiple-choice questions and EMQs can be used to assess the application of knowledge level of cognition. The EMQ is easier to write with less chance of cueing or guessing the correct option and is optimal for evaluation and treatment scenario type questions.

Proceedings of Contributed Presentations 10th World Federation of Chiropractic/Association of Chiropractic Colleges Education Conference

Proceedings of Contributed Presentations 10th World Federation of Chiropractic/Association of Chiropractic Colleges Education Conference

Investigating Saudi University EFL Teachers’ Assessment Literacy: Theory and Practice

Teacher assessment literacy (TAL) is believed to have positive impact on student learning outcomes. Therefore, attempts are made, especially, in advanced educational contexts to increase TAL. In the context of Saudi higher education, available empirical evidence indicates that EFL teacher assessment literacy is replete with loopholes. This mixed-method research investigated Saudi EFL teachers’ construction of assessment tasks, the influence the tasks had on students’ learning and the extent to which teachers’ assessment practices were in alignment with recommended assessment practices. The data were collected through analyzing teachers’ summative assessment tasks and a student survey with both close and open-ended questions. Apart from the participants’ responses to the open-ended questions of the survey, the data went through quantitative data analysis for frequencies and percentages. The findings revealed a serious incongruity between teachers’ assessment tasks and course learning outcomes. For instance, higher order learning outcomes were not assessed at all. Most of the tasks were selected-response questions (SRQs). As confirmed by the survey data, the assessment tasks mainly triggered memorization as a learning strategy. Therefore, suggestions are made that university teachers’ professional development with particular focus on their assessment literacy is placed at the center of higher education policies. Without valid assessment in place, the edifice of Saudi (higher) education system may lose its efficacy.

Embedding Formative Assessment and Promoting Active Learning

In this contribution we outline how the York Science project is using a ‘backward design’ approach to teaching science to students aged 11–14. We then present some examples of formative assessment tasks and show how simple selected-response questions can be modified to provide teachers with detailed information about students’ ideas. Finally we indicate how such tasks can help promote active learning.

Leadership Development in Dental Education: Report on the ADEA Leadership Institute, 2000–14

The American Dental Education Association’s Leadership Institute (ADEA LI) is the association’s flagship development program for those aspiring to leadership in dental and higher education. As with previous studies of the ADEA LI, ADEA will use information from the survey described in this report to improve the ADEA LI curriculum and to guide other leadership development efforts. In 2014-15, ADEA distributed a 50-item online survey via email to all ADEA LI alumni from the classes of 2000 through 2014. The survey included selected-response questions, closed-ended questions, and open-response questions. The survey had an overall response rate of 47% (133/285); response rates to individual items varied. The mean age of the respondents when they participated in the Institute was 48.5 years. Men and women were almost equally represented among the respondents. Nearly half reported their ultimate career goal as department chair, associate dean, or assistant dean, while 20 (15.8%) indicated a goal of becoming dean and 15 (11.8%) aspired to administrative roles higher than dean. Areas the respondents recommended for improvement included more programming in budgeting and financial management, fundraising, and personnel management. Almost 100% of the respondents indicated they would recommend the ADEA LI to others. Overall, the survey respondents confirmed the value of the ADEA LI in their assessment of their fellowship and its subsequent application to their careers. Comparison of elements from this study to previous studies of ADEA LI alumni demonstrates the effectiveness of past changes made to the Institute and the creation of additional ADEA leadership initiatives.

Pharmacy Student Performance on Constructed-Response Versus Selected-Response Calculations Questions

Objective. To introduce PharmD students to changes in calculations question types (constructed-response versus selected-response questions); measure and compare student performance on constructed-response and selected-response questions in a pharmaceutics course; and collect student feedback on the use of differing question types.Methods A pharmaceutics/pharmaceutical calculations examination was administered that included 15 pairs of questions; each pair consisted of a constructed-response question and a similar selected-response question. An online questionnaire was conducted to collect student feedback.Results. Of the 15 topics, the class scored higher on the constructed-response question for 4 topics and higher on the selected-response question for 10 topics. Eighty percent of the class preferred selected-response questions, although 47.8% felt constructed-response questions better prepared them for a career in healthcare.Conclusions. Students correctly answered more selected-response questions than constructed-response questions and felt more confident in doing so. Additional constructed-response teaching and testing methods should be incorporated into pharmacy education.

A mixed model study evaluating lean in the transformation of an Orthopaedic Radiology service

ObjectiveThe purpose of the study was to explore the proposition that lean is an effective methodology for service improvement within healthcare by using it to evaluate and implement change in a poor service. DesignA mixed model approach was used with data being collected before and after change. SettingThe orthopaedic radiology out-patient pathway in a large district general hospital was receiving increasing numbers of complaints with long waiting times and poor levels of satisfaction amongst patients and staff. ParticipantsData were collected in the form of qualitative and quantitative data taken from questionnaires completed by staff and patients, and quantitative data extracted from the Radiology Information System. A proportionate stratified random sampling method was used to collect data from the patients and a theoretical sample was used for the staff. InterventionThe pre-implementation data was collected during a lean event in which a value stream map of the pathway was created. From this information, changes were planned and implemented, before collecting the post-implementation data using the same methods. Main outcome measuresThe aims of the study were to compare pre-implementation and post-implementation data related to patient journey time, patient experience and staff and patient satisfaction. The quantitative data were analysed using Levene's test for equality of variance and a 2-sample t-test to test for comparison of the means. Descriptive statistics were used to analyse data collected from the polar and the selected response questions from the questionnaires, whilst thematic analysis was used to analyse the qualitative data. ResultsResults demonstrated that the new service offered a better quality of patient experience with higher levels of staff satisfaction whilst enabling an improvement in productivity. ConclusionsThe author proposes that these findings support the proposition that lean is an effective methodology for service improvement within a healthcare setting.

Evaluating professionalism in medical undergraduates using selected response questions: findings from an item response modelling study

BackgroundProfessionalism is a difficult construct to define in medical students but aspects of this concept may be important in predicting the risk of postgraduate misconduct. For this reason attempts are being made to evaluate medical students' professionalism. This study investigated the psychometric properties of Selected Response Questions (SRQs) relating to the theme of professional conduct and ethics comparing them with two sets of control items: those testing pure knowledge of anatomy, and; items evaluating the ability to integrate and apply knowledge ("skills"). The performance of students on the SRQs was also compared with two external measures estimating aspects of professionalism in students; peer ratings of professionalism and their Conscientiousness Index, an objective measure of behaviours at medical school.MethodsItem Response Theory (IRT) was used to analyse both question and student performance for SRQs relating to knowledge of professionalism, pure anatomy and skills. The relative difficulties, discrimination and 'guessabilities' of each theme of question were compared with each other using Analysis of Variance (ANOVA). Student performance on each topic was compared with the measures of conscientiousness and professionalism using parametric and non-parametric tests as appropriate. A post-hoc analysis of power for the IRT modelling was conducted using a Monte Carlo simulation.ResultsProfessionalism items were less difficult compared to the anatomy and skills SRQs, poorer at discriminating between candidates and more erratically answered when compared to anatomy questions. Moreover professionalism item performance was uncorrelated with the standardised Conscientiousness Index scores (rho = 0.009, p = 0.90). In contrast there were modest but significant correlations between standardised Conscientiousness Index scores and performance at anatomy items (rho = 0.20, p = 0.006) though not skills (rho = .11, p = .1). Likewise, students with high peer ratings for professionalism had superior performance on anatomy SRQs but not professionalism themed questions. A trend of borderline significance (p = .07) was observed for performance on skills SRQs and professionalism nomination status.ConclusionsSRQs related to professionalism are likely to have relatively poor psychometric properties and lack associations with other constructs associated with undergraduate professional behaviour. The findings suggest that such questions should not be included in undergraduate examinations and may raise issues with the introduction of Situational Judgement Tests into Foundation Years selection.

Student Approaches to Learning in Aviation Contexts

Self-regulated learning is an important element of student performance and has been found to be linked with content domains. Aviation courses are complex yet serve as the foundation for student success in the flight environment. Since it is critical that students master the content, it is important to determine how students approach learning in these courses. Participants in this study completed a survey consisting of course-related selected-response questions, and open-ended questions focusing on their study habits. Results of the study portray an interesting insight into the learning strategies used by collegiate aviation students. Analyses appear to indicate that learning techniques may need to be improved to promote more successful learning in these types of courses. Improving Learning in Aviation Contexts Students entering collegiate aviation programs with the dream of becoming an airline pilot face unique educational challenges. Learning to operate sophisticated modern aircraft equipped with advanced technologies in the flight environment places intense academic requirements on students. The thrill of flight alone does not necessarily carry students through difficult scientific and technical content. Traditional aviation curricula are comprised ofboth classroom and flight components. Before students can perform effectively in the flight environment, it is imperative that they have a thorough understanding of the various aspects of flight. In general, the classroom component is designed to provide students with the principles underlying the application oftechnicalknowledge as well as information regarding meteorology, physics, governmental regulations, air traffic control and operations within the national airspace. Because of the depth and complexity of the subject matter, students need to use learning and comprehension monitoring strategies that will enable them to become cognitively engaged. They need to invest effort to make connections, elaborate, translate, organize and reorganize in order to think and process deeply. For many, the subject matter covered is unfamiliar, and unlike any topics they may have encountered during their high-school years. The classroom component, however, plays a critical role in providing the student with a strong foundation of knowledge. To be effective, aviation academic programs must ensure that the educational process involves an in-depth, effective transfer of knowledge across a broad spectrum of aviation subjects (Karp, Turney, Green, Sitler, Bishop, & Niemczyk, 2002). As in most collegiate classrooms, aviation classrooms consist of a variety of learners some struggling, some strategic and some exhibiting characteristics of either fiom time to time. Students that struggle have difficulty learning and remembering; much that they encounter is perplexing and fiustmtiug. If they do not perform successfully on a task, they may experience feelings of defeat, discouragement, and even apathy. On the other hand, students that are strategic seem to learn rapidly and with apparent ease. They approach instructional tasks with a high degree of confidence that they can accomplish the task. They understand that learning is an active process and they must take responsibility for doing it. Strategic learners are actively engaged with the material, and have some awareness of when they are learning it, and maybe more importantly, when they are not. They look at learning as a process they control (Eggen & Kauchak, 1999; Weinstein & Hume, 1998). In general, most educational activities are teacherdirected with students attempting to cany out the instructional activity using the learning strategies they know. Unfortunately, many students have a limited set of Page 19 JAAER, Fall2008 1 Niemczyk: Student Approaches to Learning in Aviation Contexts Published by ERAU Scholarly Commons, 2008