Pool Of Questions Research Articles

A maximum likelihood based offline estimation of student capabilities and question difficulties with guessing

In recent years, the computerised adaptive test (CAT) has gained popularity over conventional exams in evaluating student capabilities with desired accuracy. However, the key limitation of CAT is that it requires a large pool of pre-calibrated questions. In the absence of such a pre-calibrated question bank, offline exams with uncalibrated questions have to be conducted. Many important large exams are offline, for example the Graduated Aptitude Test in Engineering (GATE) and Japanese University Entrance Examination (JUEE). In offline exams, marks are used as the indicator of the students’ capabilities. In this work, our key contribution is to question whether marks obtained are indeed a good measure of students’ capabilities. To this end, we propose an evaluation methodology that mimics the evaluation process of CAT. In our approach, based on the marks scored by students in various questions, we iteratively estimate question parameters such as difficulty, discrimination and the guessing factor as well as student parameters such as capability using the 3-parameter logistic ogive model. Our algorithm uses alternating maximisation to maximise the log likelihood estimate for the questions and students’ parameters given the marks. We compare our approach with marks-based evaluation using simulations. The simulation results show that our approach out performs marks-based evaluation.

ASSESSMENT DESIGN THROUGH AN EXPERT SYSTEM AND ITS APPLICATION TO A COURSE OF HYDRAULICS

An assessment consists of questions addressing the required learning outcomes of a course. If a pool of questions of various types is made available then assessment design reduces to selection of questions, one by one, from the pool. Since the number of possible questions for a course may be quite large, and several preferences have to be matched, manual selection of a suitable question is not possible. This paper presents an enhanced implementation of a previously presented idea of a methodology for assessment design with an application to a course of Hydraulics with an initial pool of 1,000 questions. Each question is tagged with a set of attributes. The rules are generated by the expert system itself. The idea of a score of relevance has been introduced. The enhanced implementation displays a set of questions with their relevance scores rather than a single question to let the instructor choose from them. An instance of MS SQL Server at Azure database is used for the web-based cloud implementation.

Development study on ISKEBE Violence against Women Attitude Scale (ISKEBE Attitude Scale)

Objective: To develop an attitude scale in order to determine the attitudes of individuals towards violence against women. Methods: A question pool of 58 items was created as a result of the literature reviews, interviews with individuals and expert advice. The trial form prepared was applied to 949 people between the age of 15 and 65, who are at least primary school graduates. Results: The Kaiser-Meyer-Olkin value of the scale was found as 0.96, while the result of Bartlett’s test was found to be 21972.02. As a result of the factor analysis, a scale consisting of two factors and 30 items was obtained. In this two-factor scale, it was determined that the first factor explains 38.2% of the total variance, the second factor explains 9.6% of the total variance, while these two factors together explain 47.9% of the total variance. In the determination of the internal validity of the scale, the upper and lower groups of 27% were assessed with the ‘t-test in independent groups’, and it was concluded that the difference was significant. The Cronbach’s alpha and the test-re-test reliability method were used for the reliability of the scale, the Cronbach’s alpha value was determined as 0.86, and the test-retest correlation coefficient was determined as 0.81. Conclusion: The validity and reliability analyses performed showed that the scale can be applied to female and male individuals between the age of 15 and 65, who are at least primary school graduates.

Evaluating the Impact of Releasing an Item Pool on a Test's Empirical Characteristics

Protecting the security of examination questions is an important task for high-stakes examining boards/agencies and university programs. To maintain the security of questions, examining boards and university programs use a combination of prevention, detection, and enforcement strategies. A common prevention strategy is to establish a number of controls on access to questions; however, restricting access can motivate examinees to try harder to reconstruct questions that may appear on future versions of the test. Moreover, access to study materials by some groups and not others can present a challenge to the fairness of examinations. The aim of this study was to evaluate the empirical stability of test characteristics. Specifically, the primary research objective was to investigate the empirical stability of the items and test forms of a written examination before and after a specific policy decision was implemented. As a response to both of these concerns, this article describes a study that evaluated how psychometric (i.e., statistical) properties of test forms and individual questions might be affected by publicly releasing a larger number of questions from an item question pool. A series of analyses were conducted, including item drift to evaluate stability of the characteristics. The results suggest that empirical characteristics of the test forms and individual questions have remained relatively stable since the release policy was implemented. Specifically, statistical properties of the test forms have continued to perform similarly to test forms that were constructed prior to the release. Although the results of this study were promising, the context of this specific testing program may have offered additional protections such as a limited number of administrations that others may not. Therefore, testing/examining agencies and university programs may want to consider this strategy with appropriate caution.

Methodological Challenges in Developing a Youth Questionnaire, Life & Health Young People, for Comparative Studies in Thailand and Sweden: About Bridging the Language Gap Between Two Non-English-Speaking Countries.

To develop a Thai questionnaire ชีวิตและสุขภาพของวัยรุ่นในประเทศไทย (TYQ) to explore girls' and boys' living conditions, lifestyles, and self-reported health with special focus on sexuality, based on a Swedish questionnaire, Liv & Hälsa ung (SYQ). Challenges in developing a youth questionnaire for comparative studies are described. A multistep translation, sociocultural adaptation procedure, and a mixed-method validation test were performed using English as a common language within the research group. Three versions of SYQ were used as a pool of questions to develop the questionnaire. From a field test, unclear questions were identified and minor adjustments made. Life & Health Young People in a Thai version was successfully developed. The English version was used to bridge the language gap. This unique multistep methodology, including mixed-method validation procedure, can be used by researchers in countries where English is not the main language.

Towards Tailored Patient's Management Approach: Integrating the Modified 2010 ACR Criteria for Fibromyalgia in Multidimensional Patient Reported Outcome Measures Questionnaire.

Objectives. To assess the validity, reliability, and responsiveness to change of a patient self-reported questionnaire combining the Widespread Pain Index and the Symptom Severity Score as well as construct outcome measures and comorbidities assessment in fibromyalgia patients. Methods. The PROMs-FM was conceptualized based on frameworks used by the WHO Quality of Life tool and the PROMIS. Initially, cognitive interviews were conducted to identify item pool of questions. Item selection and reduction were achieved based on patients as well as an interdisciplinary group of specialists. Rasch and internal consistency reliability analyses were implemented. The questionnaire included the modified ACR criteria main items (Symptom Severity Score and Widespread Pain Index), in addition to assessment of functional disability, quality of life (QoL), review of the systems, and comorbidities. Every patient completed HAQ and EQ-5D questionnaires. Results. A total of 146 fibromyalgia patients completed the questionnaire. The PROMs-FM questionnaire was reliable as demonstrated by a high standardized alpha (0.886–0.982). Content construct assessment of the functional disability and QoL revealed significant correlation (p < 0.01) with both HAQ and EQ-5D. Changes in functional disability and QoL showed significant (p < 0.01) variation with diseases activity status in response to therapy. There was higher prevalence of autonomic symptoms, CVS risk, sexual dysfunction, and falling. Conclusions. The developed PROMs-FM questionnaire is a reliable and valid instrument for assessment of fibromyalgia patients. A phased treatment regimen depending on the severity of FMS as well as preferences and comorbidities of the patient is the best approach to tailored patient management.

Generating large question banks of graded questions with tailored feedback and its effect on student performance

Formative assessments have been developed to give students practice with simple mathematical manipulations necessary in physical chemistry. A series of computer programs has been used to generate large pools of questions to accompany a core undergraduate physical chemistry module. In each assessment the questions are graded to increase the cognitive load gradually and feedback is tailored to each individual response. Furthermore, the assessments are arranged in a “daisy chain” to ensure that one is completed before the next may be attempted. The difference in terminal examination results between cohorts prior and post intervention is presented. There appears to have been a positive effect on the examination performance of the post intervention students.

Challenges & Perspectives of Immunotherapy Biomarkers & The HistoOncoImmune™ Methodology

ABSTRACTThe confluence of the immunology and oncology tributaries has precipitated a biomarker turbulence that is novel to the cancer community. On the heels of reversing the dismal outcomes for numerous patients suffering from terminal disease, it has also left behind deep pools of biomarker questions – which ones, when to use them, how to use them and what cells express them – to name just a few. The tumor microenvironment, an arena potentially housing a complex mix of immune infiltrators in the midst of tumor cells, is a wildly heterogeneous terrain, one that severely challenges our single biomarker model. This article examines the recently FDA-approved biomarker assays for PD-1 therapeutics, considers the technical shortcomings of the single biomarker model, scans alternate biomarker technologies and proposes a model that may allow investigators a method to systematically address these complexities.

Investigating an alternate form of the cognitive reflection test

AbstractMuch research in cognitive psychology has focused on the tendency to conserve limited cognitive resources. The CRT is the predominant measure of such miserly information processing, and also predicts a number of frequently studied decision-making traits (such as belief bias and need for cognition). However, many subjects from common subject populations have already been exposed to the questions, which might add considerable noise to data. Moreover, the CRT has been shown to be confounded with numeracy. To increase the pool of available questions and to try to address numeracy confounds, we developed and tested the CRT-2. CRT-2 questions appear to rely less on numeracy than the original CRT but appear to measure closely related constructs in other respects. Crucially, substantially fewer subjects from Amazon’s Mechanical Turk have been previously exposed to CRT-2 questions. Though our primary purpose was investigating the CRT-2, we also found that belief bias questions appear suitable as an additional source of new items. Implications and remaining measurement challenges are discussed.

How do they view us? Oncology providers’ perceptions of early/concurrent palliative care.

114 Background: Early integration of palliative care has been increasingly recognized as an important component in the care of cancer patients, including during active anticancer treatment. Yet, palliative and supportive care interventions remain to be well integrated into standard oncology practice. We sought to develop an original survey instrument in order to describe oncology providers’ perceptions of palliative care, particularly when introduced in the early and concurrent setting. Methods: We conducted a systematic review of literature pertaining to perceptions of palliative care and barriers to referral, and extracted survey items where they were found. This pool of questions was narrowed to focus on early/concurrent palliative care, and supplemented with original items. The draft survey was then systematically validated using a standardized scoring system and content-validity approach. Results: Twenty-two studies were reviewed. From an initial pool of fifty items, twenty-nine were selected for the draft survey. The draft was then reviewed by ten oncology providers, who scored each item and provided comments. Conclusions: We have produced a fully validated survey instrument that will be used to characterize oncology providers’ perceptions of early and concurrent palliative care interventions. The validated survey will be administered to oncology clinicians (including nurses, social workers and chaplains) at four academic medical centers at which an early palliative care intervention is currently being implemented. We hope to assess perceptions before and after implementation of the early palliative care intervention.

Development of a Set of Rehabilitation Related Multiple-choice-questions in Medical Education

In the rehabilitation related teaching as in other subjects of the medical training multiple choice (MC) examinations are the most frequent type of examinations. Compared to other subjects only a few MC questions are available for the interdisciplinary subject Rehabilitation. Therefore an internet-based online platform "Pool of rehabilitation related MC questions" was developed to assist teachers regarding the provision, design and organization of high-quality rehabilitation related MC questions. A total of 502 existing MC questions were collected from 12 German Medical Faculties. After removal of 59 questions not suitable for formal and content reasons a total of 443 questions were presented to 6 reviewers for triple reviews (a total of 1 329 expert reviews received). Of the 502 questions 335 (67%) were included in the final pool including short cases with 46 case studies. The questions refer to the following learning objectives: principles of rehabilitation (40%), rehabilitative interventions (20%), diagnosis and assessment (18%), initiation and control of the rehabilitation process (12%) and methods/quality of rehabilitative interventions (10%). The use of the online platform modules resp. the questions are for free for lecturers. This includes the compilation and output of complete examinations, the statistical evaluation, and other audit-related materials. This examination pool counteracts the current lack of quality-assured rehabilitation-related MC questions and contributes to set common standards for the Medical Faculties to rehabilitation related examinations.

MO‐DE‐BRA‐01: Enhancing Radiation Physics Instruction Through Gamification and E‐Learning

Purpose:This project sought to “gamify” the instruction of radiation interaction physics concepts for technology students. Gamification applies game mechanics and user interactions in active learning contexts. In one part of this project, a self‐guided eModule was developed for conceptual radiation interaction instruction. In a second part, a web‐based game, Particle Launch (http://particle‐launcher.ist.unomaha.edu), was created to challenge students to quickly apply radiation interaction concepts in a way that is stimulating and motivating.Methods:The eModule, focused on conceptual interaction physics, was designed in Adobe Captivate and incorporates animation, web videos, and assessment questions in order to generate student interest. Navigating the whole module takes 40 minutes for beginners. Assessments after three main sections are comprised of 3–4 questions randomly selected from a question pool. In collaboration with the University of Nebraska at Omaha's College of Information Science and Technology, the Particle Launch game was created with the Unity gaming engine and designed with a game‐play look and feel. The object of the game is to utilize different particles, energies, and directions to destroy a target given a limited number of resources and time to complete the task. A rewards system encourages accurate shots.Results:The eModule part of the project encourages a flipped classroom model in which class time is devoted to application of concepts rather than information‐based lectures. Currently, eModule assessments are not tracked but this feature could be incorporated to encourage participation. Furthermore, in a class of five technology students, the game was found to be fun and engaging and had the effect of reinforcing basic concepts from the eModule.Conclusion:Gamification has significant potential to alter medical physics instruction. Game‐play feedback is an important part of the learning process. Students found Particle Launch inviting and challenging and further research could help game design.This project was generously supported by the Office of the Vice‐Chancellor for Academic Affairs and the University of Nebraska Medical Center.

Development and Validation of a Safety Climate Scale for Manufacturing Industry.

BackgroundThis paper describes the development of a scale for measuring safety climate.MethodsThis study was conducted in six manufacturing companies in Iran. The scale developed through conducting a literature review about the safety climate and constructing a question pool. The number of items was reduced to 71 after performing a screening process.ResultsThe result of content validity analysis showed that 59 items had excellent item content validity index (≥ 0.78) and content validity ratio (> 0.38). The exploratory factor analysis resulted in eight safety climate dimensions. The reliability value for the final 45-item scale was 0.96. The result of confirmatory factor analysis showed that the safety climate model is satisfactory.ConclusionThis study produced a valid and reliable scale for measuring safety climate in manufacturing companies.

Talent quotient for Indian management teachers

The objective of the paper is to develop and validate a measurement scale for defining talent for management teachers in India in terms of a quotient. Structured questionnaire was prepared for the teachers teaching in Indian management colleges/management department/business schools having more than ten years of existence. A pretesting is carried out with 15 experts of the high academic experience to identify parameters to be used in a pool of questions and then confirmatory factor analysis technique has been adopted to draw scale composite reliability and average variance extracted and later discriminate validity is checked thorough factor inter-correlation matrix. A six-attribute scale is developed and validated which will be utilised in determining teacher's talent quotient in Indian business schools.

A comparative analysis of the progress of e-learning education of nursing students in the field of Reliability in scientific research. E-learning education

Aim. A comparative analysis of two methods of organising e-courses designed for students of nursing and conducted throughout 2012/13 and 2013/14 at the Medical University of Warsaw (MUW) within the framework of education in “Reliability in scientific research”. Material and Methodology. A group of 285 students of the 1st degree course at the Nursing Faculty, including 141 subjects from the academic year 2012/13 and 144 from 2013/14. Data from reliability reports were analysed comparatively as well as the scored results that represented the assessment results for the solved tasks prepared in the form of Short Answer Questions (SAQs) or tests generated automatically (Multiple-Choice Questions, MCQs). Results. Students who passed the e-course in SAQ format, spent a considerable longer amount of time on the MoodleTM platform (229.1 ± 122.23 minutes, academic year of 2012/13) than the students who passed the e-course taking MCQs (175.3 ± 93.38 minutes, academic year 2013/14) (U Mann-Whitney test = 7298.0, P = 0.0002). Moreover, the activities of students in consecutive months of the academic year 2012/13 were similar to the activities of the academic year 2013/14. Reliability evaluation of the pool of test questions points to a high degree of internal consistency of the exam set (α-Cronbach = 0.93, standard error = 0.12). Conclusion. Applying an MCQ e-test ensured a reliable evaluation of the students’ progress. The final MCQ type of the test with the set time limit increases the reliability of educational measurement. Regardless of the applied system of organising e-courses, lack of systematic work on part of the students presented the greatest problem throughout the whole term.

Institutions collaborating on an information literacy assessment tool

Purpose – The purpose of this paper is to create an information literacy (IL) instruction assessment tool that responds to the unique needs of individual institutions and provides a strategic and relevant model for assessing IL skills among undergraduate students. Design/methodology/approach – The research team designed a post-test questionnaire comprised of two demographic questions, two open-ended questions and a pool of skill-based multiple-choice questions mapped to Association of College and Research Libraries Information Literacy (ACRL IL) Competency Standards for Higher Education. Participating librarians used a customized questionnaire to assess student learning at the end of their one-shot instruction sessions. Findings – In their responses to the multiple-choice questions, students demonstrated a clear understanding of ethical use of information and a strong ability to select appropriate tools for accessing information sources. Student responses to the open-ended questions revealed a wide range of confidence and ability levels, and provided insight into the frequency, depth and breadth with which various ACRL Standards are being addressed in library sessions. Research limitations/implications – This paper reports on student responses to questions that have subsequently been identified as problematic; therefore, strong inferences cannot be made about student learning from these responses. Questions have since been improved with further revision. In addition, the sample sizes for individual questions were too small to be generalizable. Practical implications – The intentional and strategic approach to the development of the assessment tool and its implementation is that it be practical and easy to implement for partner libraries. It is intended to make assessment of IL in the undergraduate context be assessable to all academic librarians who desire to participate. Originality/value – This paper describes a unique assessment tool that is designed to be responsive to local needs and provide a cost-free assessment option for academic libraries.

Online assessments in pharmaceutical calculations for enhancing feedback and practice opportunities

ObjectivesThe objectives of the study were to evaluate the effectiveness of, and student preference for, an online approach to carrying out the formative assessments of a first-year pharmaceutical calculations course. Traditionally, these assessments were completed as in-class paper–pencil Scantron® quizzes. After the change, they were completed online as homework assignments with detailed feedback and ample opportunities for repeated practice once the assignments were graded. MethodsClasses of 2012, 2013, and 2014 students in the Doctor of Pharmacy program at Texas A&M Rangel College of Pharmacy completed these formative assessments online using WebCT or Blackboard. Individualized question sets were randomly constructed from a question pool containing multiple-choice questions and presented as online homework. The effectiveness of this approach was measured by comparing the overall course grades of these classes with the Classes of 2010 and 2011, who completed the paper–pencil Scantron® quizzes. Student preference for the online approach was assessed by a survey. ResultsThe course grades of Classes of 2012 and 2013 were significantly greater than those of Class of 2010. The differences in grades between the other classes were not significant. More than 90% of students favored the online approach and provided valuable comments about the benefits of detailed feedback and repeated practice. ConclusionsStudents preferred the online homework assignments. The online approach maintained or improved the student grade. Once the question pool is developed, instructors can provide detailed feedback to students very efficiently. Formative assessments for pharmaceutical calculations can be done online more conveniently than the traditional paper–pencil approach.

STEP: A Scalable Testing and Evaluation Platform

The emergence of online crowdsourcing sites, online work platforms, and evenMassive Open Online Courses (MOOCs), has created an increasing need for reliably evaluating the skills of the participating users in a scalable way.Many platforms already allow users to take online tests and verify their skills, but the existing approaches face many problems. First of all, cheating is very common in online testing without supervision, as the test questions often "leak" and become easily available online together with the answers.Second, technical skills, such as programming, require the tests to be frequently updated in order to reflect the current state-of-the-art. Third,there is very limited evaluation of the tests themselves, and how effectively they measure the skill that the users are tested for. In this paper, we present a Scalable Testing and Evaluation Platform (STEP),that allows continuous generation and evaluation of test questions. STEP leverages already available content, on Question Answering sites such as StackOverflow and re-purposes these questions to generate tests. The system utilizes a crowdsourcing component for the editing of the questions, while it uses automated techniques for identifying promising QA threads that can be successfully re-purposed for testing. This continuous question generation decreases the impact of cheating and also creates questions that are closer to the real problems that the skill holder is expected to solve in real life.STEP also leverages the use of Item Response Theory to evaluate the quality of the questions. We also use external signals about the quality of the workers.These identify the questions that have the strongest predictive ability in distinguishing workers that have the potential to succeed in the online job marketplaces. Existing approaches contrast in using only internal consistency metrics to evaluate the questions. Finally, our system employs an automatic "leakage detector" that queries the Internet to identify leaked versions of our questions. We then mark these questions as "practice only," effectively removing them from the pool of questions used for evaluation. Our experimental evaluation shows that our system generates questions of comparable or higher quality compared to existing tests, with a cost of approximately 3-5 dollars per question, which is lower than the cost of licensing questions from existing test banks.

Development of a food safety attitude and practice questionnaire for Iranian consumers

AbstractTo develop and to offer a promising tool to measure food safety attitude and practice for Iranian consumers, a pool of questions on four major areas of food handling was extracted from the literature by a team of experts in food, nutrition and epidemiology, and sent to 30 experts to be approved for content validity. The study was conducted in the capital city of Tehran. Two hundred and thirty‐three participants, 18 years and older, completed the 56‐item questionnaire. Principle component factor analysis (PCFA) plus a Varimax rotation method was used for exploration of construct validity. Cronbach's alpha (α) and intra‐class correlation coefficient were used to demonstrate questionnaire reliability. PCFA revealed that three and four main components existed in attitude and practice questionnaires, respectively, explaining 60.02% and 59.6% of total variations. α was 0.681, 0.632 and 0.308 for the components of attitude and 0.689; 0.674, 0.647 and 0.167 for the components of practice questionnaires respectively. The questionnaire proved to be a useful tool in the assessment of food safety attitudes and practices among Iranian consumers.

THU0037 Validation of A Disease-Specific Patient-Reported Outcome Measure for Arabic Patients with Systemic Lupus Erythematosus

BackgroundIn comparison to the common chronic diseases, SLE has poorer outcomes with negative impact on the patient's life. Therefore, it became mandatory not only to recognize disease activity core set...