Mastery Assessment Research Articles

Evaluation of inhaler technique and achievement and maintenance of mastery of budesonide/formoterol Spiromax\xae compared with budesonide/formoterol Turbuhaler\xae in adult patients with asthma: the Easy Low Instruction Over Time (ELIOT) study

BackgroundIncorrect inhaler technique is a common cause of poor asthma control. This two-phase pragmatic study evaluated inhaler technique mastery and maintenance of mastery with DuoResp® (budesonide-formoterol [BF]) Spiromax® compared with Symbicort® (BF) Turbuhaler® in patients with asthma who were receiving inhaled corticosteroids/long-acting β2-agonists.MethodsIn the initial cross-sectional phase, patients were randomized to a 6-step training protocol with empty Spiromax and Turbuhaler devices. Patients initially demonstrating ≥1 error with their current device, and then achieving mastery with both Spiromax and Turbuhaler (absence of healthcare professional [HCP]-observed errors), were eligible for the longitudinal phase. In the longitudinal phase, patients were randomized to BF Spiromax or BF Turbuhaler. Co-primary endpoints were the proportions of patients achieving device mastery after three training steps and maintaining device mastery (defined as the absence of HCP-observed errors after 12 weeks of use). Secondary endpoints included device preference, handling error frequency, asthma control, and safety. Exploratory endpoints included assessment of device mastery by an independent external expert reviewing video recordings of a subset of patients.ResultsFour hundred ninety-three patients participated in the cross-sectional phase, and 395 patients in the longitudinal phase. In the cross-sectional phase, more patients achieved device mastery after three training steps with Spiromax (94%) versus Turbuhaler (87%) (odds ratio [OR] 3.77 [95% confidence interval (CI) 2.05–6.95], p < 0.001). Longitudinal phase data indicated that the odds of maintaining inhaler mastery at 12 weeks were not statistically significantly different (OR 1.26 [95% CI 0.80–1.98], p = 0.316). Asthma control improved in both groups with no significant difference between groups (OR 0.11 [95% CI -0.09–0.30]). An exploratory analysis indicated that the odds of maintaining independent expert-verified device mastery were significantly higher for patients using Spiromax versus Turbuhaler (OR 2.11 [95% CI 1.25–3.54]).ConclusionsIn the cross-sectional phase, a significantly greater proportion of patients using Spiromax versus Turbuhaler achieved device mastery; in the longitudinal phase, the proportion of patients maintaining device mastery with Spiromax versus Turbuhaler was similar. An exploratory independent expert-verified analysis found Spiromax was associated with higher levels of device mastery after 12 weeks. Asthma control was improved by treatment with both BF Spiromax and BF Turbuhaler.Trial registrationEudraCT 2013-004630-14 (registration date 23 January 2014); NCT02570425.

Alternate Assessment Formats for Progress Monitoring Students With Intellectual Disabilities and Below Average IQ: An Exploratory Study

As part of standards-based reforms, there is increasing emphasis on ensuring that students with moderate intellectual disabilities (ID), including students with Autism Spectrum Disorders (ASD), learn to read. There is also converging evidence that explicit teaching of letter sounds, phonics, and sight words is effective for this population, but that students’ responsiveness varies. A critical part of individualizing reading instruction for students with disabilities is the reliable assessment of progress and mastery of reading skills. However, assessment of many students with ID and students with ASD is challenging because of attention, behavioral, and communication issues related to testing situation; therefore, obtaining consistent results often proves to be a difficult task. We hypothesized that alternate assessment presentation formats, as a testing accommodation, would improve the reliability, validity, and consistency of assessment performance. In this study, three different presentation formats—word lists, flash cards, and PowerPoint presentation—were used when administering proximal, curriculum-based reading assessments to determine whether a particular format increased student engagement, reduced the need for prompts, and increased accuracy of identifying known items on the test. While statistical analyses did not support the hypothesis of a format by student effect, visual analysis of the data did suggest that the number of prompts required varied by student as a function of assessment format. Most noteworthy, assessment reliability, estimated with generalizability theory, indicated that reliability increased as a function of format by student.

Because Life Is Open Book: An Open Internet Family Medicine Clerkship Exam.

A core principle of family medicine is information mastery, or application of principles of evidence-based medicine in clinical practice. While information mastery teaching and assessment are beginning to permeate postgraduate family medicine training programs, and while exciting literature on new open resource assessment methods is emerging, there are no prior descriptions of examinations that specifically assess medical students' information mastery competency. To test information mastery competency, a novel final exam for the family medicine clerkship was developed, implemented, and evaluated. During the timed exam, the competency-based information mastery assessment (IMA) requires students to look up evidence-based information using web resources to answer case-based questions. Exam feasibility was tested with pilot examinees whose reactions were gauged. Student performance on the traditional closed book knowledge assessment (KA) was compared with performance on the open internet IMA. Exam performance was compared with preceptor ratings of students' clinical performance. Low performers were further analyzed for preceptors' ratings of specific student skills in information mastery and self-directed learning. An open internet IMA testing knowledge application and information mastery skills is not only feasible but can also be educational. Student performance scores on the open internet IMA do not differ from scores on the closed book KA. Students describe many positive features of this open internet IMA. Student performance on the competency-based IMA correlates with clinical ratings by preceptors and with preceptors' judgment of information mastery skills. A novel approach to assessment in family medicine clerkships may be used to assess student competency in information mastery. Further research is needed for enhanced exam validation.

Predicting Individual Differences in Reading and Spelling Skill With Artificial Script–Based Letter–Speech Sound Training

In this study, we examined the learning of letter–speech sound correspondences within an artificial script and performed an experimental analysis of letter–speech sound learning among dyslexic and normal readers vis-à-vis phonological awareness, rapid automatized naming, reading, and spelling. Participants were provided with 20 min of training aimed at learning eight new basic letter–speech sound correspondences, followed by a short assessment of mastery of the correspondences and word-reading ability in this unfamiliar script. Our results demonstrated that brief training is moderately successful in differentiating dyslexic readers from normal readers in their ability to learn letter–speech sound correspondences. The normal readers outperformed the dyslexic readers for accuracy and speed on a letter–speech sound matching task, as well as on a word-reading task containing familiar words written in the artificial orthography. Importantly, the new artificial script-related measures were related to phonological awareness and rapid automatized naming and made a unique contribution in predicting individual differences in reading and spelling ability. Our results are consistent with the view that a fundamental letter–speech sound learning deficit is a key factor in dyslexia.

Metacognitive mastery in persons with first-episode psychosis and their caregivers: Implications for timely help-seeking and caregiver experiences. Brief report

While metacognitive mastery seems to be closely linked to general functioning in persons with psychosis, little is known of metacognitive capacity of family members, who often play an important role in recovery and often report high levels of distress. We gathered assessments of metacognitive mastery from persons with first-episode psychosis and their mothers. We found that relatively higher overall mean mastery levels among the dyads, was related to less negative caregiver experiences, and less caregiver criticism. Greater levels of mastery were related to decreased duration of untreated psychosis. Clinical implications in terms of family intervention programmes repertoire are discussed.

A Reliable and Valid Objective Structured Assessment of Technical Skill for the Application of a Pavlik Harness Based on International Expert Consensus.

The use of competency-based training is increasing in medical education as it offers individualized learning opportunities to master required skills. Inherent to this method of teaching is the need for standardized and objective assessments of skill mastery. In orthopaedic surgery, educational programs have focused on surgical skills with lesser emphasis on nonoperative techniques. Accordingly, formal evaluation tools specific to Pavlik Harness application do not exist, despite its widespread use and potential complications from inappropriate application. This study sought to develop a reliable and valid evaluation tool based on international expert consensus to standardize and evaluate Pavlik Harness application for developmental dysplasia of the hip. Consensus was sought from 10 content experts using Delphi methodology. Resulting items formed the Objective Structured Assessment of Technical Skill (OSATS). Thirty-five participants were selected into 3 a priori groups (expert, intermediate and novice) based on Pavlik Harness experience. On 2 occasions, 3 content experts assessed randomized and deidentified videotapes of each participant applying a Pavlik Harness to an infant model using the OSATS and global rating scales (GRS). The reliability and validity of the OSATS was determined with intraclass (ICC) and Pearson correlations and analysis of variance (ANOVA). Consensus was obtained after 2 rounds of structured surveying and resulted in a 25-item OSATS. The reliability of the OSATS was excellent with an ICC of 0.96 for interrater and 0.98 for test-retest reliability. Construct validity was excellent with high correlations between OSATS and GRS (>0.90). In addition, the OSATS discriminated between expert, intermediate, and novice users. We have developed a competency-based evaluation tool for Pavlik Harness application based on consensus from international experts. The OSATS has been shown to be a reliable and valid method for assessing Pavlik Harness application that can discriminate between expert, intermediate, and novice users. Level II.

A Study of Home Emergent Literacy Experiences of Young Latino English Learners

This empirical study explored the home environment literacy practices of young Latino English learners and their families. The participants were 217 incoming Kindergarten Latino EL students and parents. The data collection included a completed HLEQ by the parents. In addition, children were administered the PPVT, the preLAS, the PALS-K screening, the Woodcock Reading Mastery assessment, and the Wide Range Achievement test. All of the literacy assessments given to the children provided the researchers with comprehensive look at their literacy knowledge base. The results of this study indicate that there were two significant paths for students’ achievement: availability of books and child initiated literacy factors that were directly related to the phonological processing efforts of students.

Predicting responsiveness to intervention in dyslexia using dynamic assessment

In the current study we examined the value of a dynamic test for predicting responsiveness to reading intervention for children diagnosed with dyslexia. The test consisted of a 20-minute training aimed at learning eight basic letter–speech sound correspondences within an artificial orthography, followed by a short assessment of both mastery of these correspondences and word reading ability in this unfamiliar script. Fifty-five (7- to 11-year-old) children diagnosed with dyslexia engaged in specialized intervention during approximately 10months and their reading and spelling abilities were assessed before and after. Our results indicated that the dynamic test predicted variance in reading skills at posttest, over and above traditional static measures, such as phonological awareness and rapid naming. These findings indicate that responsiveness to learning new letter–speech sound correspondences has a prognostic value for the success of specialized reading intervention.

Making the case for mastery learning assessments: key issues in validation and justification.

Theoretical and empirical support is increasing for mastery learning, in which learners must demonstrate a minimum level of proficiency before completing a given educational unit. Mastery learning approaches aim for uniform achievement of key objectives by allowing learning time to vary and as such are a course-level analogue to broader competency-based curricular strategies. Sound assessment is the cornerstone of mastery learning systems, yet the nature of assessment validity and justification for mastery learning differs in important ways from standard assessment models. Specific validity issues include (1) the need for careful definition of what is meant by "mastery" in terms of learners' achievement or readiness to proceed, the expected retention of mastery over time, and the completeness of content mastery required in a particular unit; (2) validity threats associated with increased retesting; (3) the need for reliability estimates that account for the specific measurement error at the mastery versus nonmastery cut score; and (4) changes in item- and test-level score variance over retesting, which complicate the analysis of evidence related to reliability, internal structure, and relationships to other variables. The positive and negative consequences for learners, educational systems, and patients resulting from the use of mastery learning assessments must be explored to determine whether a given mastery assessment and pass/fail cut score are valid and justified. In this article, the authors outline key considerations for the validation and justification of mastery learning assessments, with the goal of supporting insightful research and sound practice as the mastery model becomes more widespread.

Development and Assessment of a Horizontally Integrated Biological Sciences Course Sequence for Pharmacy Education

To design and assess a horizontally integrated biological sciences course sequence and to determine its effectiveness in imparting the foundational science knowledge necessary to successfully progress through the pharmacy school curriculum and produce competent pharmacy school graduates. A 2-semester course sequence integrated principles from several basic science disciplines: biochemistry, molecular biology, cellular biology, anatomy, physiology, and pathophysiology. Each is a 5-credit course taught 5 days per week, with 50-minute class periods. Achievement of outcomes was determined with course examinations, student lecture, and an annual skills mastery assessment. The North American Pharmacist Licensure Examination (NAPLEX) results were used as an indicator of competency to practice pharmacy. Students achieved course objectives and program level outcomes. The biological sciences integrated course sequence was successful in providing students with foundational basic science knowledge required to progress through the pharmacy program and to pass the NAPLEX. The percentage of the school's students who passed the NAPLEX was not statistically different from the national percentage.

Gamification in assessment: Do points affect test performance?

Gamification, applying game mechanics to nongame contexts, has recently become a hot topic across a wide range of industries, and has been presented as a potential disruptive force in education. It is based on the premise that it can promote motivation and engagement and thus contribute to the learning process. However, research examining this assumption is scarce. In a set of studies we examined the effects of points, a basic element of gamification, on performance in a computerized assessment of mastery and fluency of basic mathematics concepts. The first study, with adult participants, found no effect of the point manipulation on accuracy of responses, although the speed of responses increased. In a second study, with 6–8 grade middle school participants, we found the same results for the two aspects of performance. In addition, middle school participants' reactions to the test revealed higher likeability ratings for the test under the points condition, but only in the first of the two sessions, and perceived effort during the test was higher in the points condition, but only for eighth grade students.

Design and Assessment of a Skills-Based Geoscience Curriculum

ABSTRACT There is increasing interest in science curricula that engage students in scientific practices to deepen their understanding of the core ideas of the field. There are, however, few examples of their designs or assessments available for use as guides for understanding the strengths and weaknesses of particular designs. We offer a summary of our experience in designing and assessing a skills-based geoscience curriculum contextualized within a recently proposed, generalized framework for curricular designs and assessment. Our novel approach to curriculum assessment, with its emphasis on ensuring that the skills highlighted in different courses align with the objectives of the curriculum, is meant to complement more-direct assessments of student mastery of various skills. Assessment results reveal good alignment of course and curricular objectives, in some cases, whereas other parts of the curriculum require refinement. The most challenging of which is the failure of the introductory-level courses to emphasize the process of science. That result is consistent with recent work demonstrating that current approaches to teaching the nature of science are often ineffective. Thus, some of the most challenging problems with more traditional curricula, such as how to help students develop a more-advanced understanding of the nature of science, are not necessarily solved by focusing curricula on the integration of skills with content.

Grammatical rhymes in Polish poetry: A quantitative analysis

Analysis and interpretation of poetry is based on qualitative features of its text such as its semantics or means of expression as well as on general knowledge about the author and artistic period. Recent advances in automatic text processing allow for performing quantitative analysis of large sets of poetry. Their results may facilitate assessment of linguistic capabilities of its author or, in other words, his poetic mastery. This contribution presents a method of calculating the share of grammatical rhymes in Polish poetry. It is used to create a ranking of both historic and contemporary Polish poets. Comparative study and statistical analysis is developed using Pan Tadeusz by Adam Mickiewicz as a reference poem for Polish poetry. Assessment of technical mastery is one step towards the introduction of objective measures of poetic quality.

Reliability of a Minimal Competency Score for an Annual Skills Mastery Assessment

To determine whether the modified Angoff process can be used to calculate a reliable minimal competency ("cut") score for the Annual Skills Mastery Assessment (ASMA). Three panels of pharmacy faculty members used a modified Angoff method to create a minimal competency score for 60 previously used test items. The panels did not know which items were included. Data were analyzed to determine differences between rating sessions, faculty type, item difficulty, and rater scoring bias. The cut score generated was not significantly different by session or faculty type. The range of cut scores varied by less than 3% per examination. Faculty panelists correctly predicted student performance on items grouped as easy, medium, and hard. A properly constructed faculty panel can determine a reliable cut score and accurately rank relative test item difficulty using the modified Angoff process.

P69 The use of assessment in the delivery of bronchoscopy simulation- Are skills retained at assessment?

IntroductionBronchoscopy simulation is becoming increasingly used as a tool to improve the confidence and knowledge of trainees, with the intention of increasing the speed of skill acquisition. Although this training...

Keck School of Medicine of the University of Southern California

Keck School of Medicine of the University of Southern California

The Educational Relevance of Communication Disorders

The Educational Relevance of Communication Disorders

Tying the Surgical Square Knot

OPEN ACCESSJune 4, 2010Tying the Surgical Square Knot William Brescia, PhD, Steven Van Voorst, MD William Brescia, PhD University of Tennessee Health Science Center College of Medicine Google Scholar More articles by this author , Steven Van Voorst, MD HTHSC Google Scholar More articles by this author https://doi.org/10.15766/mep_2374-8265.7817 SectionsAboutAbstract ToolsDownload Citations ShareFacebookTwitterEmail AbstractAbstract Introduction: There are two basic techniques for tying the surgical square knot: the one-hand technique and the two-hand technique. The techniques differ in the fingers that they use. Students are often challenged as much by the skill of tying the surgical square knot as by anything else in the surgery clerkship. Although introduction to the task should take place in a face-to-face learning environment, the necessity for practice means study aids are useful in helping students learn to tie the surgical square knot. Methods: This self-directed learning module provides instructions on how to accomplish this foundational skill. The instructor's guide presents an overview of the module, including video content, assessment of mastery, advance preparation, and required resources. The 14.5-minute video covers the two basic techniques of tying the surgical square knot. Different grips are presented, as well as real-time demonstrations and step-by-step guidance for each technique. Instrument tying of square knots is also demonstrated. Results: Assessment of tying the surgical square knot begins with self-assessment. Once the learner feels comfortable with the skill, it must be demonstrated during a clinical situation. This is a pass/fail skill—a knot that is not tied properly will leak. The test is simple: The knot works or it does not. Discussion: This module has been used with third- and fourth-year medical students for a number of years. It functions best when combined with face-to-face instruction and the included PDF handout. Educational Objectives By the end of this module, learners will be able to demonstrate the two basic techniques of tying surgical square knots. Sign up for the latest publications from MedEdPORTAL Add your email below FILES INCLUDEDReferencesRelatedDetails FILES INCLUDED Included in this publication: Instructor's Guide.doc Surgical Square Knot.pdf SgKnot.mov To view all publication components, extract (i.e., unzip) them from the downloaded .zip file. Download editor’s noteThis publication may contain technology or a display format that is no longer in use. Copyright & Permissions© 2010 Brescia and Van Voorst. This is an open-access publication distributed under the terms of the Creative Commons Attribution-NonCommercial-Share Alike license.KeywordsTyingSquare KnotEquipmentSurgical TechniquesOperative Surgical Procedures Disclosures None to report. Funding/Support None to report. Loading ...

Development of a Reliable, Valid Annual Skills Mastery Assessment Examination

To develop a methodology for a reliable, valid annual skills mastery assessment examination to provide formative student feedback, inform curricular review, and comply with the Accreditation Council for Pharmacy Education (ACPE) Standards 2007. A sample of program-level ability-based outcomes skills were chosen for the examination. Test items were written, underwent quality control, and were scored for level of difficulty. Versions of the examination for first-, second-, third-, and fourth-year pharmacy students were developed and administered, the results were analyzed, reliability and validity were evaluated, and reports were generated. Item-writing guidelines, quality control procedures, and examination production steps were codified to create a criterion-referenced examination. Students and faculty advisors received detailed score reports and results were used to guide student performance and stimulate a review of curricular outcomes. Content, criterion, and construct validity were analyzed as defined in the literature for the intended use of this assessment tool. Data suggest the Annual Skills Mastery Assessment (ASMA) examination is both reliable and valid. Students and faculty members were surveyed regarding the usefulness of the examination. Results indicate general satisfaction with the assessment program. A reasonably reliable, reasonably valid multiple-choice annual skills mastery assessment for selected outcomes statements providing formative feedback and informed curricular review was developed.

Redesigning College Algebra: Combining Educational Theory and Web-Based Learning to Improve Student Attitudes and Performance

In 2001, Black Hills State University (BHSU) redesigned college algebra to use the computer-based mastery learning program, Assessment and Learning in Knowledge Spaces [1], historical development of concepts modules, whole class discussions, cooperative activities, relevant applications problems, and many fewer lectures. This resulted in a 21% increase in passing rate (from 54% to 75%), a 300% increase in enrollment in the next mathematics course in the program (trigonometry), a 25% improvement in attendance, and a statistically significant increase in Collegiate Assessment of Academic Proficiency (CAAP, a nationally normed test) scores. This article explores BHSU's struggles, successes, and plans for future work in making mathematics courses more accessible, engaging, and worthwhile to students.