Evaluation Of Educational Intervention Research Articles

A Review of Statistical Methods for Generalizing From Evaluations of Educational Interventions

School-based evaluations of interventions are increasingly common in education research. Ideally, the results of these evaluations are used to make evidence-based policy decisions for students. However, it is difficult to make generalizations from these evaluations because the types of schools included in the studies are typically not selected randomly from a target population. This paper provides an overview of statistical methods for improving generalizations from intervention research in education. These are presented as a series of steps aimed at improving research design—particularly recruitment—as well as methods for assessing and summarizing generalizability and estimating treatment impacts for clearly defined target populations.

Effect of educational intervention on knowledge, attitude and practice of hepatitis B vaccine among medical students

Background: Aim of the study were to assess the impact of education on knowledge, attitude and practice of Hepatitis B vaccine among medical students and to educate the students regarding Hepatitis B vaccination.Methods: Non-randomized before and after comparison study was conducted to evaluate changes in knowledge, attitude and practice of Hepatitis B vaccination (HBV), assigning structured questionnaire of 26 different statements concerning knowledge, attitude and practice by pre and post educational intervention on Hepatitis B and its vaccination.Results: The study was conducted among 100 second year medical students. The response rate was 100%. There was statistically significant improvement in knowledge from pre-test mean scores for modes of transmission (87.4+4.70 vs 95.8+1.61; p=0.0001), preventive measures (92+0.47 vs 98+0.94; p=0.001) and Hepatitis B vaccine (71+4.72 to 84.7+6.65; p=0.0001) to post-test. While the increase in mean scores from pre-test for attitude (68.6+9.21 vs 77.43+11.1) and practice (55+25.41 vs 65.6+32.6) were statistically significant in post-test (p=0.0001; p=0.001 respectively).Conclusions: Structured educational intervention among medical students about Hepatitis B vaccination showed improved knowledge and behaviour and also increased the percentage of students willing to get screened and their participation in health education programmes related to Hepatitis B. However, there is slight lack of knowledge regarding the transmission of Hepatitis B and its vaccination schedule. In this regard, implementation and evaluation of educational intervention is needed as a preventative measure.

Unconscious hegemonic research and evaluation of education interventions.

Unconscious hegemonic research and evaluation of education interventions.

Systematic review of educational interventions for looked-after children and young people: Recommendations for intervention development and evaluation.

Looked‐after children and young people (LACYP) are educationally disadvantaged compared to the general population. A systematic review was conducted of randomised controlled trials evaluating interventions aimed at LACYP aged ≤18 years. Restrictions were not placed on delivery setting or delivery agent. Intervention outcomes were: academic skills; academic achievement and grade completion; special education status; homework completion; school attendance, suspension, and drop‐out; number of school placements; teacher‐student relationships; school behaviour; and academic attitudes. Fifteen studies reporting on 12 interventions met the inclusion criteria. Nine interventions demonstrated tentative impacts. However, evidence of effectiveness could not be ascertained due to variable methodological quality, as appraised by the Cochrane risk of bias tool. Theoretical and methodological recommendations are provided to enhance the development and evaluation of educational interventions.

Parental self-efficacy and the management of childhood atopic eczema: development and testing of a new clinical outcome measure.

Effective parental management of childhood eczema requires education and support to reduce disease severity and improve the child's quality of life. Self-efficacy is a key factor influencing effective chronic disease management, yet there are no published scales to measure parental self-efficacy in managing childhood eczema. The Parental Self-Efficacy with Eczema Care Index (PASECI) was designed to measure parental self-efficacy in managing childhood eczema as a pre- and postintervention tool in the evaluation of a structured Eczema Education Programme (EEP). To develop and test the validity, reliability and sensitivity of a new outcome measure (PASECI) designed to assess parental self-efficacy in managing their child's eczema to determine pre- vs. postintervention changes in educational intervention evaluation studies. PASECI was developed from the literature, expert consultation and piloting of a 40-item prototype. The final 29-item scale is arranged in four subscales. Parents of children with eczema aged 0-16 years (n = 242) attending the EEP were assessed at 1 week pre-EEP and 4 weeks postintervention. Cronbach's α and factor analyses were undertaken. PASECI has face, content and construct validity. It is reliable, with high item internal consistency (α > 0·87 in all domains). Factor analysis revealed four viable domains. It was sensitive to change for postintervention measures using sign tests (P < 0·001). PASECI is a useful, valid, reliable and sensitive evaluative outcome measure of self-efficacy in parents managing childhood eczema.

Development of early mathematical skills with a tablet intervention: a randomized control trial in Malawi.

Evaluation of educational interventions is necessary prior to wide-scale rollout. Yet very few rigorous studies have been conducted on the effectiveness of tablet-based interventions, especially in the early years and in developing countries. This study reports a randomized control trial to evaluate the effectiveness of a tablet intervention for supporting the development of early mathematical skills in primary school children in Malawi. A total sample of 318 children, spanning Standards 1–3, attending a medium-sized urban primary school, were randomized to one of three groups: maths tablet intervention, non-maths tablet control, and standard face-to-face practice. Children were pre-tested using tablets at the start of the school year on two tests of mathematical knowledge and a range of basic skills related to scholastic progression. Class teachers then delivered the intervention over an 8-weeks period, for the equivalent of 30-min per day. Technical support was provided from the local Voluntary Service Overseas (VSO). Children were then post-tested on the same assessments as given at pre-test. A final sample of 283 children, from Standards 1–3, present at both pre- and post-test, was analyzed to investigate the effectiveness of the maths tablet intervention. Significant effects of the maths tablet intervention over and above standard face-to-face practice or using tablets without the maths software were found in Standards 2 and 3. In Standard 3 the greater learning gains shown by the maths tablet intervention group compared to both of the control groups on the tablet-based assessments transferred to paper and pencil format, illustrating generalization of knowledge gained. Thus, tablet technology can effectively support early years mathematical skills in developing countries if the software is carefully designed to engage the child in the learning process and the content is grounded in a solid well-constructed curriculum appropriate for the child’s developmental stage.

Randomized impact evaluation of education interventions: experiences and lessons from a reading to learn intervention in East Africa

This paper presents the experiences and lessons learnt during the design and implementation of the randomized impact evaluation (IE) of a reading to learn (RtL) intervention in early primary grades. The study was to assess the impact of RtL on literacy and numeracy among pupils in low-performing districts in East Africa. The intervention was designed in a way that in each country one district implemented an intervention package that included teacher training, and teaching and learning materials; while the other district implemented this same package with an additional parental involvement component. Baseline data were collected in mid-2009 in Grades 1 and 2, and in 2010 for the incoming Grade 1. The endline data were collected in mid-2011. A total of 119 treatments and 110 control schools participated in the study. The randomized design provided an opportunity to attribute causality and also qualitatively establish pathways through which the intervention impacted on learning. The paper provides lessons learnt and demonstrates how the challenges faced during the evaluation were addressed. Additionally, the paper shows how IE was utilized to provide robust evidence of what works thus enabling policy-makers to make decisions on sustainability and scale-up of education interventions.

Caregivers' Knowledge, Concerns and Management of Pediatric Febrile Convulsions.

PURPOSE: The purpose of this study was to investigate caregivers' knowledge, concerns, and management of children with febrile convulsions (FC).METHODS: A descriptive correlation study was conducted with 133 caregivers whose children had been diagnosed with a FC.A self-administered questionnaire was used for data collection. Descriptive statistics, t-test, one-way analysis of variance and Pearson's correlation were used for data analysis.RESULTS: The mean percent of correct answers related to knowledge was 48.5%. Many caregivers believed that FC causes brain damage and did not know that risk of subsequent epilepsy in FC is rare. Levels of concern about FC were high. Caregivers were highly concerned about further FC attacks in the night and tended to worry that Febrile children were apt to get a fever. Many caregivers used management practices which are not recommended for FC in children. There was a statistically significant negative correlation between caregivers' knowledge and concerns about FC. There was also a positive correlation between caregivers' knowledge and management of FC.CONCLUSION: Findings suggest that improvements are needed in caregivers' knowledge and management of FC. Caregivers' concerns related to misconception need to be addressed.Development and evaluation of educational interventions on changing caregivers' management of FC are recommended.

A Multi-Institutional Medical Educational Collaborative

Educational collaboratives offer a promising approach to disseminate educational resources and provide faculty development to advance residents' training, especially in areas of novel curricular content; however, their impact has not been clearly described. Advocacy training is a recently mandated requirement of the Accreditation Council for Graduate Medical Education that many programs struggle to meet.The authors describe the formation (in 2007) and impact (from 2008 to 2010) of 13 California pediatric residency programs working in an educational collaboration ("the Collaborative") to improve advocacy training. The Collaborative defined an overarching mission, assessed the needs of the programs, and mapped their strengths. The infrastructure required to build the collaboration among programs included a social networking site, frequent conference calls, and face-to-face semiannual meetings. An evaluation of the Collaborative's activities showed that programs demonstrated increased uptake of curricular components and an increase in advocacy activities. The themes extracted from semistructured interviews of lead faculty at each program revealed that the Collaborative (1) reduced faculty isolation, increased motivation, and strengthened faculty academic development, (2) enhanced identification of curricular areas of weakness and provided curricular development from new resources, (3) helped to address barriers of limited resident time and program resources, and (4) sustained the Collaborative's impact even after formal funding of the program had ceased through curricular enhancement, the need for further resources, and a shared desire to expand the collaborative network.

Evaluations of Educational Interventions: Getting Them Published and Increasing Their Impact

Evaluations of Educational Interventions: Getting Them Published and Increasing Their Impact

Implementation of a multi-level evaluation strategy: a case study on a program for international medical graduates

Evaluation of educational interventions is often focused on immediate and/or short-term metrics associated with knowledge and/or skills acquisition. We developed an educational intervention to support international medical graduates working in rural Victoria. We wanted an evaluation strategy that included participants' reactions and considered transfer of learning to the workplace and retention of learning. However, with participants in distributed locations and limited program resources, this was likely to prove challenging. Elsewhere, we have reported the outcomes of this evaluation. In this educational development report, we describe our evaluation strategy as a case study, its underpinning theoretical framework, the strategy, and its benefits and challenges. The strategy sought to address issues of program structure, process, and outcomes. We used a modified version of Kirkpatrick's model as a framework to map our evaluation of participants' experiences, acquisition of knowledge and skills, and their application in the workplace. The predominant benefit was that most of the evaluation instruments allowed for personalization of the program. The baseline instruments provided a broad view of participants' expectations, needs, and current perspective on their role. Immediate evaluation instruments allowed ongoing tailoring of the program to meet learning needs. Intermediate evaluations facilitated insight on the transfer of learning. The principal challenge related to the resource intensive nature of the evaluation strategy. A dedicated program administrator was required to manage data collection. Although resource-intensive, we recommend baseline, immediate, and intermediate data collection points, with multi-source feedback being especially illuminating. We believe our experiences may be valuable to faculty involved in program evaluations.

An evaluation of educational interventions aimed at decreasing aedes aegypti mosquito breeding sites: A systematic Review.

Review Questions/Objectives The objective of this systematic review is to synthesize the best available evidence on whether educational interventions are effective at reducing Aedes aegypti mosquito breeding places. Specific review questions to be answered are: Do educational interventions: Reduce mosquito breeding sites? Increase knowledge, attitudes and practice aimed at decreasing aedes aegypti mosquito breeding sites as measured by specific indices An additional review question is: Do educational interventions have any side effects? Inclusion Criteria Types of participants This systematic review will consider comparative studies where the focus was any human individuals living in communities infested by dengue disease. Types of interventions All studies evaluating educational interventions aimed at decreasing aedes aegypti mosquito breeding sites will be included in this review. Educational interventions will be defined as any educational programme and/or campaign which disseminate information and knowledge about dengue to inhabitants and populations where dengue is considered a problem. Types of outcomes The primary outcome of interest is the objective measure of Aedes aegypti mosquito breeding places as measured by House Index (HI), Container Index (CI), Breteau Index (BI) and Ovitrap Index (OI). House Index (HI) describes the percentage of houses positive for larvae, Container Index (CI) is the percentage of positive container (water filled- containers) for larvae and Breteau Index (BI) represents the number of positive container for larvae per 100 houses. Ovitrap Index (OI) is related to the number of ovitraps positively divided by the total number of ovitraps examined multiplied by 100%. Secondary outcomes will include changes in community behavior resulting from increased knowledge, attitudes and practice (KAP) and possible side effects of interventions. These will be measured by qualitative findings and KAP surveys utilising pre-structured and pre-tested questionnaires if included within papers that are chosen for inclusion.

Uso prudente de antibióticos y propuestas de mejora en los centros sociosanitarios

Antibiotics are among the most prescribed drugs in long-term-care facilities, accounting for nearly 40% of all prescribed drugs. It is estimated that the likelihood that a resident patient will receive at least one course of antibiotics per year is 70%. Prevalence is 4%-10%, and incidence between 3 and 4 treatments per 1,000 stays, though there is great variability between apparently similar centres in terms of consumption. On the other hand, a high percentage of antibiotic treatments are considered inappropriate because most are empirical, often prescribed to treat bacterial colonisation and too many broad-spectrum antibiotics are used for too long. The main consequences of inappropriate use are the increase of microbial resistance and side effects in this frail, elderly population. The challenges of improving the use of antibiotics lie in the basic characteristics of these centres, with limited resources to carry out organised activities to control infections and antibiotic policies. Some possibilities for improvement include organising local programmes focused on the most basic aspects of appropriate use of antibiotics, from the diagnosis of infection and the development of empirical and targeted treatment protocols, to the implementation of minimal activities of analysis and surveillance. Centers with more resources and greater experience should lead the implementation and evaluation of educational interventions and research projects.

Evaluation of a clinical examination skills training course in an undergraduate pharmacy programme

Abstract Objective To introduce and evaluate a short clinical examination skills course in a BPharm programme. The study objectives were to assess needs, explore attitudes, record perceived competence requirements and assess the value of physical examination skills learning. Setting BPharm programme in Auckland, New Zealand. Participants were students enrolled in years 3 and 4 of the programme (2003). Method The design was a longitudinal, dual cohort, educational intervention evaluation using a self-completed questionnaire. An examination skills component was added to the fourth year of the pharmacy programme. Year 3 and 4 students were recruited, and completed the questionnaire at two points. Year 3 students were sampled one year before and soon after the skills sessions. Year 4 students were sampled after the sessions and 18 months later (once registered). The questionnaire sought their attitudes towards clinical skills training and practical relevance for future practice, and evaluated their learning experience. Key findings Response rates at the four points were 42–67%. Year 3 students identified a similar set of appropriate skills to those actually taught in year 4. Overall, attitudes to introducing examination skills learning were positive at all points. At follow-up, both cohorts agreed more strongly that examination skills training should be core (significantly for registered pharmacists versus year 3, P &amp;lt; 0.006). Measuring manual blood pressure was deemed the most difficult skill. All taught skills were used in practice except for respiratory rate; most used were body mass index (BMI), temperature and peak flow measurement. Conclusions There was a close correlation between what was offered in the course, what students felt they needed to learn and what was relevant in practice. Once registered, pharmacists were aware of their limitations and level of competence in relation to clinical skills. The small changes in attitudinal scores appear to reflect maturity and experience. The study design allowed us to adapt the educational component to student need. Health professional educators need to be aware of and respond to changes in professional scopes of practice.

Evaluating the Effectiveness of a Reading Remediation Program in a Public School Setting

This article reports an evaluation of a reading remediation program delivered to 151 high-risk students in a public school setting. Implementation fidelity was assessed by the amount of time each child was exposed to the program. Despite receiving fewer than the recommended hours of instruction, analyses of standard achievement scores indicated significantly greater change than expected in reading comprehension, reading fluency, and word attack skills. Number of individual hours of instruction yielded the most conservative estimates of change across time. Findings are discussed within the context of conducting effectiveness evaluations of educational interventions and the need to monitor implementation fidelity at the individual level to best inform decision making and resource allocation.

Mixed‐Method Evaluation of Continuing Professional Development: Applications in Cultural Competence Training

Continuing professional development has been identified as an important tool for advancing the competence of social workers, yet outcomes of professional development are conventionally evaluated using methods that cannot provide trustworthy information about program effectiveness. This paper suggests that innovations in social research can be applied to strengthen the evaluation of educational outcomes from continuing professional development. The benefits of designing a mixed‐method evaluation of educational interventions are explored by reviewing the experience of implementing a cultural competence continuing professional development workshop series in Toronto, Canada. Review of the process demonstrates that the combination of methods results in a richer data set that can answer questions about both the process of the learning experience, and the effect that training has on professional practice.

Evidence‐based medical education –quo vadis?

The evidence base for most educational initiatives, at least until very recently, is largely composed of low-level evidence. Four major barriers underlie this historical observation, namely: (1) perceived ethical and acceptability problems arising from the unequal treatment of learners in experimental designs; (2) limited choice of outcome measures and validated instruments; (3) time and resource constraints; and (4) methodological issues concerning contextual confounding and small sample sizes. We advocate the adoption of a 'balanced scorecard' approach in the evaluation of education interventions that brings together a comprehensive panel of outcomes under one framework. We require a diversity of rigorously applied methods to generate these outcomes, drawing from the quantitative and qualitative disciplines of epidemiology, psychology and economics. We further suggest that the research community discuss and agree on a standardized set of common metrics or benchmarks. We conclude with a case study examining whether a hand-held computer clinical decision support tool improves clerkship learning of evidence-based medicine. The era of Brownian motion in health education research is over. What we demand in terms of burden of proof for educational effectiveness should be no less rigorous than our call for an ever escalating threshold concerning evidence of clinical care.

Changing public attitudes to antibiotic prescribing: can the internet help?

Antimicrobial resistance is a global problem with serious implications for modern medicine. Education of the public is essential for reducing patient pressure on GPs and subsequent inappropriate prescribing. Evaluation of educational interventions is necessary to assess their impact on public knowledge and attitudes. The aim of this study was to evaluate the effect of a health information website, part of the National electronic Library of Infection, on user knowledge and attitudes. Questionnaires testing user knowledge and attitudes before and after using the website. There were significant improvements in knowledge about the use of antibiotics and antibiotic resistance. Expectations that antibiotics should be prescribed were significantly reduced after using the website. Health professionals showed a significantly greater knowledge about antibiotics and were less likely to expect antibiotics to be prescribed for acute otitis media than non-health professionals before using the website. There was no significant difference between the knowledge of these groups after using the website, but non-health professionals continued to have higher expectations of antibiotics being prescribed than health professionals. Health information websites can play a significant role in influencing public knowledge and attitudes. Further research is needed to investigate how people learn from these interventions and to determine their long-term impact on public attitudes and subsequent behaviour.

Reducing psychiatric stigma and discrimination: evaluation of educational interventions in UK secondary schools.

The persistent and disabling nature of psychiatric stigma has led to the establishment of global programmes to challenge the negative stereotypes and discriminatory responses that generate social disability, but these initiatives are rarely evaluated. To assess the effectiveness of an intervention with young people aimed at increasing mental health literacy and challenging negative stereotypes associated with severe mental illness. A total of 472 secondary school students attended two mental health awareness workshops and completed pre- and post-questionnaires detailing knowledge, attitudes and behavioural intentions. Young people use an extensive vocabulary of 270 different words and phrases to describe people with mental health problems: most were derogatory terms. Mean positive attitude scores rose significantly from 1.2 at baseline to 2.8 at 1-week follow-up and 2.3 at a 6-month follow-up. Changes were most marked for female students and those reporting personal contact with people with mental illness. Short educational workshops can produce positive changes in participants' reported attitudes towards people with mental health problems.

Moving Toward the Eradication of Syphilis

This study analyzed syphilis incidence among active duty U.S. Sailors and Marines and explored opportunities and strategies for the Department of the Navy to contribute to the achievement of the National Syphilis Elimination Plan. From 1987 through 1999, there were 1,886 cases of syphilis among active duty members reported to the Navy Environmental Health Center. Most were male (90%), younger than 30 years (81%), and black (64%). Most were diagnosed within the continental United States (79%). Incidence rates of primary and secondary syphilis per 100,000 among the active duty force declined steadily, from 37 in 1990 to 3 in 1999. Strategies suggested to further reduce syphilis among active duty members include the following: (1) Navy Medicine adoption and tracking of the national target of < 0.4 cases per 100,000 by 2005; (2) expand training of medical professionals in client-centered prevention counseling; (3) include affected populations in the design of interventions; (4) conduct outcome evaluations of educational interventions; (5) examine condom access policies; and (6) evaluate the current syphilis surveillance system.