Formative Evaluation Activities Research Articles

FORMATIVE ASSESSMENT AND THE INTERACTION BETWEEN TO AND RRS IN MATHEMATICS TEACHING

This article investigates how Formative Assessment can be influenced by Objectification Theory (OT) and how alternatives to traditional assessment can consider the ethical and subjective development of students. The analysis is carried out in the context of mathematics teaching in the 7th grade of Elementary School, focusing on the relationships between OT and RRS. To achieve this goal, we have outlined specific objectives that include the analysis of existing assessment practices, the proposition of training strategies, the incorporation of RRS in assessment, and the assessment of the impact of these practices on students’ cognitive and affective development. The case study fulfilled its objective by proposing alternatives for formative evaluation activities.

Support for people with dementia experiencing severe responsive behaviours: Unpacking the disconnect between policy and practice.

The Severe Behaviour Response Team (SBRT) program, which was established in 2015 to support aged care residents with dementia experiencing very severe and extreme responsive behaviours received far fewer referrals than projected during its first year. This article describes the outcomes of a brief survey to identify potential barriers to referrals and identify opportunities to improve the uptake of the service. A pragmatic, quasi-experimental study was conducted involving clinical leads working in a representative sample of care homes that had not used the SBRT. The study was part of the formative evaluation activities of an ongoing program evaluation. Of the 53 clinical leads that participated in the survey, one-third had not heard of the SBRT prior to being contacted. The remaining two-thirds (n=36) had not used the service due to the availability of existing resources and concerns regarding responsiveness of, and access to, the new service. Three themes emerged from the study relating to awareness of the service, responsiveness and the interface between local aged care and health services. Referrals increased following interventions to address the first two themes; however, they continue to remain well below the number projected. This indicates a fundamental disconnection between the policy design process and the day-to-day experience of residential aged care. The study highlights the importance of aged care clinical leads being engaged in dementia policy and program development processes to support improved targeting of resources.

Evaluation process associated with multiple intelligences in high school students of the Ecuadorian educational system

Formative assessment is ideal for improving teaching and learning processes. However, little is practiced systematically and a traditional approach still persists in many educational institutions that value the product rather than the process. The objective of this study is to determine the evaluative process associated with multiple intelligences in high school students from an Educational Unit in the city of Manta. Methodologically, this research project is of a mixed type with a qualitative-quantitative approach since it obtains real data that evaluates the results. To collect the information, information from primary sources will be used and will be carried out through a survey via Google to teachers and secondary information will be taken from accredited sources of virtual libraries, indexed magazines, and other academic Google publications. The results will be processed using statistical methods that will allow their analysis and interpretation. As a result of it, a set of formative evaluation activities underlying the pedagogical practice of the teacher was obtained, assessing the general process of formative evaluation of learning included in the teaching-learning processes. The process is useful for conducting a conscious and systematic formative assessment.

Lessons Learned From Dissemination of Evidence-Based Interventions for HIV Prevention.

In 1999, IOM issued a report that recommended that the Centers for Disease Control and Prevention should disseminate evidence-based HIV prevention interventions (EBIs) to be implemented by health departments, community-based organizations, drug treatment centers, and clinics. Based on these recommendations, the Diffusion of Effective Behavioral Interventions Project was initiated in 2000 and began disseminating interventions into public health practice. For 15 years, the Centers for Disease Control and Prevention has disseminated 29 EBIs to more than 11,300 agencies. Lessons were identified during the 15 years of implementation regarding successful methods of dissemination of EBIs. Lessons around selecting interventions for dissemination, developing a dissemination infrastructure including a resource website (https://effectiveinterventions.cdc.gov), and engagement with stakeholders are discussed. A continuous development approach ensured that intervention implementation materials, instructions, and technical assistance were all tailored to the needs of end users, focus populations, and agency capacities. Six follow-up studies demonstrated that adopters of EBIs were able to obtain comparable outcomes to those of the original efficacy research. The Diffusion of Effective Behavioral Interventions Project may offer guidance for other large, national, evidence-based public health dissemination projects.

User-driven Development of an Inquiry-Based Learning Platform: Qualitative Formative Evaluations in weSPOT

This paper describes the formative evaluation activities that were designed and implemented during the development of the weSPOT inquiry based learning platform. With the ambition to provide a platform that supports a broad range of inquiry activities in accordance with end-users needs, an agile software development approach was followed as a process of co-design between practitioners, researchers and developers. The paper focuses on the design of end-user centric evaluation activities for fully exploiting the potential of agile development. A detailed overview of several case studies is presented to demonstrate how implementing a continuous evaluation cycle allowed to pinpoint and help resolve arising issues in a process of collaboration between technology development and pedagogy.

¿Es posible una evaluación más rica a nivel universitario?

Es una práctica común, a nivel universitario, que la función de la evaluación se limite a la acreditación. El alto número de alumnos, el escaso tiempo con que se cuenta para desarrollar los programas, además de una creencia generalizada –no explícita– de que la educación universitaria puede soslayar el aspecto formativo de la evaluación sin afectar su calidad, conspiran contra un uso continuo y efectivo de la información que la evaluación puede proveer para realizar un feedback positivo del proceso educativo.En este trabajo se describe una experiencia evaluativa llevada a cabo en un curso de Álgebra y Geometría Analítica, materia de primer año de las carreras de Ingeniería en la Universidad Tecnológica Nacional de la República Argentina. La misma consistió en introducir actividades de evaluación formativa periódica y una instancia de evaluación sumativa enriquecida en el trabajo del aula, con el objetivo de sacar provecho de la información continua que estas prácticas pueden ofrecer, tanto a docentes como a alumnos, para sustentar la realimentación de los procesos de enseñanza y aprendizaje, en busca de un mejoramiento de la calidad de la formación matemática de los futuros ingenieros.

Design and evaluation of mixed interactive museographic exhibits

This paper describes the four steps of a codesign process dedicated to the development of mixed interactive exhibits in Museums. The goal is to propose a well identified set of steps for accompanying the development of such systems, to document them and to suggest the most appropriate tools for each step. Based on a concrete design experience of a mixed interactive system in a museum, each step is illustrated and design decisions are highlighted with regards to the steps of the design process. A comparison of our codesign process with similar approaches applied in the field of interactive software engineering, participatory design or museum codesign is then performed; lessons learnt from the use of the process highlight benefits and limitations. Tightly coupled to this process, we then present different mechanisms and settings for evaluating visitors’ experience. Such mechanism involves lab and field activities requiring different methodologies and technical platforms. These formative evaluation activities are illustrated through some results of document-based usability inspections, user tests, field experiments and field studies. This paper reports lessons learnt and descriptive results of the prototype evaluations. Finally, we propose the ways for improving the use of codesign and evaluation of mixed interactive museographic exhibits.

Summative Assessment in Medicine: The Promise of Simulation for High‐stakes Evaluation

Throughout their careers, physicians are exposed to a wide array of assessments, including those aimed at evaluating knowledge, clinical skills, and clinical decision-making. While many of these assessments are used as part of formative evaluation activities, others are employed to establish competence and, as a byproduct, to promote patient safety. In the past 10 years, simulations have been successfully incorporated in a number of high-stakes physician certification and licensure exams. In developing these simulation-based assessments, testing organizations were able to promote novel test administration protocols, build enhanced assessment rubrics, advance sophisticated scoring and equating algorithms, and promote innovative standard-setting methods. Moreover, numerous studies have been conducted to identify potential threats to the validity of test score interpretations. As simulation technology expands and new simulators are invented, this groundbreaking work can serve as a basis for organizations to build or expand their summative assessment activities. Although there will continue to be logistical and psychometric problems, many of which will be specialty- or simulator-specific, past experience with performance-based assessments suggests that most challenges can be addressed through focused research. Simulation, whether it involves standardized patients (SPs), computerized case management scenarios, part-task trainers, electromechanical mannequins, or a combination of these methods, holds great promise for high-stakes assessment.

A formative evaluation in maternal and child health practice: the Partners for Life Nutrition Education Program for pregnant women.

This paper describes the formative evaluation of the Partners For Life Program that was developed to change dietary behavior of low income pregnant women in the Mississippi Delta region. A diverse group of nutrition/health professionals, adapted the Expanded Food and Nutrition Education Program's (EFNEP) methodology for pregnant patients in the local Women, Infants, and Children program who were receiving maternity care at the county health department. Formative evaluation activities included gathering data to determine 1) whether a new nutrition curriculum, modeled after EFNEP could be created, 2) whether peer educators could be recruited and trained to deliver the intervention, 3) whether pregnant women could be recruited to participate in the new program, 4) whether a pilot test of the new intervention would produce short-term impact in nutrition knowledge and dietary behavior, and 5) reactions of pregnant women on the usability of the new program. Data were gathered through use of the Program Implementation Index, focus groups, and a retrospective record review. Pilot test assessments included tests of nutrition knowledge and self-reported changes in dietary behavior. The formative evaluation demonstrated both positive and negative outcomes. Positive data included 1) successful recruitment and training of the peer educators to deliver the intervention; 2) successful recruitment of the targeted population for the pilot study; 3) complete information on project questionnaires and measuring forms; and 4) among those who completed the program, a statistically significant improvement in nutrition knowledge and dietary behavior. Two negative aspects in this formative study were 1) the Program Implementation Index quantitatively showed that the program experienced problems retaining participants it recruited and 2) deviation of the timeframe for intervention delivery. Program length was identified as the primary reason for participant attrition. It is important for program developers to use results from formative evaluations to make changes in problem areas prior to implementation of a full-scale impact evaluation.

Exploring the potential of a computer tool for instructional developers

Information and communication technology tools currently permeate almost every professional domain. Those geared toward the field of instructional development have emerged in recent years. This article explores the potential for linking the domains of computer support and instructional development. This article reports on the design and evaluation of CASCADE (Computer Assisted Curriculum Analysis, Design and Evaluation), a computer system that supports instructional developers during formative evaluation efforts. Five prototypes of CASCADE were created and evaluated on the basis of their validity (reflection of state-of-the-art knowledge and internal consistency); practicality (ability to meet the needs, wishes and contextual constraints of the target group); and effectiveness (improved user task performance). The results of this study suggest that the use of CASCADE could: (a) improve the consistency of formative evaluation plans and activities; (b) motivate developers by elevating their confidence in using formative evaluation activities; (c) save time; and (d) help to provide justifications for decisions made.

Courseware Evaluation Activities in Europe

In the ECOSET-project data on evaluation activities of nineteen organizations in ten member states of the EG are gathered by questionnaire. Four of the organizations are interviewed for more and detailed information about these activities. Ten out of nineteen organizations do as well summative as formative evaluations and four organizations only perform summative evaluation activities. Five organizations are involved in formative evaluation activities only. The summative evaluation activities in the member states vary from a critical appraisal of a piece of courseware to extensive in-depth testing. Three groups of organizations are distinguished. The first group does not use formal instruments. The data indicate that this group are more or less on the line of product reviews of educational software. A second group (5 organizations) are using formal instruments, but the evaluation in real classroom situations is not a standard procedure. The third group of organizations use formal instruments and the educational software is tested in real classroom-situations. There are indications of differences between group 1 and 2 in the relation between development activities and the target group for the results. A remarkable aspect is the difference in dissemination of results between group 2 and group 3. Teachers in both groups are an important target group for the results, but in group 2 there is only one organization that use educational or other magazines for the dissemination of the results. In group 3 all the organizations make use of magazines. Fifteen organizations perform formative evaluation activities in the decision as well as in the design and development phase. Nine of them are also involved in evaluation activities in the dissemination phase. In the decision phase thirteen organizations do comparisons with other products and eleven of them are looking at the technical feasibility of products. Ten organizations do needs analysis studies. As evaluation activities in the design & development phase the evaluations of pre-releases of the software and release of the documentation are most often mentioned. Other activities are evaluations of prototypes and continuing consultancy during the design and technical implementation stages. Products most often mentioned as being used as input for activities in this phase are the descriptions of the ideas behind the product, global prototypes and product definitions. In the dissemination phase three organizations assess results as one of their evaluation activities. In the group of organizations performing formative evaluation activities two elements are remarkable. Organizations with few (<6) evaluation activities hardly train their evaluators. Almost all other organizations give training to in-house evaluators, subject matter specialists and/or teachers. If the number of evaluation activities gives an indication of the role of evaluation in the development process, you might say that those organizations also value training for evaluators as a necessity for good evaluations. The organizations with a lot of evaluation activities all use data of own or other observers. The aspect of training may also be connected with this fact. For observations the inter subjectivity is an important factor. Inter subjectivity you will get by training your observers.

Historical overview of formative evaluation of instructional media products

Though the term “formative evaluation” was coined only in 1967, formative evaluation activities, by different labels, have been going on for a long time. This article traces those activities—from study of World War I training films in the 1920s to detailed models and microcomputer technology currently employed.

Formative evaluation of instructional development

In this paper we have tried to describe the research pertinent to the activity of formative evaluation. While the number of good examples of formative evaluation is expanding, the level of research into the process is relatively limited. Perhaps a compromise to the difficult task of accumulating research data on formative evaluation might be suggested. When formative evaluation activities have been successful in terms of program effects and staff satisfaction, then detailed technical reports might be made available to the evaluation public. Formative evaluation might improve as a consequence of the technology developed in the course of finding solutions to development data problems. Both the increase in technology as a result of dissemination of successful procedures and the pursuit of experimental research on the process of evaluation might ultimately remove formative evaluation recommendations from the realm of seers and clairvoyants and thrust it closer to, if not into, the domain of scientific application.