Evaluation Efforts Research Articles

Collaborative capacity-building for collective evaluation: a case study with informal science education centers

ABSTRACT This paper reports on the results of a research-practice partnership that included a U.S. university research team and a state-funded grants program that supports informal science education. The goal of the 2-year partnership was to develop an empirically-tested model for collaboration and capacity-building (CCB) in support of collective evaluation. The model was implemented with 23 informal science education (ISE) organizations across one U.S. state. A comparison of pre and post-surveys indicate that the goals of the CCB model were attained. Participants reported an (1) increased perception of a common agenda, (2) increased capacity for conducting evaluation, and (3) increased peer-peer engagement. Participants also highlighted ongoing barriers to collective evaluation and suggested specific supports to overcome these barriers. This collaborative capacity-building model, although tested with ISE providers, is generalizable to other collective evaluation efforts.

A roadmap for improving nursing program outcomes

Evaluating various nursing program outcomes helps faculty assess program quality. Evaluation efforts may reveal programs are not meeting their established benchmarks. Nurse educators, along with internal and external stakeholders, should engage in systematic, yet purposeful quality improvement efforts to boost program outcomes that display declines in performance. Faculty may struggle to determine needed changes and how to successfully implement and evaluate modifications that can lead to enhanced performance. This article presents a roadmap for improving program outcomes. Systematic, purposeful, and ongoing quality improvement efforts help prepare caring, safe, and competent nurses that positively impact healthcare.

PyFLEXTRKR: a flexible feature tracking Python software for convective cloud analysis

Abstract. This paper describes the new open-source framework PyFLEXTRKR (Python FLEXible object TRacKeR), a flexible atmospheric feature tracking software package with specific capabilities to track convective clouds from a variety of observations and model simulations. This software can track any atmospheric 2D objects and handle merging and splitting explicitly. The package has a collection of multi-object identification algorithms, scalable parallelization options, and has been optimized for large datasets including global high-resolution data. We demonstrate applications of PyFLEXTRKR on tracking individual deep convective cells and mesoscale convective systems from observations and model simulations ranging from large-eddy resolving (∼100s m) to mesoscale (∼10s km) resolutions. Visualization, post-processing, and statistical analysis tools are included in the package. New Lagrangian analyses of convective clouds produced by PyFLEXTRKR applicable to a wide range of datasets and scales facilitate advanced model evaluation and development efforts as well as scientific discovery.

Benchmarking high-resolution hydrologic model performance of long-term retrospective streamflow simulations in the contiguous United States

Abstract. Because use of high-resolution hydrologic models is becoming more widespread and estimates are made over large domains, there is a pressing need for systematic evaluation of their performance. Most evaluation efforts to date have focused on smaller basins that have been relatively undisturbed by human activity, but there is also a need to benchmark model performance more comprehensively, including basins impacted by human activities. This study benchmarks the long-term performance of two process-oriented, high-resolution, continental-scale hydrologic models that have been developed to assess water availability and risks in the United States (US): the National Water Model v2.1 application of WRF-Hydro (NWMv2.1) and the National Hydrologic Model v1.0 application of the Precipitation–Runoff Modeling System (NHMv1.0). The evaluation is performed on 5390 streamflow gages from 1983 to 2016 (∼ 33 years) at a daily time step, including both natural and human-impacted catchments, representing one of the most comprehensive evaluations over the contiguous US. Using the Kling–Gupta efficiency as the main evaluation metric, the models are compared against a climatological benchmark that accounts for seasonality. Overall, the model applications show similar performance, with better performance in minimally disturbed basins than in those impacted by human activities. Relative regional differences are also similar: the best performance is found in the Northeast, followed by the Southeast, and generally worse performance is found in the Central and West areas. For both models, about 80 % of the sites exceed the seasonal climatological benchmark. Basins that do not exceed the climatological benchmark are further scrutinized to provide model diagnostics for each application. Using the underperforming subset, both models tend to overestimate streamflow volumes in the West, which could be attributed to not accounting for human activities, such as active management. Both models underestimate flow variability, especially the highest flows; this was more pronounced for NHMv1.0. Low flows tended to be overestimated by NWMv2.1, whereas there were both over and underestimations for NHMv1.0, but they were less severe. Although this study focused on model diagnostics for underperforming sites based on the seasonal climatological benchmark, metrics for all sites for both model applications are openly available online.

Facilitators of palliative care quality improvement team cohesion: Lessons from a seven-site implementation project in India.

The Palliative Care: Promoting Access and Improvement of the Cancer Experience (PC-PAICE) initiative is a team-based, palliative care (PC) quality improvement (QI) project working to promote high-quality PC in India. As a PC QI initiative, PC-PAICE implementation relied upon building interdisciplinary teams, providing the ideal context for understanding facilitators of team cohesion that compelled clinical, organizational, and administrative team members to work together. There is an opportunity to leverage the intersection between QI implementation and organizational theory to inform and improve implementation science. As a subaim of a larger implementation evaluation, we aimed to identify facilitators of team cohesion within QI implementation context. A quota sampling approach captured the perspectives of 44 stakeholders across three strata (organizational leaders, clinical leaders, and clinical team members) from all seven sites through a semistructured interview guide informed by the Consolidated Framework for Implementation Research (CFIR). We used a combination of inductive and deductive approaches informed by organizational theory to identify facilitators. We identified three facilitators of PC team cohesion: (a) balancing formalization and flexibility around team roles, (b) establishing widespread awareness of the QI project, and (c) prioritizing a nonhierarchical organizational culture. Leveraging CFIR to analyze PC-PAICE stakeholder interviews created a data set conducive to understanding complex multisite implementation. Layering role and team theory to our implementation analysis helped us identify facilitators of team cohesion across levels within the team (bounded team), beyond the team (teaming), and surrounding the team (culture). These insights demonstrate the value of team and role theories in implementation evaluation efforts.

Quantifying Value of Worksite Well-Being Programs With a Health Equity Lens: It's Time to Raise the Bar.

Calls to address workforce health inequities have been met with expanded well-being programming, without an assessment of programming relevance or evaluation of equity-related outcomes. The goal of this commentary is to summarize current practices and gaps in evaluation of well-being programs, provide evidence supporting the need for broader participation in well-being offerings, and make recommendations to incorporate health equity measures into planning and evaluation efforts conducted by both employers and vendors. Data sharing, expanded measurement, more rigorous evaluation methods, and alignment of goals are some of the recommendations to better address health inequities and differential participation among employees.

Association of Training in Basic Life Support with the Evolution of Cardiopulmonary Resuscitation Performed by Firefighters.

This study aimed to compare the results of professional technical and anthropometric anamnesis data with the transmission of external chest compressions performed by military firefighters at different execution times. The objective was to evaluate the performance and perceived effort of the sequence of external chest compressions performed in two minutes, as well as the evolution of the technique over time. This was a descriptive, correlational study involving adult firefighters who were members of a specific firefighter group, comprising a population of 105 individuals with a voluntary sample of 44 participants. The study used a Bayesian statistical approach to provide probabilistic expressions. The participants had an average work experience of 17 years, an average age of 38.6 years, an average weight of 81.48 kilograms, an average height of 176 centimeters, and an average of 2.5 qualifications. The results indicated that the firefighters performed external chest compressions with excellent technique and a moderate level of perceived effort in a two-minute evaluation. The evaluation of the evolution of the technique over time showed that the participants were able to maintain high-quality compressions for an average of 6 minutes, with a maximum of 20 uninterrupted minutes. The study underscores the critical role of professional firefighters in performing and maintaining high-quality external chest compressions, which has the potential to reduce morbidity and mortality in cases of cardiorespiratory arrest.

Generating consensus on road safety issues and priorities in Ghana: A modified Delphi approach

BackgroundImplementation of evidence-based approaches to reduce the substantial health, social, and financial burdens of road traffic injuries and deaths in Ghana and other low-and-middle-income countries (LMICs) is vitally important. Consensus from national stakeholders can provide insight into what evidence to generate and which interventions to prioritize for road safety. The main objective of this study was to elicit expert views on the barriers to reaching international and national road safety targets, the gaps in national-level research, implementation, and evaluation, and the future action priorities. Materials and MethodsWe used an iterative three-round modified Delphi process to generate consensus among Ghanaian road safety stakeholders. We defined consensus as 70% or more stakeholders selecting a specific response in the survey. We defined partial consensus (termed “majority”) as 50% or more stakeholders selecting a particular response. ResultsTwenty-three stakeholders from different sectors participated. Experts generated consensus on barriers to road safety goals, including the poor regulation of commercial and public transport vehicles and limited use of technology to monitor and enforce traffic behaviors and laws. Stakeholders agreed that the impact of increasing motorcycle (2- and 3-wheel) use on road traffic injury burden is poorly understood and that it is a priority to evaluate road-user risk factors such as speed, helmet use, driving skills, and distracted driving. One emerging area was the impact of unattended/disabled vehicles along roadways. There was consensus on the need for additional research, implementation, and evaluation efforts of several interventions, including focused treatment of hazardous spots, driver training, road safety education as part of academic curricula, promotion of community involvement in first aid, development of strategically positioned trauma centers, and towing of disabled vehicles. ConclusionThis modified Delphi process with stakeholders from Ghana generated consensus on road safety research, implementation, and evaluation priorities.

Reaching Populations to Address Disparities in Cancer Care Delivery: Results From a Six-Site Initiative.

Large segments of the US population do not receive quality cancer care due to pervasive and systemic inequities, which can increase morbidity and mortality. Multicomponent, multilevel interventions can address inequities and improve care, but only if they reach communities with suboptimal access. Intervention studies often underenroll individuals from historically excluded groups. The Alliance to Advance Patient-Centered Cancer Care includes 6 grantees across the United States who implemented unique multicomponent, multilevel intervention programs with common goals of reducing disparities, increasing engagement, and improving the quality of care for targeted populations. The Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework informed the evaluation efforts across sites. Each Alliance site identified their intended populations, which included underrepresented minorities (eg, Black and Latinx persons), individuals who prefer a language other than English, and rural residents. We evaluated the demographic characteristics of participants to determine program reach. Between 2018 and 2020, a total of 2,390 of 5,309 potentially eligible participants were enrolled across the 6 sites. The proportion of enrolled individuals with selected characteristics included 38% (n=908) Black adults, 24% (n=574) Latinx adults, 19% (n=454) preferring a language other than English, and 30% (n=717) rural residents. The proportion of those enrolled who were the intended population was commensurate to the proportion with desired characteristics in those identified as potentially eligible. The grantees met or exceeded enrollments from their intended populations who have been underserved by quality cancer care into patient-centered intervention programs. Intentional application of recruitment/engagement strategies is needed to reach individuals from historically underserved communities.

Police Recruits’ Cognitive Engagement in a Racial Literacy Education Program: Does Racial Ideology Matter?

Studies suggest shorter term racial diversity education is ineffective in changing police officers’ attitudes and behaviors, partly due to strong emotional reactions and resistance to this type of content ( Schlosser, 2013 ; Zimny, 2015 ). In this investigation, we explored across two studies whether police recruits’ racial beliefs were related to their level of cognitive engagement in a racial literacy education program. Consistent with the research hypothesis, findings from Study 1 with 81 mostly White male police recruits suggested that recruits with higher color-blind racial beliefs (i.e., greater denial or minimization of institutional racism) as assessed in the first two weeks of the academy were less cognitively engaged in 10 hours of racial literacy education that they received in the training academy. In Study 2, we replicated and extended the results with a separate sample of 74 police recruits. In addition to completing a measure of color-blind racial beliefs at the beginning of their training, participants completed evaluations after each of the three education sessions offered over the course of the police academy. Findings indicated that the recruits’ level of color-blind racial beliefs at the beginning of police academy was associated with lower cognitive engagement in the education sessions. Limitations of the findings are discussed as well as the implications for future evaluation and racism education programming efforts.

Phase 2 of collaborative action around the implementation of virtual hearing aid care: Evaluation of a clinical practice guideline.

Following the onset of the COVID-19 pandemic, a clinical practice guideline (CPG) around virtual hearing aid practices was developed to fill a knowledge gap within the field of audiology. Details outlining the development and mobilization of this draft guideline were outlined as Phase 1 (described in a paired paper). This study describes Phase 2 of this project as part of the Knowledge-to-Action Framework, including an evaluation of the methodological quality of the guideline and the resulting tailored version of the document (v2.0). The Appraisal of Guidelines for Research and Evaluation II instrument was used to assess methodological quality and to guide revisions. Twenty-two clinicians, from a variety of clinical backgrounds, participated in the evaluation. Findings reported across six domains suggest high mean scores, ranging from 78% to 81%, in order of scope and purpose (highest rated), stakeholder involvement, rigour of development, applicability, clarity of presentation, and editorial independence. Specific recommendations made by in international co-creation team during the evaluation informed the final version of the CPG. Future development and evaluation efforts should aim to include greater representation from nontraditional practice contexts to strengthen global applicability.

Evaluating the learning outcomes of interfaith initiatives: a systematic literature review

ABSTRACT Interfaith initiatives are seen as promising sites for societal change and personal transformation. However, many questions about the actual outcomes of such initiatives have remained unanswered. Therefore, a systematic review was conducted, focusing on 1) the outcomes of interfaith initiatives and 2) the methods used to evaluate these outcomes. This review analyses 47 empirical studies on interfaith initiatives. The review of the initiatives’ outcomes indicates that interfaith initiatives can positively impact participants’ knowledge about other worldviews, communication skills and appreciation for other worldviews. At the same time, it can be challenging for interfaith initiatives to positively impact participants’ knowledge of socio-political challenges, awareness of power imbalance and privilege and behaviour as interfaith leaders. The analysis of the methods used to evaluate the impact of interfaith initiatives shows that most studies used qualitative methods. For studies using quantitative or mixed methods designs, a variety of measurement instruments were used, whereby there was little overlap or consensus regarding the concepts being used. The findings of this article will stimulate scholars to focus more on learning activities and evaluation methods that take societal learning outcomes into consideration. It emphasises the need to build on previous evaluation and research efforts.

Implementation of a School Mental Health Learning Collaborative Model to Support Cross-Sector Collaboration.

Facilitating success for students with behavioral health challenges requires effective collaboration among professionals from traditionally disparate systems (e.g., education, health, and mental health). The current investigation describes a case-study implementation of a school-based learning collaborative model and explores its effectiveness in promoting knowledge, skill, efficacy, and systems-related improvements in cross-sector collaboration. The learning collaborative (LC) was offered to school teams over the course of a year and consisted of a combination of didactic and experiential learning opportunities, guest speakers, district-specific improvement goals, peer learning and support, and individualized consultation support. Evaluation efforts included evidence demonstrating the efficacy of the LC, improvement in person-centered knowledge skills and competencies, and generation of concrete changes in school systems. Respondents consistently shared that the quality of the LC was high that the topics were highly useful for their day-to-day practice, and that they would recommend the LC to their colleagues and peers. In turn, this process fostered improvement in educators' knowledge, skills, and confidence, and generated systemic improvement in districts to support children with behavioral health needs and their families. Specific components of this model that best account for changes are discussed, along with implications for application and next steps.

Examining the Comprehensiveness of Amended Flavored Tobacco Product Sales Restrictions.

Flavored tobacco products contribute significantly to youth tobacco initiation and tobacco use disparities. In the last decade, 362 jurisdictions have enacted policies restricting the sale of flavored tobacco products; however, many policies are not fully comprehensive due to menthol and adult-only retailer exemptions. Although several of these restrictions have been amended since their original passage, to date, little is known about how amendments have affected policy comprehensiveness. To describe how amendments to flavored tobacco product sales restrictions affect policy comprehensiveness. We identified flavored tobacco product sales restrictions that had been amended at least once using an internal database of US state and local flavored tobacco product sales restrictions. To characterize policy comprehensiveness, we applied a 6-level flavored tobacco policy classification scheme-level 6 being most comprehensive-to amended restrictions. We conducted a descriptive analysis of each initial policy and its most recent amendment to identify changes in retailer, product, and flavor inclusions, as well as overall comprehensiveness. Comprehensiveness of amended flavored tobacco product sales restrictions. As of March 31, 2022, no states and 50 localities had amended their flavored tobacco product sales restriction. Amendments largely increased policy comprehensiveness; most laws prior to amendment were categorized as level 1 (n = 28, 56.0%), while after amendment, the plurality were categorized as level 6 (n = 25, 50.0%). Most commonly, amendments removed menthol exemptions (n ≥ 30, 60.0%) and adult-only retailer exemptions (n = 12, 24.0%). Several local flavored tobacco product sales restrictions have been amended. Nearly all amendments increased policy comprehensiveness, primarily by removing exemptions for menthol products and exemptions for adult-only retailers. While policy advocates remain focused on passing comprehensive policies at initial passage, amendments have served as a tool to strengthen existing sales restrictions. This study along with ongoing flavored tobacco product sales restriction surveillance can inform policy advocacy and evaluation efforts.

Elevating Women's Voices: The Illinois Maternal Health Digital Storytelling Project.

Numerous initiatives at the national, state, and local levels are focused on addressing the U.S. maternal health crisis. However, most efforts fail to center women's voices and personal perspectives about pregnancy experiences and complications. This article describes the Illinois Maternal Health Digital Storytelling Project conducted by the University of Illinois at Chicago, in partnership with StoryCenter, a storytelling nonprofit organization based in Berkeley, California. The project aimed to elevate voices (stories) to complement other sources of data on maternal outcomes. We relied on partners to help recruit storytellers with a self-identified perinatal-related challenge. We screened interested individuals using a trauma-informed approach and a social worker provided storytellers with support during the project. Two groups participated in this project, one in 2021 and another in 2022, culminating with 10 digital stories. Both instances were conducted virtually and included group and individual skills-based sessions where storytellers learned to create and edit their digital stories. The storytellers own their digital stories and provided consent to include them in ongoing dissemination efforts. In September 2021, a webpage was launched to house the stories, and this resource is being shared widely. Evaluation efforts of the webpage are ongoing to understand how these digital stories are being used to inform maternal health efforts. Digital stories add richness to ongoing maternal health prevention work and may serve as a powerful tool to elicit understanding among providers, practitioners, and constituents.

Exploratory Time-Series Analysis of Consecutive Case Series Data: A Quality Improvement and Adherence Study of a Behavior Analytic Service Provider.

Program evaluation is an essential practice for providers of behavior analytic services, as it helps providers understand the extent to which they are achieving their intended mission to the community they serve. A proposed method for conducting such evaluations, is through the use of a consecutive case series design, for which cases are sequentially gathered following the onset of a specific occurrence. Given the sequential nature in which data are collected within a consecutive case series, analytic techniques that adopt a time-series framework may be particularly advantageous. Although such methods are commonly used for program evaluation in medicine and economics, their application within the field of applied behavior analysis is largely absent. To serve as a model for providers undertaking evaluation efforts, I conducted a program evaluation of an outpatient severe behavior clinic, in which I employed quasi-experimental methods using an interrupted time-series analysis.

On the resistance of new lightweight block ciphers against differential cryptanalysis

Many recently proposed lightweight block ciphers lack security evaluation against generic cryptanalytic attacks such as differential cryptanalysis. In this paper, we contribute towards security evaluation efforts by investigating four lightweight Feistel-based block ciphers including SLIM, LBC-IoT, SCENERY, and LCB. SLIM claims resistance to differential cryptanalysis since, using a heuristic technique, its designers could only find a 7-round differential trail. Despite having no analysis of security against attacks such as differential cryptanalysis, the designers of LBC-IoT and LCB claimed that their ciphers are secure. Meanwhile, the designers of SCENERY claim that the best 11-round differential trail for the cipher has a probability of 2−66. To substantiate these claims, we propose attacks on all four ciphers based on differential cryptanalysis. We presented practical key recovery attacks on SLIM which can retrieve the final round key for up to 14 rounds with a time complexity of 232. LBC-IoT was found to be weaker against differential cryptanalysis despite sharing many similarities with SLIM, whereby a key recovery attack of up to 19 rounds is possible with time complexity 231. For SCENERY, we found a differential trail of up to 12 rounds with probability 2−60, which was used as the distinguisher for a 13-round key recovery attack. We also discovered that LCB's design lacks nonlinearity, allowing us to easily derive deterministic differential trails regardless of the number of rounds. This flaw allowed us to perform a trivial distinguishing attack using a single known ciphertext. By using a different S-box to address this flaw, LCB is now more resilient to differential cryptanalysis than SLIM and LBC-IoT when using the same number of rounds. Our paper presents new independent cryptanalysis results for these ciphers.

Analisis Manajemen Risiko Pengelolaan Limbah Medis dalam Upaya Pencegahan dan Pengendalian Covid-19 di RS MMC Jakarta

Hospitals are medical services that produce medical waste. The increasing number of positive cases of Covid-19 has resulted in an increase in the amount of medical waste. An assessment of the risks caused by the hazards is required. This study aims to analyze the risk management of medical waste in efforts to prevent and control Covid-19 at the MMC Jakarta Hospital. Using qualitative methods. Located at the MMC Hospital Jakarta in May-June 2022. Primary data collection comes from observations of waste management. Secondary data obtained from graphic documents. Risk identification was carried out from several problems by conducting interviews with the hospital. After that, an analysis is carried out based on the identification of the problem. MMC Hospital medical waste is treated in accordance with the designated waste designation based on applicable regulations. There are adequate policies and SOPs, human resources, funds, facilities and infrastructure. Sorting has been carried out between domestic waste and medical waste, with the exception of the Covid-19 treatment room. But still found disaggregation that is not appropriate. The volume of waste generated by the MMC Hospital from 2020-2021 has increased by 21.79% resulting in a 5x increase in spending. The regulations in force have not been implemented properly. Hospitals should increase monitoring and evaluation efforts in all aspects of managing Covid-19 medical solid waste. Conclusion There is a risk of increasing the volume of waste and increasing expenditure of funds in waste management. Research suggestions are to comply with the policies that have been made, adjust the budget based on needs evaluation, carry out guidance and supervision as evaluation material, conduct education and training related to waste management, pay attention to each component so that it is in line with hospital sanitation goals.

A commensurate industry approach to DEC A safety analysis

The new revision of the WENRA reference level 2014 (WENRA, 2014a) and the IAEA safety standard (International Atomic Energy Agency, 2016) require that accidents under design extension conditions without significant fuel degradation (DEC A) be identified and safety analysis be performed. To that aim, Tractebel has developed a practical method to concretely address the high level requirements of WENRA RL 2014 and IAEA SSR 2/1 and SSG2 guidelines. For the selection of DEC A sequences, a global methodology using PSA, internal and external hazard as well Combination of Events screening and characterization methods has been derived. For the analysis of DEC A, a graded approach has been derived. It aims at optimizing engineering effort for uncertainties treatment evaluation as well as decision making process regarding potential plant improvements resulting from the DEC A safety analyses. Some typical results of the applications are presented as example.

A Summary of the Recommendations to Increase Cultural Responsiveness in the Field of Applied Behavior Analysis

Cultural contingencies help shape the development of an individual’s behavioral repertoire. Researchers within field of behavior analysis have, within the past decade, reiterated the importance of connecting behavioral principles to a client's cultural practices. The incorporation of cultural preferences within behavioral services is paramount to the effectiveness and sustainability of those services. The present paper summarizes the guidance of researchers for practitioners beginning to develop this repertoire and apply it to their clinical practice. This paper begins by acknowledging that the foundation of the field of applied behavior analysis set the stage for understating how cultural considerations make the field applied. Different themes related to increasing the cultural responsiveness of the field are summarized by reviewing the supporting literature. The research is further summarized into guiding principles for practitioners as they begin to sharpen this skill. The paper ends with a call for scientist practitioners and researchers to capture data related to culturally responsive service provision to inspire efforts in empirical evaluation of the recommendations provided.