Program Evaluation Research Research Articles

Cross-cultural adaptation of the Patient-Doctor Relationship Questionnaire (PDRQ-9) in Brazil.

ABSTRACTOBJECTIVE To describe the process of cross-cultural adaptation of the Patient-Doctor Relationship Questionnaire (PDRQ-9), as well as compare the agreement between two different types of application.METHODS This is a cross-sectional study with 133 adult users of a Primary Health Service in Porto Alegre, State of Rio Grande do Sul, Brazil. The PDRQ-9 was answered by the participants as a self-administered questionnaire and in an interview. The instrument was also validated by interview, using data from 628 participants of the Mais Médicos Program Evaluation Research, which is a cross-sectional study with a systematic sample of Primary Care Services in all regions of Brazil. We evaluated the semantic, conceptual, and item equivalence, as well as factor analysis and reliability.RESULTS All items presented factor loading > 0.5 in the different methods of application and populations in the factor analysis. We found Cronbach’s alpha of 0.94 in the self-administered method. We found Cronbach’s alpha of 0.95 and 0.94 in the two different samples in the interview application. The use of PDRQ-9 with an interview or self-administered was considered equivalent.CONCLUSIONS The cross-cultural adaptation of the PDRQ-9 in Brazil replicated the factorial structure found in the original study, with high internal consistency. The instrument can be used as a new dimension in the evaluation of the quality of health care in clinical research, in the evaluation of services and public health, in health management, and in professional training. Further studies can evaluate other properties of the instrument, as well as its behavior in different populations and contexts.

An Evaluation of Halal Tourism Program in East Lombok Regency Using Kirkpatrick’s Model

This research was aimed to describe an evaluation of Halal Tourism program in East Lombok Regency, Nusa Tenggara Barat Province, Indonesia. This study was a program evaluation research which employed Kirkpatrick’s evaluation model (Reaction, Learning, Behavior, Result) and descriptive qualitative approach. Instruments of data collection were interview guideline, observation guideline, document review, and questionnaire. The data was analysis by using Miles and Huberman’s model which consisted of data display, data reduction, and conclusion drawing/verification. The result of this research showed that the reaction of Government Tourism Office and local community in East Lombok Regency towards Halal Tourism program was at moderate category (75.46%); community’s reaction towards the learning and carrying capacity of Halal tourism program was at moderate category (75.17 %); community’s knowledge of Halal Tourism scored 82.93 %; management system, both operational and administrative, scored 70.62 %; and Community’s reaction towards the aspects of Halal Tourism program scored 79.03%. The effectiveness of Halal Tourism program according to community’s reaction towards facility aspects scored 65.52 or was at moderate category. Moreover, the result from questionnaire revealed that 91.67% of participants thought that the knowledge they obtained in Halal Tourism program was less practical.

Evaluation Program on the Implementation of Industrial Apprenticeship (Prakerin) in Electrical Engineering

This research aims to find out the evaluation program of the Industrial apprenticeship (Prakerin) in electrical engineering. This research includes on four variables of CIPP. (1). Context (a). programme planning (b). design. (2). Input (a). readiness of students (b). performance of vocational education teachers (c). Facilities and infrastructure, (3). process (a). performance students (b). performance mentors, (4). Product (a). readiness of student work. This research is a type of program evaluation research with Stake model approach. Data collection methods used are questionnaires with closed questions and frequently asked questions.

Estimating the Associations between SNAP and Food Insecurity, Obesity, and Food Purchases with Imperfect Administrative Measures of Participation

Administrative data are considered the “gold standard” when measuring program participation, but little evidence exists on the potential problems with administrative records or their implications for econometric estimates. We explore issues with administrative data using the FoodAPS, a unique dataset that contains two different administrative measures of Supplemental Nutrition Assistance Program (SNAP) participation as well as a survey-based measure. We first document substantial ambiguity in the two administrative participation variables and show that they disagree with each other almost as often as they disagree with self-reported participation. Estimated participation and misreporting rates can be meaningfully sensitive to choices made to resolve this ambiguity and disagreement. We then document similar sensitivity in regression estimates of the associations between SNAP and food insecurity, obesity, and the Healthy Eating Index. These results serve as a cautionary tale about uncritically relying on linked administrative records when conducting program evaluation research.

Data Sharing in the Federal Statistical System: Impediments and Possibilities

While federal agencies have engaged in data sharing for decades, current systems of exchange must be transformed to meet the growing needs of analysts in public policy, program evaluation, and basic research. Primary barriers to intra-agency data sharing include both perceived legal barriers and actual financial barriers. To address these problems, the potential path forward could include mandating data sharing for key sources, increasing transparency, and improving efficiency by making it easier for agencies to share data (including developing standards, templates, and incentives). We argue that an agency within the federal statistical system should serve as an intermediary for data integration across the federal statistical system and that the U.S. Census Bureau, with its experience and uniquely broad authority to seek data from any public or private entity for statistical purposes, is well positioned to fill this role.

Insecurity, distress and mental health: experimental and randomized controlled trials of a psychosocial intervention for youth affected by the Syrian crisis.

BackgroundStrengthening the evidence base for humanitarian interventions that provide psychosocial support to war‐affected youth is a key priority. We tested the impacts of an 8‐week programme of structured activities informed by a profound stress attunement (PSA) framework (Advancing Adolescents), delivered in group‐format to 12–18 year‐olds in communities heavily affected by the Syrian crisis. We included both Syrian refugee and Jordanian youth.MethodsWe followed an experimental design, comparing treatment youth and wait‐list controls over two programme implementation cycles, randomizing to study arm in cycle 2 (ClinicalTrials.gov ID: NCT03012451). We measured insecurity, distress, mental health difficulties, prosocial behaviour and post‐traumatic stress symptoms at three time‐points: baseline (n = 817 youth; 55% Syrian, 43% female), postintervention (n = 463; 54% Syrian, 47% female), and follow‐up (n = 212, 58% Syrian, 43% female). Regression models assessed: prospective intervention impacts, adjusting for baseline scores, trauma exposure, age, and gender; differential impacts across levels of trauma exposure and activity‐based modality; and sustained recovery 1 year later. We analysed cycle‐specific and cycle‐pooled data for youth exclusively engaged in Advancing Adolescents and for the intent‐to‐treat sample.ResultsWe found medium to small effect sizes for all psychosocial outcomes, namely Human Insecurity (β = −7.04 (95% CI: −10.90, −3.17), Cohen's d = −0.4), Human Distress (β = −5.78 (−9.02, −2.54), d = −0.3), and Perceived Stress (β = −1.92 (−3.05, −0.79), d = −0.3); and two secondary mental health outcomes (AYMH: β = −3.35 (−4.68, −2.02), d = −0.4; SDQ: β = −1.46 (−2.42, −0.50), d = −0.2). We found no programme impacts for prosocial behaviour or post‐traumatic stress reactions. Beneficial impacts were stronger for youth with exposure to four trauma events or more. While symptoms alleviated for both intervention and control groups over time, there were sustained effects of the intervention on Human Insecurity.ConclusionsFindings strengthen the evidence base for mental health and psychosocial programming for a generation affected by conflict and forced displacement. We discuss implications for programme implementation and evaluation research.

EVALUASI PROGRAM ROAD SAFETY PARTNERSHIP ACTION POLICE GOES TO SCHOOL DI SMA NEGERI 6 DAN SMK PEMBANGUNAN KOTA BOGOR

The purpose of this study is to know the Road Safety Partnership Action Program Police Goes To School at SMAN 6 and SMK Pembangunan in Bogor City.The theory used in this research is by using policy evaluation from William N. Dunn which stating that to measure the success of an evaluation of the program can be measured through six variables: effectiveness, efficiency, adequacy, equity, responsiveness, and appropriateness. The method used is descriptive analysis using quantitative approach. Data analysis technique used is by using the calculation of Weight Mean Score (WMS). Data collection techniques used are literature studies and field studies. The study population are 1,857 students from two schools. While the sampling technique used is proportionate stratified random sampling (stratified proportional). The samples using Yamane formula thus obtained sample are 180 respondents.The results of Program Evaluation Research Road Safety Partnership Action Police Goes To School in SMA 6 and SMK Pembangunan Bogor City obtained a score of 3.64 (in scale of 5) which, according to the interpretation of the criteria that are in “good” categories, although it is still the number of violations that occurred among students. This happens due to the lack of awareness of students in traffic, do not care about the safety and ignore the socialization conducted by the police about the ethics of a good drive to support traffic safety and reduce the number of traffic violations or accidents. Efforts are being made to address the problem is by imposing preventive methods (prevention) and applying repressive methods (in applying force). Keywords: Evaluation, Program, Violation

EVALUATION OF PROGRAM BORDER MANAGEMENT TO KEEP SOVEREIGN NATION THE NORTH KALIMANTAN (Regional leadership Based on Transformational Leadership and lokal wisdom)

This research aimed to evaluate on five (5) steps;: (1) design of Border Management, (2) installation for Border Management,(3) process of Border Management, (4) product of Border Management, and cost / benefit for Border Management, for keep sovereignNation on North Kalimantan, This research is the programme evaluation research use Discrepancy Evaluation Model (DEM), data collection techniques are using; interview, observation, documentation and questionnaire for analyzer discussion support. Analysis of the data use Raph Tylor model as follows data reduction, data display and verification as conclusion.Results of this research include: (1) draft design, (2) border management plan, (3) compile of policy strategy with the help of stakeholders and HR agencies (4) the process of planning, organizing, staffing, directing, coordinating, reporting and budgeting for border management, and (5)lack of budget management.Besides is non-functioning of transformational leadership and effective local wisdom on North Kalimantan Keywords: Program Evaluation, Border Management, Regional Leadership, Transformational Leadership, Local Wisdom, Discrepancy Evaluation Model (DEM)

EVALUATION OF PROGRAM BORDER MANAGEMENT TO KEEP SOVEREIGN NATION THE NORTH KALIMANTAN (Regional leadership Based on Transformational Leadership and lokal wisdom)

This research aimed to evaluate on five (5) steps;: (1) design of Border Management, (2) installation for Border Management,(3) process of Border Management, (4) product of Border Management, and cost / benefit for Border Management, for keep sovereignNation on North Kalimantan, This research is the programme evaluation research use Discrepancy Evaluation Model (DEM), data collection techniques are using; interview, observation, documentation and questionnaire for analyzer discussion support. Analysis of the data use Raph Tylor model as follows data reduction, data display and verification as conclusion.Results of this research include: (1) draft design, (2) border management plan, (3) compile of policy strategy with the help of stakeholders and HR agencies (4) the process of planning, organizing, staffing, directing, coordinating, reporting and budgeting for border management, and (5)lack of budget management.Besides is non-functioning of transformational leadership and effective local wisdom on North Kalimantan Keywords: Program Evaluation, Border Management, Regional Leadership, Transformational Leadership, Local Wisdom, Discrepancy Evaluation Model (DEM)

Tuberculosis control program in the municipal context: performance evaluation.

ABSTRACTOBJECTIVE The objective of this study is to evaluate the performance of the Tuberculosis Control Program in municipalities of the State of São Paulo.METHODS This is a program evaluation research, with ecological design, which uses three non-hierarchical groups of the municipalities of the State of São Paulo according to their performance in relation to operational indicators. We have selected 195 municipalities with at least five new cases of tuberculosis notified in the Notification System of the State of São Paulo and with 20,000 inhabitants or more in 2010. The multiple correspondence analysis was used to identify the association between the groups of different performances, the epidemiological and demographic characteristics, and the characteristics of the health systems of the municipalities.RESULTS The group with the worst performance showed the highest rates of abandonment (average [avg] = 10.4, standard deviation [sd] = 9.4) and the lowest rates of supervision of Directly Observed Treatment (avg = 6.1, sd = 12.9), and it was associated with low incidence of tuberculosis, high tuberculosis and HIV, small population, high coverage of the Family Health Strategy/Program of Community Health Agents, and being located on the countryside. The group with the best performance presented the highest cure rate (avg = 83.7, sd = 10.5) and the highest rate of cases in Directly Observed Treatment (avg = 83.0, sd = 12.7); the group of regular performance showed regular results for outcome (avg cure = 79.8, sd = 13.2; abandonment avg = 9.5, sd = 8.3) and supervision of the Directly Observed Treatment (avg = 42.8, sd = 18.8). Large population, low coverage of the Family Health Strategy/Program of Community Health Agents, high incidence of tuberculosis and AIDS, and being located on the coast and in metropolitan areas were associated with these groups.CONCLUSIONS The findings highlight the importance of the Directly Observed Treatment in relation to the outcome for treatment and raise reflections on the structural and managerial capacity of municipalities in the implementation of the Tuberculosis Control Program.

Disassembly Program Evaluation Research Based on Fuzzy AHP, Grey Correlation and TOPSIS Combined Approach

Disassembly Program Evaluation Research Based on Fuzzy AHP, Grey Correlation and TOPSIS Combined Approach

A collaborative higher education initiative for leadership development: Lessons for knowledge sharing

The higher education sector needs to prepare youth for the fast changing innovative focused economy. Visionary leadership is needed to facilitate this knowledge transfer for novelty creation. A collaborative initiative could facilitate leadership development and foster knowledge sharing in this context. The initiative with three stakeholder groups started with an Interactive Qualitative Analysis exercise to identify needs and formulate a collaboration charter to ensure reciprocal benefits. A programme evaluation research design was applied three years after implementation using a logic-model framework. Process evaluation entailed collecting data from 103 questionnaires and 11 semi-structured interviews. Based on interpretation of data according to an integrated framework for managing knowledge across boundaries, we identified three areas for further development: contextual knowledge, collaborative partnerships and common vision. Working collaboratively across institutional and knowledge boundaries towards a common vision could enhance leadership development and contribute to longer-term transformation and sustainable change in complex environments.

What Can We Learn From A Doubly Randomized Preference Trial?—An Instrumental Variables Perspective

AbstractThe doubly randomized preference trial (DRPT) is a randomized experimental design with three arms: a treatment arm, a control arm, and a preference arm. The design has useful properties that have gone unnoticed in the applied and methodological literatures. This paper shows how to interpret the DRPT design using an instrumental variables (IV) framework. The IV framework reveals that the DRPT separately identifies three different treatment effect parameters: the Average Treatment Effect (ATE), the Average Treatment Effect on the Treated (ATT), and the Average Treatment Effect on the Untreated (ATU). The ATE, ATT, and ATU parameters are important for program evaluation research because in realistic settings many social programs are optional rather than mandatory and some people who are eligible for a program choose not to participate. Most of the paper is concerned with the interpretation of the research design. To make the ideas concrete, the final section provides an empirical example using data from an existing DRPT study.

A systematic literature review of the physical and psychosocial correlates of Special Olympics participation among individuals with intellectual disability.

Special Olympics (SO) is commonly cited to play an important role in the lives of individuals with intellectual disabilities (ID). The purpose of the current review was to (a) synthesise key findings regarding the physical, psychological/emotional, social and/or intellectual/cognitive correlates of SO participation for individuals with ID and (b) highlight limitations in the extant research as well as directions for future research. A systematic review of electronic databases was undertaken. A total of 46 articles were confirmed to meet study criteria. Quality assessments of included studies were conducted using checklists from the Scottish Intercollegiate Guidelines Network methodology checklists (SIGN 50; SIGN 2008). There was a larger amount of support for physical, psychological/emotional and social outcomes as compared with cognitive/intellectual outcomes; however, many studies were confounded by measurement difficulties, sampling procedures and a lack of replicable methods, which hinder generalisation of results. This review highlights the need for a continued critical focus on SO programme evaluation research with more rigorous and replicable methods.

Meta-Analysis and Program Outcome Evaluation

Meta-analysis is a technique for statistically representing and analyzing the findings from a set of empirical research studies. In application to program evaluation research, it provides a means for systematically synthesizing knowledge about the characteristic and outcomes of effective programs. Six lessons learned from meta-analysis of evaluation research illustrate the application and findings of this approach: (1) many social programs are more effective than generally realized; (2) individual evaluations can easily produce erroneous results; (3) the methods used in an evaluation play a large role in the program effects found in the evaluation; (4) program effectiveness is a function of identifiable program characteristics; (5) there is much room for program improvement; and (6) the most credible evidence about program effects comes through integration of multiple evaluation studies.

A Jack of All Trades But a Master of None? Evaluating the Performance of the Level of Service Inventory–Revised (LSI-R) in the Assessment of Risk and Need

ABSTRACTWe evaluate the performance of the LSI-R in assessing risk and need among 26,001 offenders released from Minnesota prisons between 2003 and 2011. Using multiple performance metrics, the results reveal the LSI-R incorrectly classified 41 percent of the offenders, and its area under the curve (AUC) was 0.628. Drawing on the extensive program evaluation research conducted on Minnesota prisoners over the last decade, we evaluated the LSI-R’s assessment of need by examining the relationship between LSI-R domain scores, participation in effective interventions, and recidivism outcomes. The findings provide, at best, moderate support for the LSI-R’s performance in assessing need. Interventions targeting the education/employment, alcohol/drug, and emotional/personal domains generally yielded better recidivism outcomes among offenders with higher scores in these domains.

Evaluating Public Health Interventions: 1. Examples, Definitions, and a Personal Note.

In the first contribution to a new section in AJPH that will address critical methodological issues in evaluations of public health interventions, I will discuss topics in study design and analysis, covering the most innovative emerging methodologies and providing an overview of best practices. The methods considered are motivated by public health evaluations, both domestic and global. In this first contribution, I also define implementation science, program evaluation, impact evaluation, and cost-effectiveness research, disciplines that have tremendous methodological and substantive overlap with evaluation of public health interventions--the focus of this section.

Consumer Information and Treatment Resources for Posttraumatic Stress Disorder

In the context of multiple treatment options for posttraumatic stress disorder (PTSD) and a large, growing need for consumer information regarding accessible and effective treatments, this article identifies and reviews available information and treatment resources. Multiple search strategies identified a suite of information sources, including meta-analyses and systematic reviews of PTSD treatments, the program evaluation and implementation literature, the economics literature, Internet sites, and other resources for veteran and civilian consumers. Resources were evaluated with regard to their target audiences, depth and breadth of treatment options covered, nature of the information provided, and accessibility to consumers. A large body of research covers the various treatments and sets of treatment guidelines for PTSD. Despite the extensive scientific information targeted at providers and researchers, the quality, accessibility, and usability of the published research varies widely. The Veterans Health Administration provides the most extensive information on various treatment options and where to obtain treatment within that system. Publicly available websites provide information on multiple treatment options, but information to help nonveterans navigate treatment choices is limited. Published reports of PTSD program-evaluation and implementation studies are sparse. Information on PTSD treatment options available to consumers can be overwhelming and confusing, which places an unnecessary burden on an already vulnerable group of patients and their families. Exacerbating the situation is the shortage of program-evaluation and implementation research. The dearth of centralized and accessible information related to nonveteran PTSD patient groups needs to be addressed.

Social norms theory and concussion education.

Secondary prevention of harm from sport-related concussion is contingent on immediate removal from play post-injury. To-date, educational efforts to reduce the prevalent risk behavior of continued play while symptomatic have been largely ineffective. Social norms theory may hold promise as a foundation for more effective concussion education aimed at increasing concussion reporting. The primary objective of this study was to assess whether perceived team concussion reporting norms would be less supportive of an individual's safe concussion symptom reporting behavior than objective team norms. Participants were 328 male and female US collegiate athletes. Written surveys were completed in person during the spring of 2014. Among both male and female athletes, team concussion reporting norms were significantly misperceived, with athletes tending to think that they themselves have safer attitudes about concussion reporting than their teammates. Perceived norms were associated with symptom reporting intention, independent of the team's objective reporting norm. A social norms approach to concussion education, in which misperceived group norms are corrected and shifted in the direction of safety, is an important avenue for program development and evaluation research aimed at the secondary prevention of harm from concussion. Implications for the design of this type of educational programming are discussed.

Evaluating the Impact of National Educational Policy to Reduce Retention and Increase Achievement in Compulsory Education

Despite the accumulated evidence that retention is an ineffective and potentially harmful remedial strategy, several countries struggle with high levels of retention in compulsory schooling. This article provides evidence of the impact of the Portuguese national educational policy “Programa Mais Sucesso Escolar” (PMSE) using class size, class composition, and differentiated instruction to reduce student retention and increase achievement. We found that PMSE significantly reduced retention, but had mixed effects on educational achievement, pointing to evidence of grade inflation on teacher-determined measures of achievement. In the discussion, we analyze the implications of these results for policy, practice, and program evaluation research.