Perspective Of Evaluation Research Articles

DARPA's explainableAI(XAI) program: A retrospective

Summary of Defense Advanced Research Projects Agency's (DARPA) explainable artificial intelligence (XAI) program from the program managers' and evaluator's perspective.

Further Evidence on the Negativity Bias in Performance Evaluation: When Does the Evaluator's Perspective Matter?

ABSTRACT In a setting involving a firm using multiple performance measures, we experimentally examine the effect of an evaluator's perspective on the relative performance evaluations of two managers. The performance outcomes of the two managers are linearly equivalent, but one manager's performance includes an equivalent number of above target and below target outcomes (e.g., the mixed manager), while the other manager's performance only includes above target outcomes (e.g., the positive manager). In this setting, we provide new evidence on the negativity bias and whether the bias is moderated based on evaluators' role and the importance of the measures with negative outcomes. Participants are assigned to the role of the supervisor, the mixed manager, or the positive manager and asked to evaluate each manager's performance. We predict and find that participants in all three roles exhibit the negativity bias. In addition, we predict that the strength of the negativity bias exhibited by mixed or positive managers relative to the supervisor's negativity bias depends on whether the measures with negative outcomes are more or less important. As expected, we find that when negative outcomes involve less important measures, the relative performance evaluations of mixed (positive) managers are similar to (differ from) those of supervisors. In contrast, when negative outcomes involve more important measures, the relative performance evaluations of positive (mixed) managers are similar to (differ from) supervisors. Understanding whether and when managers' relative performance evaluations differ from their supervisors is important, in part, because conflicts and potentially dysfunctional behavior are likely to arise when their relative performance evaluations differ.

Critical Issues in Relocation Cases: A Custody Evaluator's Response to Parkinson and Cashmore (2015) and Thompson (2015)

Parkinson and Cashmore () and Thompson () have written comprehensive articles outlining suggested reforms to the family law system in relocation cases. This brief article, from a child custody evaluator's perspective as opposed to researchers' and legal scholars' perspectives, highlights areas of agreement, in hope of leading to increased consensus, as well as areas of disagreement, furthering the discussion and debate of critical issues in relocation matters. Rather than an either/or approach to relocation presumptions, this article will identify a both/and perspective on presumptions for these cases. It will also focus on suggestions for evaluators in an effort to help guide ways that evaluations can be most helpful to the court.

Evaluating the barriers to specific industrial energy efficiency measures: an exploratory study in small and medium-sized enterprises

Abstract Industrial energy efficiency has raised to the top of European energy policy agenda, and particular efforts should be placed to tackle the barriers to energy efficiency of small and non-energy intensive industrial users. To enhance the adoption rate of a specific industrial energy efficiency measure, it is crucial to properly evaluate the barriers. Without a proper level of detail – considering both the measure's specificity and the evaluation's perspective – company managers are hampered when making investment decisions, and policymakers are prevented from developing the most effective policies. The paper aims at highlighting that the value of a given intervention-dependent barrier may differ when evaluating it at the company level, by technology area, or with respect to a specific measure. To achieve this objective, an exploratory investigation of Italian small and medium-sized enterprises into the metalworking sector has been performed with respect to selected cross-cutting technologies. Although general barriers appear as most relevant at the company level, large differences appear when considering intervention-dependent barriers by technology area as well as at the energy efficiency measure level. The research presents innovative results, as the literature and policymakers have not accounted so far for how the value of barriers can effectively vary with respect to a single measure, but have only considered barriers at company level. Additionally, the paper highlights that differences can be appreciated when evaluating the intervention-dependent barriers according to some firm characteristics (i.e. size, production complexity, and innovativeness). The study concludes with suggestions for policymakers as well as industrial decision-makers.

Toward a Theory of Social Judgments of Organizations: The Case of Legitimacy, Reputation, and Status

The proposed theory extends research on cognitive and sociopolitical legitimacy, reputation, and status by advancing an evaluator's perspective on these concepts as forms of social judgment, each addressing a different evaluator's question about the organization. I describe how evaluators make their social judgments under conditions of bounded rationality and how cognitive and social factors influence this process. The proposed process model of social judgment formation highlights the complex and nondeterministic nature of this process.

Falling from a Great Height: Principles of Good Practice in Performance Measurement and the Perils of Top Down Determination of Performance Indicators

The author uses an evaluator's perspective and practical experience of implementing performance measurement systems in the public, private and voluntary sectors to devise ten principles of good practice in performance measurement: conceptualisation, stakeholder approach, clarity, balance, ownership, usefulness, accuracy, contextualisation, dynamism and value for money. These principles are not easily achieved and require adjustments that are difficult to implement in top down performance measurement systems. Locally determined performance measurement systems have a higher chance of meeting learning-based and practical purposes of performance measurement as opposed to political and symbolic purposes. Incorporating an evaluation element can moderate some of the potential weaknesses of a top down approach and take advantage of the scale of a national system to provide a powerful contextualised learning repository.

An External Evaluator's Perspective on the CEFP

As the external evaluator of the Collaborative Evaluation Fellows Project (CEFP), the author describes how the CEFP model is applied to the program's own evaluation and presents early findings, mostly based on the 14 studies conducted in the first year of implementation. Outcomes, benefits, and design and implementation issues are examined. Early findings are that all of the basic premises of the CEFP model have proven feasible. The CEFP has been well received within the American Cancer Society and universities alike. The roles of the CEFP evaluation facilitator, evaluation faculty, and evaluation fellow have been assumed by appropriately qualified persons. The American Cancer Society staff and volunteers generally regard CEFP evaluations as trustworthy and useful, and, in most cases, they actually use them. Evaluation fellows place high value on the CEFP as a learning experience, and a surprising number pursue related careers. What was most feared at the CEFP planning stage--conflicts between the practical needs of the American Cancer Society and the academic orientations of its partners--have arisen much less than expected, probably because of careful provisions in the CEFP program design to guard against this.

Evaluating Practice: The Dual Roles of Clinician and Evaluator

When single-subject design is the method used to evaluate clinical work, the same person simultaneously conducts counseling and evaluation. The author describes situations in which the roles of clinician and evaluator are compatible as well as circumstances in which difficulties in exercising the demands of both roles occur. The various phases of the treatment/evaluation process are followed, and suggestions are given regarding ways to resolve differences between the clinician's and the evaluator's perspectives.

Hawthorne Elementary School: The Evaluator's Perspective

In this article I present comparisons of 1994 and 1995 criterion-referenced achievement data from the state's TAAS annual spring testing program for Hawthorne and an aggregation of the results for all other elementary schools in the SAISD. TAAS results are reported in terms of the percentages of students meeting the passing standard at each grade level and in terms of scale scores or the Texas Learning Index (TLI). The TLI scale-score system was developed in 1994 to determine whether students were meeting TAAS standards in Grades 3 through 8 that were predictive of the passing standard at the high school exit level.

Understanding resistance to sex and race-based affirmative action: A review of research findings

The public discussion of affirmative action appears to be complicated by disagreements regarding definitions and by the lack of a theoretical framework from which to begin to understand this complex public policy. The present review attempts to synthesize the available research into a model from which resistance to affirmative action can be understood. Within the model, resistance to affirmative action policies is viewed as influenced by two main categories of factors—the evaluator's perspective and the context in which the policy is embedded. Based on these categories, specific hypotheses are constructed regarding resistance. In addition, recommendations for enhancing acceptance of affirmative action are made based on the model and previous research.

Certifying Sustainable Forestry: The Evaluator's Perspective

Certifying Sustainable Forestry: The Evaluator's Perspective

Speech Evaluation Differences as a Function of Perspective (Participant Versus Observer) and Presentational Medium

Using 40 naturally occurring conversations (20 same sex and 20 mixed sex), the purpose of this study was to examine differences among speech behavior-social evaluation relationships as a function of whether the evaluator's perspective was that of a participant or an observer and, if the latter, whether the presentational medium involved an audiovisual, audio-only, or transcriptual recording. Speech behaviors examined included conversants’ speech rates, turn durations, interruptive speakovers, and vocal back-channels and the degree to which they performed these speech behaviors similar to those of their partners. Social evaluative dimensions examined included communication satisfaction, communicative competence, sociointellectual status, aesthetic quality, and dynamism. Several findings were noteworthy. First, participants viewed their conversational partners most favorably and transcript readers viewed them least favorably. Second, there were few significant correlations between social evaluative judgments across different perceivers’ perspectives. Third, while the perceivers’ social judgments of conversants were often related to the conversants’ speech behaviors, there was little consistency across perceivers’ perspectives regarding which speech behaviors were predictive of these judgments. Specifically, the relationships between audiovisual observers’ and transcript readers’ evaluations of conversants and the conversants’ speech behaviors frequently differed from relationships between participants’ evaluations of partners and the partners’ speech behaviors.

Promises to keep: An evaluator's perspective on employee assistance programs

This paper presents a critical review of the rapidly expanding literature concerning employee assistance programs (EAPs). Several essential features of EAPs are identified and the assumptions underlying these components are examined. The latter section of this report offers some tentative suggestions for correcting this situation so that the EAP movement will no longer suffer from the lack of a firm database upon which to build more coherent and defensible theories and programs.