Prior Performance Information Research Articles

The impact of prior performance information on subsequent assessment: is there evidence of retaliation in an anonymous multisource assessment system?

Few studies have engaged in data-driven investigations of the presence, or frequency, of what could be considered retaliatory assessor behaviour in Multi-source Feedback (MSF) systems. In this study, authors explored how assessors scored others if, before assessing others, they received their own assessment score. The authors examined assessments from an established MSF system in which all clinical team members - medical students, interns, residents, fellows, and supervisors - anonymously assessed each other. The authors identified assessments in which an assessor (i.e., any team member providing a score to another) gave an aberrant score to another individual. An aberrant score was defined as one that was more than two standard deviations from the assessment receiver's average score. Assessors who gave aberrant scores were categorized according to whether their behaviour was preceded by: (1) receiving a score or not from another individual in the MSF system (2) whether the score they received was aberrant or not. The authors used a multivariable logistic regression model to investigate the association between the type of score received and the type of score given by that same individual. In total, 367 unique assessors provided 6091 scores on the performance of 484 unique individuals. Aberrant scores were identified in 250 forms (4.1%). The chances of giving an aberrant score were 2.3 times higher for those who had received a score, compared to those who had not (odds ratio 2.30, 95% CI:1.54-3.44, P < 0.001). Individuals who had received an aberrant score were also 2.17 times more likely to give an aberrant score to others compared to those who had received a non-aberrant score (2.17, 95% CI:1.39-3.39, P < 0.005) after adjusting for all other variables. This study documents an association between receiving scores within an anonymous multi-source feedback (MSF) system and providing aberrant scores to team members. These findings suggest care must be given to designing MSF systems to protect against potential downstream consequences of providing and receiving anonymous feedback.

Potential for Bias: Rater Cognition and the Influence of Prior Performance Information.

Lee, Seung Ho BMSc1; Shaw, Tammy MD2; Tran, Christopher MD, MMEd3; Humphrey-Murto, Susan MD, MEd4 Author Information

The Effects of Process Transparency and Prior Performance Information on Subjective Evaluation Decisions

The Effects of Process Transparency and Prior Performance Information on Subjective Evaluation Decisions

The Effect of Prior Performance Information on Leniency in Subjective Performance Evaluation

This paper examines whether supervisors’ leniency in subjective performance evaluation is influenced by the prior performance information of subordinates. While prior studies have used proxies for leniency based on aggregate objective performance level or median of subjective scales, I develop the proxy for leniency at the performance measure level based on prior performance level which is found to be more relevant to the current subjective performance score. I analyze archival performance evaluation data of multiple state-owned enterprises (SOEs) in Korea over multiple time periods which enables me to develop the new definition of leniency. Utilizing this data set, I empirically find that lenient rating persists over time, and that larger (smaller) amount of leniency is applied to low (high) prior performers and harshly (leniently) evaluated performers in the previous period. The results have important implications for understanding the incentives of raters which differ with respect to their previous performance information.

Are raters influenced by prior information about a learner? A review of assimilation and contrast effects in assessment.

Understanding which factors can impact rater judgments in assessments is important to ensure quality ratings. One such factor is whether prior performance information (PPI) about learners influences subsequent decision making. The information can be acquired directly, when the rater sees the same learner, or different learners over multiple performances, or indirectly, when the rater is provided with external information about the same learner prior to rating a performance (i.e., learner handover). The purpose of this narrative review was to summarize and highlight key concepts from multiple disciplines regarding the influence of PPI on subsequent ratings, discuss implications for assessment and provide a common conceptualization to inform research. Key findings include (a) assimilation (rater judgments are biased towards the PPI) occurs with indirect PPI and contrast (rater judgments are biased away from the PPI) with direct PPI; (b) negative PPI appears to have a greater effect than positive PPI; (c) when viewing multiple performances, context effects of indirect PPI appear to diminish over time; and (d) context effects may occur with any level of target performance. Furthermore, some raters are not susceptible to context effects, but it is unclear what factors are predictive. Rater expertise and training do not consistently reduce effects. Making raters more accountable, providing specific standards and reducing rater cognitive load may reduce context effects. Theoretical explanations for these findings will be discussed.

How biased are you? The effect of prior performance information on attending physician ratings and implications for learner handover.

Learner handover (LH), the process of sharing of information about learners between faculty supervisors, allows for longitudinal assessment fundamental in the competency-based education model. However, the potential to bias future assessments has been raised as a concern. The purpose of this study is to determine whether prior performance information such as LH influences the assessment of learners in the clinical context. Between December 2017 and June 2018, forty-two faculty members and final-year residents from the Department of Medicine at the University of Ottawa were assigned to one of three study groups through quasi-randomisation, taking into account gender, speciality and rater experience. In a counter-balanced design, each group received either positive, negative or no LH prior to watching six simulated learner-patient encounter videos. Participants rated each video using the mini-CEX and completed a questionnaire on the raters' general impressions of LH. A significant difference in the mean mini-CEX competency scale scores between the negative (M = 5.29) and positive (M = 5.97) LH groups (P < .001, d = 0.81) was noted. Similar findings were found for the single overall clinical competence ratings. In the post-study questionnaire, 22/28 (78%) of participants had correctly deduced the purpose of the study and 14/28 (50%) felt LH did not influence their assessment. LH influenced mini-CEX scores despite raters' awareness of the potential for bias. These results suggest that LH could influence a rater's performance assessment and careful consideration of the potential implications of LH is required.

The Influence of Prior Performance Information on Ratings of Current Performance and Implications for Learner Handover: A Scoping Review.

Learner handover (LH) is the sharing of information about trainees between faculty supervisors. This scoping review aimed to summarize key concepts across disciplines surrounding the influence of prior performance information (PPI) on current performance ratings and implications for LH in medical education. The authors used the Arksey and O'Malley framework to systematically select and summarize the literature. Cross-disciplinary searches were conducted in six databases in 2017-2018 for articles published after 1969. To represent PPI relevant to LH in medical education, eligible studies included within-subject indirect PPI for work-type performance and rating of an individual current performance. Quantitative and thematic analyses were conducted. Of 24,442 records identified through database searches and 807 through other searches, 23 articles containing 24 studies were included. Twenty-two studies (92%) reported an assimilation effect (current ratings were biased toward the direction of the PPI). Factors modifying the effect of PPI were observed, with larger effects for highly polarized PPI, negative (vs positive) PPI, and early (vs subsequent) performances. Specific standards, rater motivation, and certain rater characteristics mitigated context effects, whereas increased rater processing demands heightened them. Mixed effects were seen with nature of the performance and with rater expertise and training. PPI appears likely to influence ratings of current performance, and an assimilation effect is seen with indirect PPI. Whether these findings generalize to medical education is unknown, but they should be considered by educators wanting to implement LH. Future studies should explore PPI in medical education contexts and real-world settings.

Inter-Trial Correlations in Predictive-Saccade Endpoints: Fractal Scaling Reflects Differential Control along Task-Relevant and Orthogonal Directions.

Saccades exhibit variation in performance from one trial to the next, even when paced at a constant rate by targets at two fixed locations. We previously showed that amplitude fluctuations in consecutive predictive saccades have fractal structure: the spectrum of the sequence of consecutive amplitudes has a power-law (f −α) form, indicative of inter-trial correlations that reflect the storage of prior performance information to guide the planning of subsequent movements. More gradual decay of these inter-trial correlations coincides with a larger magnitude of spectral slope α, and indicates stronger information storage over longer times. We have previously demonstrated that larger decay exponents (α) are associated with faster adaptation in a saccadic double-step task. Here, we extend this line of investigation to predictive saccade endpoints (i.e., movement errors). Subjects made predictive, paced saccades between two fixed targets along a horizontal or vertical axis. Endpoint fluctuations both along (on-axis) and orthogonal to (off-axis) the direction of target motion were examined for correlations and fractal structure. Endpoints in the direction of target motion had little or no correlation or power-law scaling, suggesting that successive movements were uncorrelated (white noise). In the orthogonal direction, however, the sequence of endpoints did exhibit inter-trial correlations and scaling. In contrast, in our previous work the scaling of saccade amplitudes is strong along the target direction. This may reflect the fact that while saccade amplitudes are neurally programmed, endpoints are not directly controlled but instead serve as a source of error feedback. Hence, the lack of correlations in on-axis endpoint errors suggests that maximum information has been extracted from previous movement errors to plan subsequent movement amplitudes. In contrast, correlations in the off-axis component indicate that useful information still remains in this error (residual) sequence, suggesting that saccades are less tightly controlled along the orthogonal direction.

Probabilistic Forecasting of Project Duration Using Bayesian Inference and the Beta Distribution

Reliable forecasting is instrumental in successful project management. In order to ensure the successful completion of a project, the project manager constantly monitors actual performance and updates the current predictions of project duration and cost at completion. This study introduces a new probabilistic forecasting method for schedule performance control and risk management of on-going projects. The Bayesian betaS-curve method (BBM) is based on Bayesian inference and the beta distribution. The BBM provides confidence bounds on predictions, which can be used to determine the range of potential outcomes and the probability of success. Furthermore, it can be applied from the outset of a project by integrating prior performance information (i.e., the original estimate of project duration) with observations of new actual performance. A comparative study reveals that the BBM provides, early in the project, much more accurate forecasts than the earned value method or the earned schedule method and as accurate forecasts as the critical path method without analyzing activity-level technical data.

Sandbagging as a Self-Presentational Strategy: Claiming to be Less than You Are

Sandbagging is a self-presentational strategy involving the false prediction or feigned demonstration of inability. Three studies explored the individual differences and situational variables influencing sandbagging behavior. Study 1 demonstrated that the newly created Sandbagging Scale possessed adequate reliability, convergent validity, and discriminant validity. The next two studies examined the social determinants of sandbagging. In Study 2, greater performance pressure led participants with higher scores on the Sandbagging Scale to predict worse performance on an upcoming task. In Study 3, participants with higher scores on the Sandbagging Scale were more likely to predict worse performance on an upcoming task to someone who was ostensibly evaluating their performance, but only when the individual did not possess prior performance information. Sandbagging is discussed as a self-presentational strategy used to reduce performance pressure and provide a low baseline against which subsequent performance can be compared.

Costly Search and Mutual Fund Flows

This paper studies the flows of funds into and out of equity mutual funds. Consumers base their fund purchase decisions on prior performance information, but do so asymmetrically, investing disproportionately more in funds that performed very well the prior period. Search costs seem to be an important determinant of fund flows. High performance appears to be most salient for funds that exert higher marketing effort, as measured by higher fees. Flows are directly related to the size of the fund's complex as well as the current media attention received by the fund, which lower consumers' search costs.

The Influence of Indirect Knowledge of Previous Performance on Ratings of Present Performance: The Effects of Job Familiarity and Rater Training

In a 2 × 2 × 2 crossed factorial design, trained or untrained subjects viewed a videotape and evaluated performance on either a familiar (college lecturer) or unfamiliar (salesperson) job. Prior to viewing the videotape, some subjects reviewed positive information about the ratee's prior performance, whereas other subjects did not review any prior performance information. To determine whether assimilation or contrast effects occurred, we compared ratings provided by subjects who reviewed positive information about prior performance with ratings provided by subjects who did not review any prior performance information. A three-way interaction was obtained. Ratings of performance on the familiar job by untrained or trained subjects revealed only a small assimilation effect. However, when rating performance on the unfamiliar job, a large assimilation effect was observed among untrained subjects, whereas a large contrast effect was observed among trained subjects. The results indicate that rater error training may reverse, rather than reduce or eliminate, rating errors that arise from knowledge of a ratee's prior performance. Implications for further understanding and reducing assimilation and contrast effects are discussed.

Prospect Theory Predictions When Escalation Is Not the Only Chance to Recover Sunk Costs

Two experiments tested the effect of risk of alternative investment opportunities on decision behavior in an escalation context. In the first experiment (n = 170), the risk of the reinvestment option, which had been unsuccessful and incurred a sunk cost, was held equal to that of an alternative investment project. Responsibility for the previous decision was manipulated between these subjects. Subjects were required to choose between the reinvestment option and the alternative. Responsible subjects demonstrated classic reinvestment (escalation) tendencies, and nonresponsible subjects exhibited a significant tendency to avoid reinvestment. Subjects in the second experiment (n = 195) completed the same decision task and were also assigned to high and low responsibility conditions. In this experiment the risk of investing further in a previously chosen project, relative to the risk of the alternative project, was manipulated. Those subjects who were responsible for the first decision (a) demonstrated no preference for or against continued pursuit of the initial project (i.e., there was no escalation tendency) and (b) preferred the less risky of the second investments, regardless of whether it was or was not the initially chosen project. Nonresponsible subjects showed no risk preference or proclivity for reinvestment. The results of these experiments suggest that salient information concerning relative risk dominates the effect of prior performance information when alternative investments are considered. We discuss the implications of these findings for decision theory and for methodology in commitment escalation research.

Processing Demands and the Effects of Prior Impressions on Subsequent Judgments: Clarifying the Assimilation/Contrast Debate

The present research had two purposes. First, to investigate the effects of observational demands in relation to the impact that prior impressions have on the encoding and storage of subsequent performance. Second, the extent to which these encoded and stored behaviors are used to modify existing impressions. Specifically, level of observational processing demands and ratee prior performance information were manipulated in a 3 × 3 factorial design. Results indicated that prior impressions either had no effect or biased processing toward impression-inconsistent behaviors and that as observational demands increased, judgments were biased toward prior impressions. These results were discussed both in relation to the debate over assimilation and contrast effects and to the more basic issue of modifying existing impressions.

The Influence of Rater Motivation on Assimilation Effects and Accuracy in Performance Ratings

Despite extensive study of ratings and performance appraisal, research that addresses the influence of rater motivation on rating accuracy is generally lacking. The present study investigated the effects of motivation on rater accuracy by offering incentives to be accurate to some raters. Each rater observed and evaluated five ratees on three dimensions where the true scores were operationally defined. Some raters were informed that the ratees had received above (or below) average ratings in their prior positions, whereas other subjects were given no prior performance information. Bias and poor elevation accuracy resulting from knowledge of ratees′ prior performance levels were eliminated when raters were offered incentives. Also, two additional measures of accuracy (differential elevation and differential accuracy) improved when incentives were offered.

Effect of prior performance information on ratings of present performance: Contrast versus assimilation revisited.

Effect of prior performance information on ratings of present performance: Contrast versus assimilation revisited.

Effect of prior performance information on ratings of present performance: Contrast versus assimilation revisited.

Effect of prior performance information on ratings of present performance: Contrast versus assimilation revisited.

Task Performance Information and Leader Participation Behavior: Effect on Leader-Subordinate Interaction, Frustration, and Future Productivity

Attribution theory research has shown that leader attributions of subordinate prior performance information influence leader-subordi nate interaction and problem solving. However, the effect of perfor mance information attributions in combination with levels of leader participation behavior on leader-subordinate interaction has received little attention. Leaders were placed randomly into high or low subordinate performance conditions, and the extent of leader partic ipation behavior was measured after the discussion. Two role-play problems representing different work contexts were utilized. Results demonstrated some support for previous research regarding leader attributions of prior performance and substantial support for the effect of leader participation behavior on problem-solving effective ness. Influences of the organizational context are discussed.