Supervisor-rated Performance Research Articles

U.S. Law Enforcement Officers’ Stress, Job Satisfaction, Job Performance, and Resilience: A National Sample

The current study examined reports of perceived stress, job satisfaction, and job performance ratings in a longitudinal study of 684 officers participating in the Officer Safety and Wellness (OSAW) Initiative. Structural equation models were estimated to examine direct effects and, in subsequent analyses, the moderating effects of officer resilience and agency wellness programming on both the stress-job satisfaction association and the job satisfaction-job performance association. Surveys were administered annually, with job performance assessed both in terms of a self-rating and a self-report of supervisory rating at each officer’s last performance review. Officers’ stress (wave 1) was negatively associated with job satisfaction (wave 2), which in turn was positively associated with supervisory ratings of job performance (wave 3). These associations remained significant among officers reporting low to moderate baseline resilience but the association between job satisfaction and performance dissipated among officers with high resilience. Stress was negatively related to job satisfaction for officers who had easy access to agency-based wellness programs, whether they had concerns about stigma or used the programs, or not. The association between stress and job performance varied according to program access, use, and concerns about stigma associated with use. Administrators and policymakers striving to retain a high-performance police workforce may consider these results in recruiting as well as academy and in-service wellness training and program decisions.

Meta-analytical estimates of interrater reliability for direct supervisor performance ratings: Optimism under optimal measurement designs.

Performance appraisal (PA) is used for various organizational purposes and is vital to human resources practices. Despite this, current estimates of PA reliability are low, leading to decades of criticism regarding the use of PA in organizational contexts. In this article, we argue that current meta-analytical interrater reliability (IRR) coefficients are underestimates and do not reflect the reliability of interest to most practitioners and researchers-the reliability of an employee's direct supervisor. To establish the reliability of direct supervisor ratings, those making PA ratings must directly supervise employee job performance instead of nonparallel rater designs (e.g., direct supervisor ratings correlated with ratings from a more senior leader). The current meta-analysis identified 22 independent samples that met this more restrictive study inclusion criterion, finding an average observed IRR of .65. We also report reliability estimates for several important contextual moderators, including whether ratings were completed in operational settings (.60) or for research purposes (.67). In sum, we argue that this study's meta-analytical IRR estimates are the best available estimates of direct supervisor reliability and should be used to guide future research and practice. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

An updated meta-analysis of the interrater reliability of supervisory performance ratings.

Given the centrality of the job performance construct to organizational researchers, it is critical to understand the reliability of the most common way it is operationalized in the literature. To this end, we conducted an updated meta-analysis on the interrater reliability of supervisory ratings of job performance (k = 132 independent samples) using a new meta-analytic procedure (i.e., the Morris estimator), which includes both within- and between-study variance in the calculation of study weights. An important benefit of this approach is that it prevents large-sample studies from dominating the results. In this investigation, we also examined different factors that may affect interrater reliability, including job complexity, managerial level, rating purpose, performance measure, and rater perspective. We found a higher interrater reliability estimate (r = .65) compared to previous meta-analyses on the topic, and our results converged with an important, but often neglected, finding from a previous meta-analysis by Conway and Huffcutt (1997), such that interrater reliability varies meaningfully by job type (r = .57 for managerial positions vs. r = .68 for nonmanagerial positions). Given this finding, we advise against the use of an overall grand mean of interrater reliability. Instead, we recommend using job-specific or local reliabilities for making corrections for attenuation. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Confounding Effects of Insufficient Effort Responding Across Survey Sources: The Case of Personality Predicting Performance

Insufficient effort responding (IER) to surveys, which occurs when participants provide responses in a haphazard, careless, or random fashion, has been identified as a threat to data quality in survey research because it can inflate observed relationships between self-reported measures. Building on this discovery, we propose two mechanisms that lead to IER exerting an unexpected confounding effect between self-reported and informant-rated measures. First, IER can contaminate self-report measures when the means of attentive and inattentive responses differ. Second, IER may share variance with some informant-rated measures, particularly supervisor ratings of participants’ job performance. These two mechanisms operating in tandem would suggest that IER can act as a “third variable” that inflates observed relationships between self-reported predictor scores and informant-rated criteria. We tested this possibility using a multisource dataset ( N = 398) that included incumbent self-reports of five-factor model personality traits and supervisor-ratings of three job performance dimensions—task performance, organizational citizenship behavior (OCB), and counterproductive work behavior (CWB). We observed that the strength of the relationships between self-reported personality traits and supervisor-rated performance significantly decreased after IER was controlled: Across the five personality traits, the average reduction of magnitude from the zero-order to partial correlations was |.08| for task performance, |.07| for OCB, and |.14| for CWB. Because organizational practices are often driven by research linking incumbent-reported predictors to supervisor-rated criteria (e.g., validation of predictors used in various organizational contexts), our findings have important implications for research and practice.

Supplemental Material for Meta-Analytical Estimates of Interrater Reliability for Direct Supervisor Performance Ratings: Optimism Under Optimal Measurement Designs

Supplemental Material for Meta-Analytical Estimates of Interrater Reliability for Direct Supervisor Performance Ratings: Optimism Under Optimal Measurement Designs

The double-edged sword of job insecurity: When and why job insecurity promotes versus inhibits supervisor-rated performance

Research findings on the effects of job insecurity on performance are mixed. Some studies show a negative relationship between job insecurity and performance; while others indicate a positive association, a curvilinear or a null relationship. We suggest that the relation between employees' subjective job insecurity and supervisor-rated performance is more complex than previously assumed. Based on stress appraisal theory, this study investigates when and why job insecurity is associated with supervisor-rated performance via two different pathways. Specifically, we propose that these two mechanisms (i.e., impression management and workplace deviance) are contingent on the dyadic leader-member exchange (LMX) between the subordinate and his/her supervisor. In a three-wave survey using data collected from 314 subordinates and their 69 supervisors in China, we found the paradoxical relations between job insecurity and supervisor-rated performance at different levels of LMX. Specifically, on the one hand, job insecurity was positively related to supervisor-rated performance via impression management when LMX was high; on the other hand, job insecurity was negatively related to supervisor-rated performance via workplace deviance when LMX was low. Finally, the theoretical and managerial implications are discussed.

Exploring the Feasibility of Assessing Cultural Competence in Police Officers

Pathology, personality, and integrity-related construct assessments have been widely used in the selection of police officers. However, the incidence of police brutality and misconduct is still concerning. The present study explored the feasibility of the assessment of cultural competence in police officers. We explored the extent to which the change to the agency’s first ever Black CEO would affect cultural competence of the officers as well as incidence of misconduct. Results showed that scores on a cultural competence factor of an in-basket simulation used for promotional assessments at a state highway patrol agency were not predictive of either supervisor-rated performance or incidence of misconduct. Whereas results showed that misconduct was not predicted by the agency’s first Black CEO, cultural competence of the officers did increase after the change in command. Practical implications for law enforcement agencies and suggestions for future research are discussed.

Theory-driven game-based assessment of general cognitive ability: Design theory, measurement, prediction of performance, and test fairness.

Games, which can be defined as an externally structured, goal-directed type of play, are increasingly being used in high-stakes testing contexts to measure targeted constructs for use in the selection and promotion of employees. Despite this increasing popularity, little is known about how theory-driven game-based assessments (GBA), those designed to reflect a targeted construct, should be designed, or their potential for achieving their simultaneous goals of positive reactions and high-quality psychometric measurement. In the present research, we develop a theory of GBA design by integrating game design and development theory from human-computer interaction with psychometric theory. Next, we test measurement characteristics, prediction of performance, fairness, and reactions of a GBA designed according to this theory to measure latent general intelligence (g). Using an academic sample with GPA data (N = 633), we demonstrate convergence between latent GBA performance and g (β = .97). Adding an organizational sample with supervisory ratings of job performance (N = 49), we show GBA prediction of both GPA (r = .16) and supervisory ratings (r = .29). We also show incremental prediction of GPA using unit-weighted composites of the g test battery beyond that of the g-GBA battery but not the reverse. We also show the presence of similar adverse impact for both the traditional test battery and GBA but the absence of differential prediction of criteria. Reactions were more positive across all measures for the g-GBA compared to the traditional test battery. Overall, results support GBA design theory as a promising foundation from which to build high-quality theory-driven GBAs. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Uncertainty about Rater Variance and Small Dimension Effects Impact Reliability in Supervisor Ratings

ABSTRACT We modeled the effects commonly described as defining the measurement structure of supervisor performance ratings. In doing so, we contribute to different theoretical perspectives, including components of the multifactor and mediated models of performance ratings. Across two reanalyzed samples (Sample 1, N ratees = 392, N raters = 244; Sample 2, N ratees = 342, N raters = 397), we found a structure primarily reflective of general (>27% of variance explained) and rater-related (>49%) effects, with relatively small performance dimension effects (between 1% and 11%). We drew on findings from the assessment center literature to approximate the proportion of rater variance that might theoretically contribute to reliability in performance ratings. We found that even moderate contributions of rater-related variance to reliability resulted in a sizable impact on reliability estimates, drawing them closer to accepted criteria.

The Double-edged Sword Effects of Job Insecurity on Supervisor-rated Performance

Research evidences show that job insecurity has incongruent predictions on employees’ performance. We suggest that the relationship between employees’ subjective job insecurity and supervisor-rated performance is more complex than previously assumed. By integrating social exchange and proactive preservation mechanisms of job insecurity with the perceived workplace control contingency, this study investigates when and why job insecurity influences supervisor-rated performance via two opposing pathways. Specifically, we propose that the effects of employees’ subjective job insecurity on impression management and workplace deviance are contingent on their leader-member exchange (LMX) relationships. In a three-wave survey using data collected from 69 supervisors and their 314 subordinates in China, we found that job insecurity exerted paradoxical effects on supervisor-rated performance via two opposing mechanisms. Specifically, job insecurity had a positive indirect effect on performance via more impression management when LMX was high, whereas job insecurity had a negative indirect effect on performance via more workplace deviance when LMX was low. The theoretical and managerial implications of the findings are discussed.

Leader–member innovative work behavior (in)congruence and task performance: The moderating role of work engagement

We focus on dyadic interactions among supervisor- and employee-innovative work behavior (IWB) and how (in)congruence ([mis]fit) conditions relate to employee task performance. The logic of the person–supervisor (P–S) fit framework is used to explicate perceived supervisory performance ratings in the high-high fit and low-low fit situations, as well as designate the moderating effects of employee work engagement on the leader–member IWB incongruence–task performance link. A time-lagged two-source field research study on a sample of 157 employees and 40 direct supervisors from four public-sector organizations operating in an EU member state indicates that having highly innovative supervisors and likewise innovative employees produces the highest levels of employee task performance. Moreover, a moderated polynomial regression analysis revealed that employee work engagement buffers the effect of perceived misfit, elevating followers’ task performance in the condition of low supervisor-IWB and high employee-IWB. Contributions and implications for the innovation management and work engagement literature are discussed.

Job insecurity and employee performance: examining different types of performance, rating sources and levels

ABSTRACT We examine the relationship between job insecurity (JI) and performance (i.e., adaptivity, proactivity, task performance) from a multilevel perspective. We suggest that different behavioural responses will be triggered depending on whether the JI refers to an employee’s relative JI within a team or a team’s collective JI. An individual employee’s relative JI within a team may evoke a withdrawal reaction (i.e., diminished performance) because the individual experiences the insecurity as a personal issue (one which does not affect the rest of the team as much; i.e., a “person-at-risk” situation). However, when JI is experienced as a collective phenomenon (one that affects the entire team as a whole because of the shared context, i.e., a “job-at-risk” situation), employees may demonstrate higher performance as they are driven by job preservation motives. We incorporated both individual employee and supervisor ratings as they have complementary value in evaluating performance. Data was obtained from 53 teams, including 403 employees and 53 supervisors. Team’s collective JI was associated with higher supervisor-rated performance at the team-level, both in terms of adaptivity and proactivity but not in terms of task performance. The employee’s relative JI within a team was associated with reduced self-rated performance in terms of both adaptivity and task performance.

The role of fear of negative evaluation in interview anxiety and social‐evaluative workplace anxiety

AbstractWe investigated the consequences of interview and social‐evaluative workplace anxiety on job performance, and whether these two anxieties share a common antecedent–fear of negative evaluation. Job applicants (n = 128) completed a survey following their interview and halfway through their work term; supervisory performance ratings were collected at the end of the work term. Fear of negative evaluation was positively correlated with both interview anxiety and social‐evaluative workplace anxiety. The correlation between interview anxiety and job performance was near zero, as was the correlation between social‐evaluative workplace anxiety and job performance; these relations were not moderated by the social‐evaluative nature of the job. This study suggests that anxious interviewees and employees perform as well as their less anxious counterparts, even in social‐evaluative jobs.

Testing the value of expert insight: Comparing local versus general expert judgment models

AbstractExpert judges often claim to utilize expert insight to tailor judgments to maximize predictive validity for a specific context. We evaluated multi‐organizational assessment data regarding the prediction of supervisory ratings of job performance from ratings on individual assessment dimensions, finding no evidence that the average expert assessor effectively tailored judgments to specific organizations to maximize prediction. Expert judgment was outperformed in all organizational contexts by linear models of expert judgment, optimal weighted regression models, as well as simple sum composites. Critically, the dimension weighting policies of the expert assessors were not consistent with optimal weights for predicting job performance at any organization. We discuss why expertise tends not to contribute to predictive validity and describe methods for improving overall judgmental accuracy.

Occupational characteristics moderate personality–performance relations in major occupational groups

Personality predicts performance, but the moderating influence of occupational characteristics on its performance relations remains underexamined. Accordingly, we conduct second-order meta-analyses of the Big Five traits and occupational performance (i.e., supervisory ratings of overall job performance or objective performance outcomes). We identify 15 meta-analyses reporting 47 effects for 9 major occupational groups (clerical, customer service, healthcare, law enforcement, management, military, professional, sales, and skilled/semiskilled), which represent N = 89,639 workers across k = 539 studies. We also integrate data from the Occupational Information Network (O*NET) concerning two occupational characteristics: 1) expert ratings of Big Five trait relevance to its occupational requirements; and 2) its level of occupational complexity. We report three major findings. First, relations differ considerably across major occupational groups. Conscientiousness predicts across all groups, but other traits have higher validities when they are more relevant to occupational requirements: agreeableness for healthcare; emotional stability for skilled/semiskilled, law enforcement, and military; extraversion for sales and management; and openness for professional. Second, expert ratings of trait relevance mostly converge with empirical relations. For 77% of occupational groups, the top-two most highly rated traits match the top-two most highly predictive traits. Third, occupational complexity moderates personality–performance relations. When groups are ranked by complexity, multiple correlations generally follow an inverse-U shaped pattern, which suggests that moderate complexity levels may be a “goldilocks range” for personality prediction. Altogether, results demonstrate that occupational characteristics are important, if often overlooked, contextual variables. We close by discussing implications of findings for research, practice, and policy.

Consequences of Remote Work Use and Intensity: A Meta-Analysis

The large-scale shift to remote work on the heels of the COVID-19 has led to a realization by many organizations that remote work could be a long-term arrangement for their employees. Yet, we have an incomplete understanding about the consequences of remote work for employees. A prior meta-analysis (Gajendran & Harrison, 2007) conducted over a decade ago surveyed only 46 studies and hence was limited in its scope. Since then, the corpus of literature on remote work has grown significantly. We conducted a meta-analysis on remote work that includes 111 studies (k = 114, N = 303,449). This meta-analysis seeks to provide a more comprehensive understanding of the consequences of remote work by focusing on not only remote work use (remote workers compared to office-based workers) but also remote work intensity (extensiveness of remote working). We find that remote workers (versus office-based workers) have higher perceived autonomy, supervisor-rated performance, job satisfaction, organizational commitment, and engagement, while also having lower turnover intentions. Further, remote work intensity enhances perceived autonomy and supervisor-rated performance, and improves job and life satisfaction. In terms of mechanisms, the effects of both remote work use and remote work intensity on work- and nonwork-related outcomes come about via perceived autonomy. In contrast to prior theorizing, we find no evidence that remote work leads to impoverished workplace relationships. Likewise, it had no effects on work-family conflict. Overall, our study clarifies the nature of relationships linking remote work to important individual and organizational outcomes, enhancing our understanding of this work arrangement.

Risky Instrumental Behavior: Conceptualization, Measurement Development, and Validation

The current research develops and validates a measure of employee Risky Instrumental Behavior (RIB), a new construct capturing a novel form of employee discretionary behavior which, while presumed to enhance one's performance, also poses physical and/or psychological risks to one’s well-being. To assess RIB, we refined and validated a 7-item scale using six samples of individuals employed in various lines of work in three countries (USA, Israel, and China). Findings indicate that the RIB scale: (a) demonstrates high reliability and content validity; (b) is positively correlated, yet distinct from, other established employee discretionary work behaviors such as organizational citizenship behavior, pro-social rule breaking behavior, and working excessively; and (c) demonstrates incremental validity in the prediction of important employee outcomes including physical health, resource depletion, work exhaustion, and supervisor-rated performance, over and above demographics and other related discretionary work behaviors. We also identify situational (work pressure) and personality (conscientiousness and neuroticism) factors as antecedents of RIB. The theoretical and empirical implications of capturing this novel form of employee discretionary work behavior are discussed.

Meta-analytic Examination of a Suppressor Effect on Subjective Well-Being and Job Performance Relationship

This paper presents a meta-analytic study of the relationship between overall subjective well-being (SWB), cognitive SWB, affective SWB, and job performance ratings. The study examined the moderator effect of the source of job performance measure (self-report vs. supervisory ratings). The database consists of 34 independent samples (n = 5,352) using supervisory performance ratings and 38 independent samples (n = 12,086) using self-reported of job performance. These samples were located through electronic and manual searches. The results indicated that, on average, the correlation for SWB- supervisory ratings (ρ = .35) was slightly larger than for SWB-self-reported performance (ρ = .33). The correlation of affective SWB was much higher with supervisory ratings (ρ = .49) than with self-reported performance (ρ = .30). A suppressor effect of cognitive SWB was found for the prediction of supervisory ratings. Finally, we discuss the implications for the theory and the practice of SWB at work and suggest new research avenues.

In Leaders We Trust, or Should We? Supervisors' Dark Triad Personality Traits and Ratings of Team Performance and Innovation.

Our study tests in a sample of 87 organizational groups (297 employees and 87 supervisors) the mediating role of leader-member exchange (LMX) and collective narcissism in the relationship between supervisors’ dark triad (SDT) personality traits and ratings of team outcomes made by supervisors and team members. We show that LMX mediates the association between SDT and team performance and innovation as rated by team members, while collective narcissism mediates the association between SDT and supervisory ratings of team innovation and team performance. Moreover, collective narcissism also mediates the association between SDT and team innovation as rated by team members. Results show that team-level performance appraisal is influenced by supervisory attributes and that the quality of relational exchanges and collective narcissism are plausible mechanisms explaining this association. The use of supervisory ratings of team outcomes in empirical research should also account for the supervisory attributes.

Do self-monitors experience less emotional exhaustion? Testing the role of supervisor-rated performance and perceived competitive climate

PurposeThe purpose of this research is to examine frontline employees' self-monitoring personality as an antecedent of their emotional exhaustion and how supervisor-rated performance mediates this relationship. In addition, the authors explored the moderating role of perceived competitive climate on the indirect relationship between self-monitoring and emotional exhaustion.Design/methodology/approachTwo hundred and thirty-seven frontline employees and their immediate supervisors working in hospitality organizations responded to the survey using time lagged research design. Measurement model was tested using confirmatory factor analysis to assess the distinctiveness of study constructs, and proposed moderated mediation model was tested using Process macro.FindingsResults show that high self-monitoring leads to high supervisor-rated performance, and this relationship is stronger in highly competitive work climate. The supervisor-rated performance was negatively related to emotional exhaustion.Originality/valueThis study is the first to examine the interaction effects of self-monitoring and perceived competitive climate on frontline employees' performance and emotional exhaustion, particularly in the frontline jobs. Supervisor-rated performance has not been previously theorized or researched as an underlying mechanism of the effect of self-monitoring on emotional exhaustion.