Validation and refinement of the “Airman Comprehensive Assessment”: Evaluating competency proficiencies of enlisted members

Abstract

ABSTRACT Beyond proficiency on occupationally specific tasks, the U.S. Air Force expects members to develop proficiency on institutionally valued “soft skill” competencies (e.g., Teamwork, Communication, and Initiative) throughout their careers. As such, all E1-E6 members are annually evaluated using Behaviorally Anchored Rating Scales (BARS) designed to measure such competencies. Despite mandated use, these Airman Comprehensive Assessment (ACA) scales previously have not been empirically evaluated. To address this gap, we surveyed Air Force supervisors, using a criterion-related sampling methodology to validate the behavioral anchors for each scale. Supervisors identified two subordinates of the same rank/career field who they viewed as having (a) high potential for future success in an Air Force career or, alternately, (b) lower potential for future career success and rated each subordinate on the individual behaviors that comprise the 12 scales. ACA items were intermixed with scale items previously identified as distinguishing top performers in civilian organizations. Results demonstrate scale reliability and generally validate the ACA competency scales as stronger differentiators of supervisor-rated career potential than competency scales developed for civilian organizations. We provide recommendations for re-calibration of scale anchors based on the relative percentage of high vs. low potential members that demonstrate each behavior, and suggest changes to improve correspondence between measured competency proficiency and supervisor-rated career potential.