Assessing Procedure Adherence Under Training Conditions in High Risk Industrial Operations

Abstract

Written procedures are an important artifact in maintaining the safety in high risk industrial operations. Procedures set out the steps required to complete safety and process critical tasks. However, as Sidney Dekkar noted, while procedures spell out how to do the job safely, “following all the procedures can lead to an inability to get the job done” (Dekker, 2003, p. 235). For instance, while a procedure will set out a safe step-by-step approach to achieving a desired outcome, a work environment, filled with uncertainty and constraints may not allow for a procedure to be executed correctly (Dekker, 2003). Although procedures are used to support safety, procedure misuse has been identified as a contributing factor in incidents and near misses in multiple industries (e.g. Alper & Karsh, 2009; Bullemer, Kiff, & Tharanathan, 2011). Such findings are often based on retrospective, case-study approach to describe deviations from expected steps and order within the procedures. While misuse of procedures has been identified as a contributing factor in major incidents, little research has been conducted on how procedures are used under normal conditions. Thus, the aim of this research is to demonstrate the applicability of an objective approach to assess adherence and worker behavior with written procedures. Consequently, the method presented will identify how and when workers deviate from procedures through the conduct of normal work. The proposed objective assessment of worker use of procedures consists of a two-tier assessment of a worker’s adherence to each step within a procedure. Within Tier 1, a worker’s compliance to each step is assessed as either completing correctly without issue (i.e., WAD = WAI) or work not completed as expected (i.e., WAD ≠ WAI). Tier 2 provides a more detailed description of how WAD is different to WAI. Specifically, for those steps not completed as expected, 7 types of deviations, ranging in their severity to potentially unforeseen consequences, are used to assess worker use of procedures. Specifically, the worker could: require assistance, struggle, iterate between steps, skip a step and go back to it later, complete a step out of order, incorrectly execute a step, or not complete a step. Using analysis of audio/visual data, the assessment method was used to test if experienced and inexperienced workers use written procedures differently in a high fidelity simulated training environment. Results identified that there is a difference between experienced and inexperienced workers, with experienced workers complying with the procedures at a higher rate. Results also identified that when experienced operators deviated from the procedure it was through either by skipping or incorrectly executing a step in the procedure. For inexperienced workers, deviation from the procedure occurred through gaining assistance or struggling with the action required in the procedure. When combining correct procedure use (Tier 1), assistance and struggle (Tier 2), there is little difference between experienced and inexperienced workers. The assessment approach described how workers use written procedure under normal work conditions. To our knowledge, this is the first time this particular assessment technique has been used. While the method offers, at the coarse level, a binary correct/incorrect assessment of procedure step compliance; it also allows for an understanding of how deviations occur. The method provides safety engineers, managers and procedure writers with data which can be used to change or modify written procedures and to improve process and safety training. Furthermore, data collection for the method is relatively unobtrusive (small camera on a safety helmet) and low cost. Given the capabilities of small portable cameras, the approach could also, theoretically, be applied in real time. From a WAD/WAI perspective, the assessment approach allows for a detailed understanding of how work occurs under normal conditions. Such an approach provides an ability to understand how deviations from WAI as a preventive approach to safety within high risk environments.