Reliability and validity of the anesthesiologist supervision instrument when certified registered nurse anesthetists provide scores.

Abstract

At many facilities in the United States, supervision of Certified Registered Nurse Anesthetists (CRNAs) is a major daily responsibility of anesthesiologists. We use the term "supervision" to include clinical oversight functions directed toward assuring the quality of clinical care whenever the anesthesiologist is not the sole anesthesia care provider. In our department, the supervision provided by each anesthesiologist working in operating rooms is evaluated each day by the CRNA(s) and anesthesiology resident(s) with whom they worked the previous day. The evaluations utilize the 9 questions developed by de Oliveira Filho for residents to assess anesthesiologist supervision. Each question is answered on a 4-point Likert scale (1 = never, 2 = rarely, 3 = frequently, and 4 = always). We evaluated the reliability and validity of the instrument when used in daily practice by CRNAs. The data set included all 7273 daily supervision scores and 1088 comments of 77 anesthesiologists provided by 49 CRNAs, as well as the 6246 scores and 681 comments provided by 62 residents, for dates of service between July 1, 2013, and June 30, 2014. Reliability of the instrument was assessed using its internal consistency. Content analysis was used to associate supervision scores (i.e., mean of the 9 answers) and presence of the verbs "see" or "saw" combined with negation in comments (e.g., "I did not see the anesthesiologist during the case(s) together"). Results are reported as the mean ± SE from among the 6 two-month periods. Supervision scores <2 were provided for 7.2% ± 0.4% of assessments and scores <3 were provided for 36.6% ± 1.1% of assessments, by 18.2 ± 0.9 and 34.0 ± 0.6 CRNAs, respectively (i.e., low scores were not attributable to just a few CRNAs or anesthesiologists). These frequencies were greater than for trainees (anesthesiology residents) (both P < 0.0001). No single question among the 9 questions in the supervision instrument explained CRNA supervision scores <2 (or <3) because of substantial (expected) interquestion correlation. Cronbach's alpha equaled 0.895 ± 0.003 among the 6 two-month periods. Among the CRNA evaluations that included a written comment, the Cronbach's alpha was 0.907 ± 0.003. Thus, like for anesthesiology residents, when used by CRNAs, the questions measured a one-dimensional attribute. The presence of a comment containing the action verb "see" or "saw," with the focus theme ("I did not see …"), increased the odds of a CRNA providing a supervision score <2 (odds ratio = 74.2, P = 0.0003) and supervision score <3 (odds ratio = 48.2, P < 0.0001). Limiting consideration to scores with comments, there too was an association between these words and a score <2 (odds ratio = 19.4, P = 0.0003) and a score <3 (odds ratio = 31.5, P < 0.0001). In Iowa, substantial anesthesiologist presence is not required for CRNA billing. More comments containing "see" or "saw" were made by CRNAs rather than residents (n = 75 [97.4%] versus n = 2 [2.6%], respectively, P < 0.0001), indicating face validity of the analysis. If some of the 9 questions were not perceived by the CRNAs as relevant to their interprofessional interactions, Cronbach's alpha would be low, not the 0.907 ± 0.003, above. Similarly, one or more of the individual questions would also not routinely be scored at its upper boundary of 4.0 ("always"). This was not so, being as the score was 4.0 for 24.9% ± 0.3% of the CRNA evaluations, and that score of 4.0 was more common than even the next most common combination of scores (P < 0.0001). The de Oliveira Filho supervision instrument was designed for use by residents. Our results show that the instrument also is reliable and valid when used by CRNAs. This is important given our previous finding that the CRNA:MD ratio had no correlation with the level of supervision provided.