Content, culture, and context: determinants of quality in psychiatry residency programs.

Abstract

Residency training programs vary across characteristics such as their didactic and clinical experiences, attributes of the incoming residents, faculty characteristics, research conducted, community service performed by the program, and eventual practice choices of graduates. Which of these characteristics are most salient for evaluating the quality of a program? Elliott1 lists characteristics of graduates, cost—effectiveness, fair and ethical treatment of trainees, and meeting societal needs as important quality indicators. Iverson2 takes a dimensional approach. His dimensions, with metrics, are: intake [U.S. Medical Licensing Examination (USMLE) scores of matched applicants]; customer satisfaction [percentage of available positions filled by match, and percentage filled by U.S. medical school graduates (USMGs)]; residency review committee (RRC) quintile scores; and outcome (specialty board pass rates). The Accreditation Council for Graduate Medical Education (ACGME) recently switched its focus from process variables to outcomes, and is encouraging RRCs to evaluate a program on how well it provides for six core competencies: patient care, clinical science; interpersonal skills and communication, professionalism, practice-based learning and improvement, and systems-based practice.3 In 1997, a task force of the American Association of Directors of Psychiatry Residency Training (AADPRT) developed a survey to define the variables important to determining a program's quality from the psychiatry resident's perspective. The 41-item questionnaire was based on feedback from focus groups of psychiatry residents and program directors and a review of the literature. A total of 180 psychiatry residents from 16 programs completed the survey. Quality of supervision and teaching conferences, respect of faculty for residents, responsiveness of the program to feedback from residents, and morale in the department were the items most important to residents' satisfaction. A detailed description of the construction of the survey and its results was published by Elliott.4 In 1998, the AADPRT's survey was repeated with psychiatry residency directors and heads of major rotations to see whether their values agreed with those of the residents.5 This paper describes the use of multidimensional scaling (MDS) of the survey's results to establish whether there were distinct groupings of program directors with different opinions about the determinants of quality in psychiatry residency programs. These groupings might represent types of psychiatry programs (or market niches) as reflected in the values and priorities of their faculty and directors. Method Multidimensional scaling is an analytic technique frequently used in marketing research to identify the psychological dimensions underlying customers' preferences with respect to multiple variables or features of a product.6 In MDS the difference between clusters or groups of variables is predicted by the distance between the variables in psychological “space,” with the dimensionality of the space equal to the number of relevant dimensions underlying the data. These dimensions can be thought of as analogous to the latent constructs derived in factor analysis. The scaling algorithm derives the dimensions and plots the coordinates of the variables in the resulting multidimensional space. MDS is an inherently interpretive procedure—it locates variables on dimensions but requires the investigator to determine whether the dimensions can be intelligibly labeled. Individual Differences Scaling (INDSCAL) is an MDS algorithm that models both the overall dimensions that underlie the perceptions of the group of respondents and individual weights on those dimensions for each respondent, allowing individuals to vary in the importance they attach to each of the dimensions. For example, for one individual, dimension A (the educational resources available, for example) may be highly salient, while dimension B (the administration of the program, for example) is relatively unimportant. For another individual, these priorities may be reversed. By examining the distribution of subject weights one can identify clusters of subjects who share similar values regarding the relative importances of the various dimensions. The questionnaire was sent in late 1998 to all psychiatry residency directors listed in the American Medical Association's 1998–1999 Directory of Accredited Graduate Medical Education (GME) Programs. The faculty members who served as the heads of the inpatient and outpatient psychiatry rotations of each program were also surveyed. These are the two major rotations of psychiatry training programs, and the opinions of the heads of these rotations (henceforth referred to as service chiefs) would most likely represent the dominant values of the program. The survey asked directors and service chiefs to rate how important the 41 items of the questionnaire were in determining the quality of a residency program. The anchors were 1 = least important, 4 = average importance, and 7 = most important. Multidimensional scaling using INDSCAL was done on the survey responses. Solutions in two to six dimensions were generated. Results Of the 186 active programs listed in the GME directory, 117 programs (63%) responded to the survey. There were 234 individual responses from the 117 programs. Of these, 142 (61%) were from program directors and 92 (39%) were from service chiefs who were not identified as directors. For some programs the head of inpatient or outpatient services also served as an associate program director, confirming our supposition that these faculty members represent the administrative backbone of the program. The Pearson correlation between the responses of the residency directors and those of the service chiefs was 0.98 (p < 0.01). We therefore pooled data from both chiefs' and residency directors' responses for the following analysis. The two-dimensional INDSCAL solution was degenerate and was discarded. The solutions in three to six dimensions were inspected for goodness of fit and interpretability. The three-dimensional configuration provided the most interpretable dimensions, and accounted for 46.4% of the variance in the data; higher-dimensional solutions accounted for only slightly more variance. Based on examination of the locations of the items, particularly those that had particularly high or low coordinates in each dimension, the dimensions seemed to correspond to three constructs: “curriculum,” “quality of the institution,” and “supportiveness of the administration of the program.” The three dimensions, and the highest-loading items on each, are given in Table 1.TABLE 1: Three Dimensions of Quality of Psychiatry Residency Programs, Based on Multidimensional Scaling of Responses by Residency Directors and Service Chiefs to a 1998 Questionnaire*Subject weights measure the importance or salience of each dimension to each respondent; they range from 0 (completely ignored) to 1 (overwhelmingly important), and need not sum to 1. In addition, each respondent is assigned a “weirdness' value, which measures the similarity of his or her responses to those of the typical respondent, based on the relative importance of each dimension and the goodness of fit for that respondent. There was a great deal of variation in individual preferences, but no distinct clusters were evident. Notably, although weights for the supportiveness of the administration of the program dimension fell between 0.25 and 0.40 for nearly all respondents, the importance attributed to the dimensions of curriculum and quality of the institution varied extensively across individuals. Figure 1 plots the weights of curriculum and quality of the institution against one another for each respondent. The unshaded polygon encloses data for more typical respondents with less than the median weirdness, represented as circles; the two shaded polygons identify two groups of less typical respondents with more than the median weirdness, represented as crosses. Two respondents in the upper left had extreme (outlier) weirdness values.Figure 1: Subject weights for the “curriculum” and “quality of the institution” dimensions of the three-dimensional INDSCAL solution. The unshaded polygon encloses subjects with less than the median weirdness (based on weights in all three dimensions), represented as circles; the two shaded polygons identify two groups of subjects with greater than median weirdness, represented as crosses. Two subjects in the upper left had extreme (outlier) weirdness values.“Weirdness” measures how similar each respondent is to the typical respondent, based on the relative importance of each dimension and the goodness of fit for that respondent.While the most typical respondents gave curriculum weights between 0.3 and 0.6, and quality of the institution weights between 0.2 and 0.4, two groups of respondents displayed different weight patterns. One group (lower right) gave curriculum substantially higher weights than typical (ranging from 0.5 to 0.75); the other group (upper left) gave quality of the institution substantially higher weights than typical (ranging from 0.25 to 0.65). On the average, most respondents' data displayed a continuum of weight patterns in which curriculum was considered to be more important than either quality of the institution or supportiveness of the administration of the program; the respondents with the lowest weirdness scores weighted these dimensions in the proportions 1.3:1:1, respectively. Discussion The three dimensions that emerged from the MDS are consistent with the many suggested quality indicators reviewed above.1,2,3 How might we conceptualize this triad? The dimensions of curriculum and supportiveness reflect two different aspects of the process of residency training. The curriculum dimension describes the content of the educational program; the supportiveness dimension reflects the culture or ambiance within which the training occurs. Residency directors and their faculty seem to differentiate between these two aspects of the program, and value both as indicators of the quality of the program. The dimension of institutional quality includes items reflecting the reputation and resources of the institution as well as items generally considered to be outcomes (i.e., board scores of graduates and graduates' job satisfaction). In this instance, these “outcome” items probably serve as proxies for (and a reflection of) the reputation of the institution and the quality of the residents it attracts, rather than as true outcome measures. This dimension could represent a general context factor, reflecting the quality of the facilities, the faculty, and of the residents themselves. This general factor could itself modify the effects of the process variables either up or down, thereby affecting the expected outcomes. Thus, this dimension may reflect the expectation of program directors and chiefs that equivalent processes (curriculum and supportiveness) could lead to better outcomes if they are provided in the context of a higher-quality institution. Donabedian lists input, process, and product as the dimensions of quality in healthcare.7 Interestingly, product (outcomes) did not emerge as a dimension of the quality of a program. Perhaps residency directors and service chiefs focus on process variables as indicators of quality since it is in the process of residency training that they deal. The neglect of outcome measures may also reflect a philosophy that, while the program is responsible for teaching, it is the residents' responsibility to learn. Outcome measures are highly confounded by the abilities and characteristics of the individual residents and, thus, may not be considered an accurate or reliable measure of the quality of the program per se. The ACGME and others who have begun the move towards outcome evaluation may wish to take this as a word of caution. Outcomes should not be considered in a vacuum. For at least some key stakeholders—the residents and faculty of the program—the context, content, and culture of the program are significant as well. No truly distinct clusters or groups of respondents emerged from the multidimensional scaling of the data. While there may be programs with different missions—programs oriented towards research or community psychiatry, for example—these missions do not seem to result in drastically different definitions of quality. This suggests that there is a core concept of quality that holds across contexts and across missions—consistent with the RRC's model of minimum standards. On the other hand, there seems to be a continuum of individual variation, rather than variation based on group membership. The individual respondents differed widely in the dimensions they considered most important. Since the individuals in this case are the faculty leaders and directors of the programs, these priorities are most likely reflected in the programs as a whole. The individual variation can be usefully segregated into three “market niches,” corresponding to the three polygons in Figure 1. Thus we could describe three types of programs: (1) programs in which the quality of the sponsoring institution (context) is paramount, (2) programs in which the quality of the curriculum (content) is paramount, and (3) programs with a more typical weighting of the three dimensions. While there was less variability in the importance attached to the supportiveness (culture) of the program, this should not be construed as lack of salience. Rather, all programs should be alert to the importance of this dimension. Residents, too, vary in the levels of importance they attribute to the various features of a training program.4 The context, content, and culture of a program may provide a good conceptual model of the dimensions along which the market varies. Programs may find it useful to identify and market themselves on the basis of these dimensions. This study focused on only one of the many stakeholder groups of residency programs. The needs and expectations of other stake-holders, such as program funders and employers of a program's graduates, remain to be defined. This study also focused only on psychiatry programs, but the dimensions of context, content, and culture would seem to be potentially applicable to other specialties as well. Repeating the study with other specialties will tell us whether indeed this triad is relevant to the quality of programs across specialties. The method of multidimensional scaling is a novel one for determining quality measures in graduate medical education. Repeated use of this technique, across stakeholders and across specialties, can help elucidate the factors most important to the evaluation of residency programs.