Gauging interest in community service: a retrospective review of admission files.

Abstract

Medical schools hold a social contract to improve the public health.1 In meeting this commitment, they have made notable strides in the provision of patient care, development of technology, generation of new knowledge, and training of professionals. Unfortunately, far less progress has been made in devising community outreach activities and meeting basic health needs.2 To remedy this, physicians should be selected and trained with the appropriate mix of skills, attitudes, and values necessary to improve the public's health, particularly of the poor and underserved.1,2 As a result of the personal orientation and premedical preparation of applicants, as well as the selection criteria of admission committees, many students enter medical school with altruistic inclinations.3 Educational programs that facilitate community service within the medical curricula are thought to reinforce and integrate altruistic values into the profession and promote social responsibility.4,5 If medical schools want to address public health concerns, some have advocated that community service be a part of medical training.6 Previous studies have documented that students who choose to participate in service—learning experiences differ in their attitudes, skills, values, and understanding about social issues compared with students who do not.7 By considering applicants' humanitarian interests, medical school admission committees play a crucial role in helping medical schools fulfill their social contract with the public—the selection of competent and caring future physicians. Validly assessing such diverse qualities, however, remains problematic. Admission committees use undergraduate grade-point average and Medical College Admission Test scores as markers of intelligence and aptitude. Applicants' humanitarianism or altruism is less amenable to valid measurement. Usually, such characteristics are expressed through the applicant's self-reported participation in community service activities. Because details regarding these experiences are often sketchy, the scope and depth of an applicant's community service experience remain difficult to ascertain and quantify. Admission committees rely on community service experience information in selecting applicants who may have altruistic inclinations. Little is known about the predictive capacity of this non-academic performance variable as it relates to behavior in medical school, or future medical practice. This study addresses three questions related to the use of applicants' community service experience for selection and prediction purposes by an admission committee. First, using key medical school admission file materials, can reviewers reliably assess applicants' experiences with, and orientation toward, community service? Second, do reviewers use specific aspects of community service to determine their overall assessments of community service involvement? Third, can these assessments predict students' voluntary participation in a new first-year service-learning elective? Method With institutional review board approval, the study utilized a retrospective review of admission file materials drawn from the medical school applications of matriculants to the University of Kentucky College of Medicine (UKCOM). The mission of UKCOM is to “assume a leadership role in assessing the health care needs of the Commonwealth of Kentucky.”8 Applicants are advised during recruitment that “demonstrated service to others” is a necessary personal attribute.8 The subject population comprised 91 students—44 females (48%) and 47 (52%) males who entered medical school in the fall of 2000 and completed the first-year curriculum. Selected study information was compiled from two primary data sources. First, from the 2000 American Medical College Application Service (AMCAS) application, extracted information included (1) post-secondary honors and awards; (2) extracurricular, community, and avocational activities; (3) chronological post-secondary history, including volunteer, part-time, and full time employment; and (4) AMCAS personal essays. Second, we examined students' responses to an item on the UKCOM secondary application: “Describe the most significant community service activity in which you have participated. How did it affect you?” Information from these documents was compiled for each subject and assigned a random number, and all individual identifiers were removed. Four of the 16 members of the admission committee volunteered to review files in early 2001—eight months after the entering students had matriculated. The reviewers had had no contact with the matriculants since their admission. In the intervening months, the file reviewers, as members of the admission committee, had screened the files of over 350 applicants to the 2001 entering class. The passage of time, volume of files screened, and removal of identifiers effectively blinded the reviewers to the subjects' identities. Key items used to rate students' interests in community service were derived from structured interviews with students from five U.S. medical schools,9 with the file reviewers undergoing a series of training sessions orienting them to the nine-item rating form. Community service was defined as “activities where the applicant has helped others by providing support or assistance, apart from their simply seeking exposure to the medical profession.” Reviewers were instructed to respond to items (see Table 1) assessing various aspects of students' community service experience, including (1) interest (V1) and commitment (V5); (2) type (V2) and duration (V4); (3) level of involvement (V3); (4) settings (V8) and target clienteles (V9); and (5) overall demonstrated level relative to other applicants (V6). A written guide described the behavioral anchors for responding to each item, and both the form and guide were pilot-tested using five student files (not in the sample).TABLE 1: Raters' Assessments of Students' Community Service Experience: Predictors of Participation in a First-year Service ElectiveTo facilitate administration, all variables consisted of dichotomous or trichotomous measures. Intra-class correlations (ICCs) based on absolute agreement of raters were used to estimate inter-rater reliability. Key items were averaged across raters using mean or modal responses, and Spearman rank-order correlation coefficients (rho) were used as nonparametric measures of association. A critical alpha of ≤. 05 was specified for all analyses. Results The first research question addresses the extent to which the four file reviewers were able to review a uniform set of admission materials and reliably assess applicants' interest in community service (V1). Using the students' descriptions and extents of their service, the reviewers classified student interest into three categories: “yes” (clear participation and description of activities), “somewhat” (lists activities but provides limited explanation of activity; time and depth of interest remain unclear), or “no” (lists one activity, but gives no additional information). Absence of information about community service activity was rated as “no activity.” No student was excluded from the study. The file reviewers assigned “yes” responses to 48% of the class, “somewhat” to 27% of the class, and “no” to 25% of the class. Coded “3,” “2,” and “1,” respectively, this resulted in a mean value of 2.20 (SD = .64). Inter-rater reliability for this item—represented by the intra-class correlation—was .79. As shown in Table 1, intra-class correlations for comparably coded variables were .64 (V2) and .84 (V3, V5). Item V6—with response categories of “high” (more than most), “average,” and “low” (lower than most) coded “3,” “2,” “1,” respectively—also had an intra-class correlation of .84. Estimates for dichotomous items (V4 and V7) were .74 and .40, respectively. The second research question identifies those aspects of service activity most strongly related to students' overall levels of community service relative to other matriculants (V6). Aspects studied were perceived interest in service (V1), medical basis of activity (V2), leadership (V3), duration (V4), commitment (V5), and types of agencies (V8) and clients (V9) served. The aspects most closely related to overall relative levels of service were perceived interest (rs = .89) and commitment (rs = .93). Medically-related service experience (V2) was negatively correlated (rs = -.72, p ≤ .05) with overall assessment of interest in community service—suggesting the raters were distinguishing between medical volunteering and community service. Four contextual variables related to the organizations and clients served were also significantly associated with raters' relative overall assessments of community service. If students' service experiences involved a school organization (rs = .44, p ≤ .05) or community/charitable organization (rs = .37, p ≤ .05), or if they served children (rs = .45, p ≤ .05) or adults (rs = .25, p ≤ .05), raters' overall assessments of students' levels of community service tended to be higher. However, the latter relationship should be interpreted cautiously due to low inter-rater reliability. The final research question addresses whether the raters' responses to those aspects of community service listed above (V1-V6, V8-V9) could predict students' participation in a first-year service-learning elective (V7). The far right column of Table 1 illustrates the levels to which items were linked to raters' abilities to identify participants in the service-learning elective. As shown, only a single variable—students' commitment to community service (V5)—shows a significant correlation with the students' actual participation in the service—learning elective (rs = .25, p ≤ .05). The seventh item, specifically asking the file reviewers whether the student would opt to participate in the service-learning elective, correlated at .21 with actual participation. From their review of admission materials related to premedical service experiences, the latter finding suggests the raters' predictions of who would be likely to participate in the first-year service-learning elective (V7) were not terribly accurate. When asked simply (yes/no) whether they considered the student to be a likely participant in this course, individual ratings correctly corresponded with actual participants 40.2% of the time (ICC = .40). In eight cases, three raters correctly predicted students' participation in the elective; however, for almost as many applicants (6), no rater correctly identified them as actual participants. In no instance did all four raters reach consensus in accurately identifying a community service participant. No contextual variables relating to organizations or clienteles significantly discriminated service elective participants from non-participants. Discussion The belief that participation in community service activities is linked to humanitarian concern for the public and patients compels admission committees to give credence to applicants' demonstrated service experience. Although committee members have typically done so informally and without standardized, operational measures, the question remains whether service experience can be reliably and validly assessed from applicants' file information. Further, can admission committee members predict students' future service activities based on an analysis of past experience? Given the importance of community service experience in our selection criteria, and because it is regularly discussed in admission committee meetings, the file reviewers were surprised to find that roughly one in four matriculants expressed little or no interest in community service activities. As a result, the file reviewers were anxious to share the results of this study with other committee members so that strategies could be formulated to enhance consideration of service interests in making selection decisions. From these findings, it appears that an acceptably reliable tool can be developed—in concert with reviewer training—to assess applicants' community service histories within a medical school admission process. Although room exists for refining the measures, these findings suggest that certain contextual aspects of students' service experiences may play differential roles in reviewers' overall assessments. In judging interest in community service, consideration should be given to the leadership, focus, and length of service experiences. Findings from this study also indicate that the service experiences involving schools, community/charitable organizations, and both child and adult populations appear to be more closely linked to overall assessments. Again, the rigor of these relationships is tempered by the inter-rater reliability of each rating. Measurement issues withstanding, equally telling is the reviewers' limited accuracy in identifying prospective participants in a first-year service-learning elective. However, since participation in this elective is just one way a student may demonstrate his or her commitment to community service, this singular outcome is perhaps an overly narrow criterion for detecting a relationship between past experience and future behavior. Mere participation or non-participation may not accurately reflect humanitarianism, and many service-oriented students may have felt that contributing 32 hours in the midst of a taxing first-year medical curriculum was simply impractical. However, with the national charge to encourage student interest in treating poor and underserved communities,1,2 it is important to pursue this line of research, establishing experiential “markers” that can be noted at admission to predict later interest in community service. These findings are limited by several factors. First, this study was conducted on a single cohort of matriculants to a primary care-oriented medical program. File folders of applicants not admitted were not reviewed, and it is unknown whether the applicants' levels of service activity contributed to the selection decision. As a result, the generalizability of the findings may be restricted to comparable settings or pools of applicants. Second, although our interview form does not contain any questions explicitly related to applicants' community service, they were not among the materials reviewed in spite of relevant information they might have provided. Subsequently, choice of items used in the review of students' community service prior to medical school was constrained by the application data on hand. Third, as interested volunteers, the file reviewers' motivations to participate may have affected the reliability of the ratings. Last, changes in reporting service experiences on the revised AMCAS 2002 form may allow for a different compilation and analysis of future data. Generally speaking, the file reviewers demonstrated marked uniformity in assessing students' demonstrated levels of community service—though they showed far less uniformity and accuracy in their predictions of future involvement. Since inter-rater agreement tended to be adequate on most items, increases in reliability from additional reviewers would likely be negligible, and would not meaningfully address the more important, overarching issue of validity. Arguably, since most applicants have some experience that can be construed as “service,” we cannot definitively conclude that a demonstrated history of service is either necessary or sufficient for future involvement in these activities. In short, it seems likely that the correlates of interest in and commitment to community service are far more diverse and complex than presented here. Additional predictors of community service (e.g., applicants' skills, values, and understandings of social issues, as well as self-ratings of desire to serve) should be identified from the social science literature, followed by theoretical and empirical studies to develop and refine valid measures and coding schemes.