Evaluation of a CME problem-based learning internet discussion.

Abstract

Research into the impact of continuing medical education (CME) demonstrates that effective interventions include practice-enabling or reinforcing strategies, sequential activities, and/or a high degree of interaction among participants.1,2 Problem-based learning (PBL), a strategy used in CME, engages participants in small-group interactive learning, creating a context that reflects the practice setting by presenting actual cases as problems to be solved.3 PBL specifically has been shown to be an effective learning strategy in CME.4,5 Traditionally, PBL participants have been required to be in the same place at the same time, but now the Internet enables interpersonal interaction that is independent of time and place. Using asynchronous (delayed) interaction via a bulletin board, learners in different locations can participate in on-line discussions at times convenient for them.6 For physicians this removes barriers (e.g., geographic location, practice responsibilities) to participating in conventional CME programs and interacting with fellow learners.7 Although the Internet provides many opportunities for medical education,8,9,10 a recent search of the medical literature revealed few studies of its use for interpersonal interaction in medical education,9,10,11,12,13,14,15,16 and only one study of on-line PBL. In that study, Chan17 attempted to determine the effectiveness of an on-line PBL program in a randomized controlled trial of 23 physicians. Group process, however, was not the focus of study, and the number of messages was small (35 over two months). Barrows advocates that successful PBL requires facilitators to perform four functions: navigating (guiding the group through the activities), facilitating (maintaining a constructive group process), questioning (using questions to deepen understanding), and diagnosing (monitoring learners' progress).18 Berge and Collins suggest facilitator roles for general on-line discussions, which include pedagogic (ensuring the educational task is accomplished), social (creating a friendly environment), managerial (administrating organizational elements), and technical (ensuring comfort with the technology) roles.19 The purposes of the present study of an Internet (on-line) CME PBL discussion were (1) to describe the roles of facilitators, both on-line and off-line, that enable on-line discussion; (2) to determine factors that influence learners' participation in the on-line discussion; and (3) to determine learners' satisfaction with the on-line discussion. The study was a process evaluation, which documents and assesses the implementation of a program's activities to guide further program planning.20 Method Family practitioners in Nova Scotia comprised the target population, but other physicians could register. We recruited participants by advertising the program locally and demonstrating it at a provincial CME event. The intervention, carried out in 1999, was an on-line case-based learning module on medication-induced headache (MIH) developed by a neurologist for a conventional PBL CME workshop and modified for Internet presentation. We chose this program because the original workshop was successful3 and the neurologist is an expert PBL facilitator interested in Internet learning. We used Web-CT21 educational courseware for the module. Besides the case-based discussion in the bulletin board, the module included a “lecture,” a quiz, a glossary, and references. We encouraged learners to review the lecture before joining the discussion. To meet the College of Family Physicians of Canada accreditation criteria for on-line CME programs; i.e., that the program be available for a defined time period and provide the opportunity for physician interaction,22 the module was available for one month and participants were required to post at least one message in the bulletin board. Using Berge and Collins' facilitator roles,19 we outlined two general roles for the facilitators of the on-line PBL discussion. These were (1) the pedagogic, or content, role, assumed by the neurologist or content facilitator, and (2) a combined social (creating a supportive environment), managerial, and technical role, assumed by two educators. A graduate student familiar with Web-CT also provided technical support. We collected data using the Web-CT electronic activity record, the program evaluation questionnaire, facilitators' records of on-line and off-line activities, bulletin board discussion transcript, a log of technical problems, and interviews with registrants who did not participate. The questionnaire was designed to evaluate all components of the on-line program and consisted of 51 closed-ended and seven open-ended questions. For this study, we used the nine closed-ended and one open-ended questions that addressed the case and bulletin-board discussion, and three closed-ended and one open-ended question that addressed the general usefulness of the module. Participants completed the questionnaire electronically or on paper. Evaluation questionnaires received electronically were automatically entered into the Web-CT database, which computes descriptive statistics. We manually entered evaluations received by paper. For the bulletin board discussion transcript, we used content analysis to categorize data and identify themes.23 Results The 31 registrants were 28 family physicians, two family medicine residents, and one neurologist. The electronic activity record showed that 12 registrants did not participate. Of these, three did not log into the program, and nine accessed the home page only. We attempted to contact these 12 and received responses from four. Two were unable to log on and had not contacted the “help line.” One reported personal computer failure and another had become “too busy.” Of the 19 who accessed the MIH module content, 14 participated in the on-line case discussion. Fifteen of the 19 (79%) participants who accessed the module completed the evaluation questionnaire. These included the 14 who posted messages and one who read bulletin board messages but did not post any. List 1 summarizes their demographic and computer usage data.LIST 1. Respondents' Demographic and Computer-use Data, Dalhousie University, 1999Table 1 summarizes the same 15 respondents' ratings of items addressing the case discussion and the overall course. Items the learners rated most highly included relevance of the content to their practices and the prompt response of the content facilitator to their messages. One learner commented on how the content facilitator responded, “Dr. P. made sure no one felt stupid about asking a question, which is very important.” They rated items addressing the bulletin board the lowest. Related comments included, “A very frustrating experience because my computer skills were not advanced enough,” and “Takes a while to get used to the bulletin board.” Thirteen of the 15 respondents indicated that they wished to have more discussion-based on-line modules and would recommend this module to their peers. Supporting comments included, “The bulletin board was great once you got in” and “I think CME on-line will prove to be a Godsend for us rural physicians.”TABLE 1: Mean Ratings of Items Addressing Case Discussion in the Bulletin Board and Overall Course by 15 Physician Participants in a CME Online PBL Program, Dalhousie University, 1999*The bulletin board discussion transcript included 122 messages. The content facilitator posted 46 messages; the educator facilitators, 23; and the 14 learners, a total of 53. The numbers of messages posted per learner ranged from one to seven, with an average of 3.8 messages per learner. The learners posted most messages in the last two weeks of the program. Most entered the bulletin board to post messages only once, although some wrote more than one message at that time. They interacted with the case and facilitators but rarely with each other. Analysis of the bulletin board transcript revealed four themes. These were: content (discussion of the case and questions, 80 messages), facilitative (supportive and encouraging comments, 22 messages), introductory (personal introductions, 16 messages), and administrative/technical (related to technical or logistic issues, four messages). The content expert and the learners posted all the content messages and the educators posted 16 of the 22 facilitative messages. The content facilitator accessed the bulletin board about every second day and responded to each new learner message, giving positive feedback and stimulating critical thinking. He spent a total of about 90 minutes on-line per week. The educator facilitators accessed the bulletin board on alternate days to welcome and encourage learners and note problems. They also spent a total of about 90 minutes on-line per week. Off-line facilitator activities included contacting registrants to encourage participation, monitoring progress, and resolving problems. The content facilitator spent about 30 minutes per week in off-line activities, and the educators, about five hours per week. In addition, the facilitators undertook pre-course activities to encourage participation. These included faxing participants a welcome letter, instructions, and help line information; conducting a teleconference to explain the on-line process; and posting a welcome message in the bulletin board. Learners reported five technical problems to the help line. Four reported difficulty accessing the Web site, and one could not post a message in the bulletin board. Staff responded as promptly as possible and resolved each problem. Discussion Analysis of the on-line discussion confirmed that the anticipated facilitator roles were fulfilled. As content facilitator, the neurologist increased the depth and breadth of the content discussion, and the educator facilitators performed a social and supportive role by welcoming and encouraging learners. However, the neurologist, through his supportive style and prompt responses, also fulfilled a social role, and, in fact, may not have needed the assistance of the educator facilitators. A program in which the content expert is less skilled in PBL facilitation may benefit more by the addition of a skilled facilitator. Encouraging participation was an important role, expanding to off-line activities and requiring more time than anticipated. The on-line administrative/technical role was small, but it was a critical off-line function. Despite these roles there were deficiencies in the PBL discussion. Equal learner participation is a goal of PBL,16 but because most learners entered the discussion in the final week or two and often did not respond to messages, facilitating an ongoing discussion was difficult. Also, as few learners interacted with each other, the discussion was teacher-centered as opposed to learner-centered. A contributing factor may have been the requirement that learners post only one message in the bulletin board to receive credit. Interaction in future modules may be improved by requesting that each participant post messages weekly and respond to co-participants. Barriers to health care professionals' adopting new communications technologies are numerous, and include the lack of adequate technical, economic, organizational and behavioral knowledge. Lessening these barriers requires intensive learning strategies.24 Participating in an Internet discussion requires physicians to both adopt a new technology and change their learning behaviors. When asked what had attracted them to taking this course, nine of the 15 participants indicated “the opportunity to use new technology,” but, with respect to their technical knowledge, seven participants rated their computer skills as “beginner.” Although we provided printed instructions, a help line, and off-line support by educational facilitators, at least two registrants did not participate in this program and another two would not participate in future programs because of technology-related issues. Our findings reinforce the need for educational software that can be easily used by learners who may lack computer proficiency and have little time for or interest in mastering new technology. Providing more extensive training may increase participation, but scheduling this for busy physicians whose time is limited is difficult. This study had several limitations. The study population was small and the learners chose to participate, so it may not represent a larger physician group. Generalizability is also limited by the lack of a control group, and although the evaluation questionnaire demonstrated face validity through a pilot-testing process, we did not test it for reliability. Of the 31 registrants, only 19 (62%) participated in the program. We learned the reasons for non-participation, important data for this study, of only four of the 12 non-participants. Ensuring reliability of the tool before repeating the study, replicating it with other populations, and being more aggressive in contacting non-participant registrants would strengthen future similar studies. In spite of limitations, this study provides insight into facilitators' roles in on-line PBL discussions and factors influencing learners' participation. It supports the view that on-line facilitators perform several roles on-line and off-line, and suggests that a challenge for facilitating PBL discussions is to promote ongoing learner-learner interaction as opposed to one-time learner-teacher interaction. Current technology hinders participation, while prompt and supportive responses by facilitators to learners' messages encourage it. All but two of the 15 learners completing the evaluation said that they would like to have more modules, indicating that the benefits outweighed the disadvantages. Placing this small study within the context of physicians' learning, technology adoption, and behavioral change assists in considering its implications. PBL is an effective CME learning method that uses participant interaction. The Internet is a powerful tool that removes traditional barriers to both physicians' participation in CME and their interaction with co-participants, but it creates new barriers related to technology and behavioral change. We need to learn ways to overcome these barriers, a task that may become easier as communication technologies and software applications improve, and as physicians entering the workforce become more experienced in using computers.