Significant Event Analysis Research Articles

A review of the current evidence base for significant event analysis

To review the literature on the perceived benefits and disadvantages associated with significant event analysis (SEA) and identify reported barriers and facilitating factors. A comprehensive search of electronic databases and peer reviewed journals was conducted during June 2006. Studies which explored or measured perceptions or attitudes in relation to SEA or assessed its impact on health care quality were included. 27 studies were identified with most undertaken in UK general practice. Perceived benefits include: improved communication, enhanced team-working and awareness of others' contributions. SEA has a strong emotional resonance which may lead to a greater commitment to change. Multiple but unverifiable changes in practice and improvements in service quality were reported through participation. Disadvantages include concerns about litigation, reprisal, embarrassment and confidentiality. The reliability of SEA is questioned because it lacks a robust, standard structured method. Evidence of its impact on health care is severely limited. Barriers include a lack of training, poor team dynamics, failings in facilitation and leadership, selective topic choice and associated emotional demands. Facilitating factors include: effective practice in meetings; protected meeting time; a structured methodical approach; and strong team dynamics and leadership. A chasm exists between the high expectations for SEA and the lack of evidence of its impact. SEA may have some merit as a team-based educational tool. However, it may not be a reliable technique for investigating serious or complex safety issues in general practice. Policy makers need to be more explicit about the actual purpose of SEA.

The reliability of workplace-based assessment in postgraduate medical education and training: a national evaluation in general practice in the United Kingdom

To investigate the reliability and feasibility of six potential workplace-based assessment methods in general practice training: criterion audit, multi-source feedback from clinical and non-clinical colleagues, patient feedback (the CARE Measure), referral letters, significant event analysis, and video analysis of consultations. Performance of GP registrars (trainees) was evaluated with each tool to assess the reliabilities of the tools and feasibility, given raters and number of assessments needed. Participant experience of process determined by questionnaire. 171 GP registrars and their trainers, drawn from nine deaneries (representing all four countries in the UK), participated. The ability of each tool to differentiate between doctors (reliability) was assessed using generalisability theory. Decision studies were then conducted to determine the number of observations required to achieve an acceptably high reliability for "high-stakes assessment" using each instrument. Finally, descriptive statistics were used to summarise participants' ratings of their experience using these tools. Multi-source feedback from colleagues and patient feedback on consultations emerged as the two methods most likely to offer a reliable and feasible opinion of workplace performance. Reliability co-efficients of 0.8 were attainable with 41 CARE Measure patient questionnaires and six clinical and/or five non-clinical colleagues per doctor when assessed on two occasions. For the other four methods tested, 10 or more assessors were required per doctor in order to achieve a reliable assessment, making the feasibility of their use in high-stakes assessment extremely low. Participant feedback did not raise any major concerns regarding the acceptability, feasibility, or educational impact of the tools. The combination of patient and colleague views of doctors' performance, coupled with reliable competence measures, may offer a suitable evidence-base on which to monitor progress and completion of doctors' training in general practice.

Workplace-based assessment for general practitioners: using stakeholder perception to aid blueprinting of an assessment battery

The implementation of an assessment system may be facilitated by stakeholder agreement that appropriate qualities are being tested. This study investigated the extent to which stakeholders perceived 8 assessment formats (multiple-choice questions, objective structured clinical examination, video, significant event analysis, criterion audit, multi-source feedback, case analysis and patient satisfaction questionnaire) as able to assess varying qualities of doctors training in UK general practice. Educationalists, general practice trainers and registrars completed a blueprinting style of exercise to rate the extent to which each evaluation format was perceived to assess each of 8 competencies derived primarily from the General Medical Council document 'Good Medical Practice'. There were high levels of agreement among stakeholders regarding the perceived qualities tested by the proposed formats (G = 0.82-0.93). Differences were found in participants' perceptions of how well qualities were able to be assessed and in the ability of the respective formats to test each quality. Multi-source feedback (MSF) was expected to assess a wide range of qualities, whereas Probity, Health and Ability to work with colleagues were limited in terms of how well they could be tested by the proposed formats. Awareness of the perceptions of stakeholders should facilitate the development and implementation of workplace-based assessment (WPBA) systems. These data shed light on the acceptability of various formats in a way that will inform further investigation of WPBA formats' validity and feasibility, while also providing evidence on which to base educational efforts regarding the value of each format.

The use of significant event analysis and personal development plans in developing CPD: a pilot study

This paper describes the work undertaken by the Postgraduate Primary Care Trust (PCT) Dental Tutor for South Yorkshire and East Midlands Regional Postgraduate Dental Education Office during the first year of a two-year pilot. The tutor has special responsibility for facilitating the writing of Personal Development Plans (PDPs) and the introduction of Significant Event Analysis to the 202 general dental practitioners in the four Sheffield PCTs. Data were collected on significant events and the educational needs highlighted as a result. A hands-on workshop format was used in small practice groups and 45% of Sheffield general dental practitioners now have written PDPs compared with a 16% national average. A library of significant events has also been collated from the data collected.

Exploring students' perceptions on the use of significant event analysis, as part of a portfolio assessment process in general practice, as a tool for learning how to use reflection in learning

BackgroundPortfolio learning enables students to collect evidence of their learning. Component tasks making up a portfolio can be devised that relate directly to intended learning outcomes. Reflective tasks can stimulate students to recognise their own learning needs.Assessment of portfolios using a rating scale relating to intended learning outcomes offers high content validity.This study evaluated a reflective portfolio used during a final-year attachment in general practice (family medicine). Students were asked to evaluate the portfolio (which used significant event analysis as a basis for reflection) as a learning tool. The validity and reliability of the portfolio as an assessment tool were also measured.Methods81 final-year medical students completed reflective significant event analyses as part of a portfolio created during a three-week attachment (clerkship) in general practice (family medicine). As well as two reflective significant event analyses each portfolio contained an audit and a health needs assessment.Portfolios were marked three times; by the student's GP teacher, the course organiser and by another teacher in the university department of general practice. Inter-rater reliability between pairs of markers was calculated. A questionnaire enabled the students' experience of portfolio learning to be determined.ResultsBenefits to learning from reflective learning were limited. Students said that they thought more about the patients they wrote up in significant event analyses but information as to the nature and effect of this was not forthcoming.Moderate inter-rater reliability (Spearman's Rho .65) was found between pairs of departmental raters dealing with larger numbers (20 – 60) of portfolios. Inter-rater reliability of marking involving GP tutors who only marked 1 – 3 portfolios was very low.Students rated highly their mentoring relationship with their GP teacher but found the portfolio tasks time-consuming.ConclusionThe inter-rater reliability observed in this study should be viewed alongside the high validity afforded by the authenticity of the learning tasks (compared with a sample of a student's learning taken by an exam question). Validity is enhanced by the rating scale which directly connects the grade given with intended learning outcomes.The moderate inter-rater reliability may be increased if a portfolio is completed over a longer period of time and contains more component pieces of work.The questionnaire used in this study only accessed limited information about the effect of reflection on students' learning. Qualitative methods of evaluation would determine the students experience in greater depth. It would be useful to evaluate the effects of reflective learning after students have had more time to get used to this unfamiliar method of learning and to overcome any problems in understanding the task.

Learning from analysis of significant events

Significant event analysis is an important tool for continuing professional development

Variations in the ability of general medical practitioners to apply two methods of clinical audit: a five‐year study of assessment by peer review

Clinical audit has a central role in the NHS clinical governance agenda and the professional appraisal of medical practitioners in the UK. However, concerns have been raised about the poor design and impact of clinical audit studies and the ability of practitioners to apply audit methods. One method of making informed judgements on audit performance is by peer review. In the west of Scotland a voluntary peer review model has been open to general practitioners since 1999, while general practice trainees are compelled to participate as part of summative assessment. The study aimed to compare the outcomes of peer review for two methods of audit undertaken by different professional and academic groups of doctors. Participants submitted a criterion audit or significant event analysis in standard formats for review by two informed general practitioners (GPs) using appropriate instruments. Peer review outcome data and the professional status of doctors participating were generated by computer search. Differences in proportions of those gaining a satisfactory peer review for each group were calculated. Of 1002 criterion audit submissions, 552 (55%) were judged to be satisfactory. GP registrars were significantly more likely than GP trainers (P < 0.001) and other established GP groups (P < 0.001) to gain a satisfactory peer review. GPs in non-training practices were less likely to achieve a satisfactory review than registrars (P < 0.001) and colleagues in training practices (P < 0.001). Of 883 SEA submissions, 541 (65%) were judged as satisfactory, with all groups gaining a similar proportion of satisfactory assessments, although GP registrars may have outperformed non-training practice GPs (P = 0.05). A significant proportion of GPs may be unable to adequately apply audit methods, potentially raising serious questions about the effectiveness of clinical audit as a health care improvement policy in general medical practice.

A pilot study of the feasibility of trainee hospital doctors undertaking significant event analysis

Participating in significant event analysis is proposed as an important method of reflective learning that can enhance patient care and safety. Peer review is one way of making informed judgments on the ability of participants to apply the technique.

Evaluation and development of integrated teams: The use of Significant Event Analysis

Significant Event Analysis (SEA) is a practical, problem-solving technique that can help teams learn from their experiences and generate practical solutions to complex issues. It has mainly been conducted in Primary Health Care Teams (PHCTs) with the use of an external facilitator. Although SEA is envisioned to address issues of integration and complement routine audit, there is limited evidence that this has been achieved. This paper reports on a 1 day workshop during which team members were trained to use SEA to address issues of service integration. This paper presents an evaluation of the workshop in order to determine if SEA is a feasible tool to support and encourage team self-evaluation without external facilitation. The main barriers to SEA are discussed as well as the need for longitudinal research to assess the impact of SEA on integration of the team and wider service.

Developing an evidence base for patient and public involvement

This paper summarises five years' experience of patient and public involvement in primary care, citing examples from the Lanark practice and Clydesdale Local Health Care Co‐operative (LHCC) in Lanarkshire, Scotland. Strategic development and models which align primary care structures within a framework for patient and public involvement are described, along with barriers to implementation. Examples derived from clinical governance, health promotion and needs assessment include patient and carer involvement in significant event analysis and audit, joint training and patient‐held record cards. Positive outcomes reported are effective dialogue between health professionals, patients and the public, service developments and quality improvements. The success of initiatives is retrospectively assessed against the Audit Commission's critical success factors.

Using Significant Event Analysis in Dental and Medical Education

As part of a new professional development teaching strand at the University of Otago's School of Dentistry, final-year dental students were each asked to write a "thought-provoking episode report" (TPER). These TPERs formed the basis for group discussion in a professional development course. This article outlines the main content themes of the reports, comparing them with similar reports written by medical students from the same university. While both sets of reports demonstrate students' commitment to high standards of care, there were significant differences in the overall themes. Of the fifty-one dental TPERs, the main themes were "difficult" patients, receiving conflicting advice from different clinical tutors, friends as patients, belittlement, and maintenance of professional standards. Key themes from medical students' TPERs included responding to patient suffering, observing or experiencing belittlement, uncertainty, error and complaints, the role of the undergraduate student, treating family and friends, causing distress in order to learn, and issues with mentoring. Possible interpretations of the similarities and differences in this data include confirmation of the value of personal experience as a basis for student discussion, the necessity for formal programs in professional development, and the presence of a hidden curriculum in dental and medical education.

Conflict and coping strategies: a qualitative study of student attitudes to significant event analysis.

To explore the attitudes of students to reflection in the form of significant event analysis (SEA), to categorise the factors influencing these attitudes and to develop a conceptual framework from the data to help ensure future students have a positive experience of this method of learning. In May 1999, SEA, based on the critical incident technique, was introduced as part of coursework within a 3-week general practice clinical rotation for fourth year students at the School of Medicine, Imperial College, London. Four focus groups of fourth year students were conducted. From these, a topic guide for use in in-depth interviews was developed. Eighteen interviews were conducted, which were recorded, transcribed and analysed. Themes grounded in the data were developed. Dominant themes emerged from the analysis. The process of SEA evoked conflicts within the students. The sources of conflict were categorised as internal, relating to phenomena specific to the student in question, or external, such as the medical curriculum and the student's relationships with others. Coping strategies employed by students and teachers also emerged from the data. Significant event analysis has the potential to provoke a number of conflicts within the student, which may reduce students' engagement with and perception of the utility of the task. By employing coping strategies, the negative effect of the conflict can be minimised. Applying this conceptual framework may inform further initiatives to promote reflective practice in undergraduate education.

Twelve tips for promoting significant event analysis to enhance reflection in undergraduate medical students

The facilitation of reflection and development of reflective abilities are increasingly considered to be an important component of professional development (Eraut, 1994). It is known that students find the process of reflection difficult and that it does not come naturally to all, requiring a safe trusting environment in which students can develop with staff support (Woodward, 1998). The structured and deliberate review of significant events has been advocated as a useful way to encourage reflection (Brookfield, 1990). These tips are based on recent research, which revealed that students' difficulties with significant event analysis arise from a range of unforeseen emotional reactions or conflicts. We pass on our tips for minimizing these conflicts and enhancing the reflective and creative aspects of significant event analysis.

Terminal care: improving teamwork in primary care using Significant Event Analysis.

Significant Event Analysis can be used to improve the effectiveness of a Primary Health Care Team caring for terminally ill patients. A method of doing this is described in detail. Positive team qualities were confirmed and enhanced. Significant areas for improvement were identified: insufficient information had been given to some key workers, changes in medication had been poorly planned, continuity of care, and emotional and professional support of team members were inadequate. A strategy for change was developed and implemented. Performance was reviewed a year later. It was found that the Significant Event Analysis had caused changes in team practices, and was valid and reliable enough to be used as a tool for improving teamwork in primary care in future.

Turning anecdotes into data--the critical incident technique.

This article describes the critical incident technique and offers practical guidance on how the technique can be used to investigate dilemmas in medicine. It extols the particular virtue of this research method for general practice that it can be used to exploit the natural tendency of doctors to tell anecdotes. How the principles of the technique relate to significant event analysis, a form of medical audit, is also discussed.

On the implementation of disturbance analysis systems for detection and diagnosis of safety-related transients in a nuclear power plant

Considerable interest has developed in the last few years in the possible applications of a computerized disturbance analysis system (DAS) to nuclear power plant fault detection and diagnosis. The present paper, after introducing some general aspects of the DAS methodology, focuses specifically on the problems associated with its application to safety related malfunctions and transients. An approach called ‘significant event analysis’ is proposed for possible application when considerable diagnostic depth is deemed necessary from the DAS. A fairly detailed application of this approach to a particular class of disturbances potentially associated with power supply systems is then described. Finally, consideration is given to the relation between the specific types of failures actually recorded in electrical systems and the potential benefits from implementation of a DAS in those systems.