Criterion-based Audit Research Articles

Understanding what works—and why—in quality improvement: the need for theory-driven evaluation

Clinicians who are asked to participate in quality improvement programmes in healthcare organizations are often heard to ask for the evidence that they ‘work’. By that, they often mean they want randomized controlled trials, which show that accreditation, or credentialing, or criterion-based audit, or adverse event monitoring, or continuous quality improvement programmes, or whatever approach is being used cause meaningful and worthwhile improvements in the quality of care [1]. When they learn that there are relatively few experimental studies of quality improvement interventions [2], and those which do exist often show weak or moderate effects at best, this state of affairs is sometimes used to argue that it is not worthwhile investing time and effort in quality improvement. After all, the argument goes, we should not embark on using a new clinical intervention such as a drug or a surgical procedure without solid experimental evidence of its effectiveness, so why should we have a lower threshold for the adoption of organizational interventions like quality improvement programmes? Surely, they too should be proven to ‘work’ before they are adopted or implemented widely? This point of view needs to be challenged. It is founded on a fundamental misunderstanding of the place of experimental methods in investigating and understanding complex social interventions, which is commonplace particularly among clinicians and biomedical researchers and which can seriously hamper those both researching …

Perinatal audit in low- and high-income countries

Audit is a term used to include case reviews, criterion-based clinical audit, enquiries into maternal mortality and perinatal deaths, and near-miss reviews. The audit cycle consists of identifying cases, collecting information, analysing the results, formulating recommendations, implementing change and re-evaluating practice, and this cycle must be repeated regularly. Implicit in the process are standards against which practice is measured. These standards are becoming increasingly explicit and may be based on hospital protocols or regional or national guidelines. When protocols or guidelines are drawn up, this must be on the basis of multidisciplinary discussion and they need to be regularly updated as new evidence emerges. Audit does not need to be expensive, but it does need the support of all staff, including managers and clinicians. Staff must understand that its purpose is not to identify errors and punish mistakes but to improve clinical care.

Audit of local performance compared with standards recommended by the national guidelines for aetiologic investigation of permanent childhood hearing impairment

National guidelines for aetiologic investigation of childhood deafness were developed as the Newborn Hearing Screening Program (NHSP) was being implemented in the United Kingdom. This guidance document was expected to be incorporated into the operational procedure of the NHSP. This criterion-based audit compared local care set against developed guidelines that can be used to assess the appropriateness of specific investigations, services and outcomes. Data on children diagnosed to have sensorineural deafness from March 2002-2004 were extracted from an established computerized database for analysis. Forty-seven children were included; 17 have bilateral severe to profound hearing loss, 25 have bilateral mild to moderate loss and 5 with unilateral loss. A high proportion of Pakistani children were from consanguineous marriages with a family history of deafness. Total 29.8% of children were diagnosed through newborn screening and 70.2% detected through hearing surveillance programmes. For children with bilateral severe to profound deafness, 53.0% accepted, 5.9% declined and 41.2% were not offered imaging of their inner ears. A total of 47.1% accepted and 52.9% declined electrocardiograph (ECG) evaluation. Total 70.6% accepted and 29.4% declined connexin mutations testing. Parental requests were required for those with lesser degree of hearing loss. Total 24% accepted, 28% declined and 48% were not offered connexin testing. None were offered ECG and imaging. Testing for congenital infections was inappropriate for children over 1 year old. Ten subjects accepted and five declined this investigation. In the total group, 63.8% accepted, 17.0% declined and 19.1% were not offered referral to the ophthalmic service. Total 46.8% accepted, 44.7% declined and 8.5% were not offered referral to genetics service. Investigations resulted in two connexin-positive children with moderate loss. Our study identified key areas where guidelines were not followed. These were related to lack of funding and parental choice. This sample has a higher connexin 'hit' rate for lesser degree deafness.

The Exeter Depression Audit Package: pilot study findings

Depression and anxiety are the commonest mental illnesses in primary care. However, the evidence indicates that general practitioner detection, assessment and management of depression is at best variable and may be suboptimal. This pilot study entailed the development and evaluation of the Exeter Depression Audit Package (EDAP) for the history, examination and management of depression in primary care. Three different audit types were employed for each EDAP section: (1) history, i.e. a retrospective case note audit, (2) examination, i.e. an 'identification index' audit and (3) management, i.e. a criterion-based audit. Each audit was repeated following a practice educational intervention. The subjects in each section included (1) history, i.e. a 5% random sample of case notes, (2) examination, i.e. 50 consecutive patent attenders per doctor and (3) management, i.e. the last 25 patents per doctor diagnosed as newly depressed. The results included (1) history, i.e. an improvement in the proportion of patients with an adequate brief mental state assessment in the presence of risk factors for depression, (2) examination, i.e. an increase in the sensitivity, specificity and positive predictive value of each practice against a diagnosis of depression as measured by a score of 11 or over on the depression subscale of the Hospital Anxiety and Depression Scale and (3) management, i.e. an increase in both practices in the number of depressed patents meeting all but one of six process measurements. In conclusion, this pilot study of the EDAP improved written documentation and produced a non-significant improvement in sensitivity but not specificity in the detection of depression

Testing clinical audit of emergency obstetric care in Ghana and Jamaica

One of the key contributors to the high rate of maternal mortality in developing countries is poor quality emergency obstetric care at referral level. In developed countries criterion-based clinical audit (CBCA) is now routinely used as part of quality assurance in health services. The audit team members who are not medically qualified screen medical records in order to determine whether the care received meets these criteria. The audit team and hospital staff then identify and implement actions to improve practice as required. In order to assess the feasibility and effectiveness of CBCA at district hospital level a study was conducted in two hospitals in Ghana and two in Jamaica. The types of obstetric complications chosen for audit were based on their life-threatening nature unambiguous definition and ability to be identified from hospital registers and patient records. They included obstetric hemorrhage eclampsia uterine rupture obstructed labor and sepsis. After devising working definitions for these conditions using best practice from relevant literature an assessment was made of baseline practice. Meetings were then held with hospital staff to encourage them to identify any poor quality management of the five life-threatening complications and how to improve including the setting of realistic targets for improvement. After 12 months of the action plan a second round of assessments using CBCA was conducted at the hospitals. Overall significant improvements were detected in the management of obstetric hemorrhage eclampsia and genital tract sepsis. Although there was some improvement in care for the other two complications the change was not statistically significant. Evidence of a continuing commitment to improving the quality of care at the four participating hospitals was reflected in the intention of staff to repeat the audit cycle until their targets were met and expand the process to other specialisms. (full text)

Holding up a mirror: changing obstetric practice through criterion-based clinical audit in developing countries

The objective of the study described is to assess the feasibility and effectiveness of using a criterion-based clinical audit to measure and improve the quality of obstetric care at the district hospital level in developing countries. The focus is on the management of five life-threatening obstetric complications — hemorrhage, eclampsia, genital tract infection, obstructed labor and uterine rupture was audited using a ‘before and after’ design. The five steps of the audit cycle were followed: establish criteria of good quality care; measure current practice (Review I); feedback findings and set targets; take action to change practice; and re-evaluate practice (Review II). Systematic literature review, panel discussions and pilot work led to the development of 31 audit criteria. Review I included 555 life-threatening complications occurring over 66 hospital-months; Review II included 342 complications over 42 hospital-months. Many common areas for improvement were identified across the four hospitals. Agreed mechanisms for achieving these improvements included clinical protocols, reviews of staffing, and training workshops. Some aspects of clinical monitoring, drug use and record keeping improved significantly between Reviews I and II. Criterion-based clinical audit in four typical district hospitals in Ghana and Jamaica is a feasible and acceptable method for quality assurance and appears to have improved the management of life-threatening obstetric complications.

Discharge and follow-up for people with stroke: what happens and why.

To assess the quality of the process of discharge from hospital and follow-up services for people with stroke. A criterion-based process audit and basic outcome measures, combined with surveys of patients' satisfaction and staff opinion of the service. All units treating stroke patients in a health care district including an acute and a community NHS trust, and 23 participating GP practices. Process audit: documented notes of 98 stroke patients admitted and discharged over a four-month period. Patient satisfaction survey: 93 surviving stroke patients. Staff opinion survey: general practitioners, hospital doctors, therapists and nurses treating stroke patients throughout the district. A poor level of service was found. The main shortcomings were poor communication and liaison and a narrow focus of rehabilitation which concentrated on the assessment and provision of basic home care and activities of daily living (ADL) required to obtain discharge. There was a paucity of provision beyond this most basic level and little follow-up after discharge. Pass rates against agreed criteria were: communication between staff and patients/carers 47%, liaison between staff 44%, assessment of home-based needs 48%, assessment of domestic skills 15.5%. Fifty-one per cent of patients were referred for follow-up therapy and of these 72% started follow-up therapy within six weeks of discharge, only 27% had any follow-up assessment of activity levels and well-being. Patients were dissatisfied with the information, support services and therapy they received. The main reasons for the shortcomings were lack of awareness of the services provided, professionals' low expectations of patients' abilities, and limitations of community-based therapy services. Evidence from other publications suggests that these results do not indicate a service that is any worse than other districts, rather it represents the poor deal offered to stroke patients. By comprehensively assessing several aspects of the service together this methodology has been able to reveal these inadequacies and the reasons for them.

A national audit to monitor and promote the uptake of clinical guidelines on the management of diabetes in pregnancy.

Seeks to assess maternity care for women with Type 1 diabetes in relation to recommendations in a national clinical guideline using a criterion-based clinical audit. The audit covered all 22 consultant-led maternity units in Scotland, focusing on 268 completed pregnancies in women with Type 1 diabetes. Results are presented and discussed. Concludes that a national audit to monitor the impact of clinical guidelines proved feasible. Antenatal care is organised in line with guideline recommendations but there is lower provision of formal prepregnancy care. Pregnancy planning and periconceptual care fall short of recommendations but care during pregnancy is meticulous. Adverse pregnancy outcomes remain commoner than in non-diabetic pregnancies.

Criteria for clinical audit of the quality of hospital-based obstetric care in developing countries.

Improving the quality of obstetric care is an urgent priority in developing countries, where maternal mortality remains high. The feasibility of criterion-based clinical audit of the assessment and management of five major obstetric complications is being studied in Ghana and Jamaica. In order to establish case definitions and clinical audit criteria, a systematic review of the literature was followed by three expert panel meetings. A modified nominal group technique was used to develop consensus among experts on a final set of case definitions and criteria. Five main obstetric complications were selected and definitions were agreed. The literature review led to the identification of 67 criteria, and the panel meetings resulted in the modification and approval of 37 of these for the next stage of audit. Criterion-based audit, which has been devised and tested primarily in industrialized countries, can be adapted and applied where resources are poorer. The selection of audit criteria for such settings requires local expert opinion to be considered in addition to research evidence, so as to ensure that the criteria are realistic in relation to conditions in the field. Practical methods for achieving this are described in the present paper.

The process of stroke rehabilitation: what happens and why.

To assess the process of hospital-based stroke rehabilitation. A criterion-based process audit, combined with surveys of patient and staff opinions and basic outcome measures. All wards and units treating stroke patients in a health care district, including an acute and a community National Health Service (NHS) trust. Process audit: documented notes of 115 stroke patients admitted over a four-month period. Patient satisfaction survey: 93 surviving stroke patients. Staff opinion survey: Hospital doctors, therapists and nurses treating stroke patients throughout the district. A disappointingly poor level of service. The main shortcomings were poor assessment of impairment (pass rate, 46%), inadequate communication between staff and with patients and carers (pass rate, 43%), and an absence of rehabilitation beyond the basic of activities of daily living and indoor mobility (pass rate for assessment of disability and emotional need, 50%). Thirty-three per cent of patients were dissatisfied with the hospital-based service they received, particularly lack of therapy, information and recovery. The main reasons for these shortcomings were low priority given to stroke patients, lack of time, shortage of staff, and lack of knowledge and awareness of stroke amongst the staff. Rehabilitation units and elderly care tended to perform better than general medical units in areas of assessment of impairment, self-care skills and mobility, and wheelchair provision. Evidence from previous publications suggests that this service was no worse than in other districts, but this audit methodology, by comprehensively examining many aspects of a service together, is better able to reveal inadequacies.

Southampton Stroke Audit: assessing service quality

This paper describes and discusses a methodology developed to comprehensively assess the quality of stroke services, the problems faced and how they were overcome. Stroke management was defined and used to carry out a criterion-based process audit, which was combined with surveys of patient satisfaction and the staff's view of the service they provided.

Nationwide audit of pill knowledge amongst family planning users in Scotland

Objectives The primary objective of this study was to estimate knowledge about oral contraceptives amongst oral contraceptive users within family planning clinics in Scotland and to ascertain if this was due to clinicians not attempting to convey certain information or to a lack of understanding of teaching.Method This was a criterion-based audit using three separate questionnaires to estimate the agreement of senior staff with criteria set by a multi-disciplinary expert panel, actual routine clinical practice and user knowledge.Results Senior clinical staff within family planning clinics in Scotland agreed with 12 out of 15 criteria set by a multidisciplinary panel in over 85% of cases. For six out of 15 criteria, there was a discrepancy of more than 30% in what clinicians did in practice compared to what senior staff thought they ought to do. For a further two criteria, there was a deficiency of over 30% between the number of clients who understood the criteria and the number of clients the clinicians thought they had taught. Most importandy, these latter criteria included the rules for safe and effective pill taking.Conclusion An improvement in user knowledge is required to achieve effective and reliable use of oral contraceptives. Methods of doing this, such as staff and client prompts, should be further explored.

A repeated criterion-based audit of radiographic usage by a paediatric dentistry unit.

To report a simple clinical audit which was conducted within a single hospital paediatric dental unit and to demonstrate the potential value of such small-scale projects. A criterion-based audit of radiographic practice which was repeated after 3 years in order to evaluate changes. Six 1 hour meetings of an audit group consisting of the staff and postgraduate students of the Unit. Various aspects of radiographic usage were quantitatively assessed using a sample of patient records and a pre-planned check list to assess the extent to which agreed good-practice criteria were being met. The initial audit identified areas for improvement which were addressed before repeating the exercise 3 years later. Clinical note-keeping to accompany radiographic examinations was found to be the least satisfactory area of practice. When the exercise was repeated, encouraging improvements were demonstrated. The audit was worthwhile because it appeared to produce improvements in standards of patient care.

The use of hormone replacement therapy in women with acute myocardial infarction: an audit of current practice.

We undertook criterion-based audit of the current practice of prescribing hormone replacement therapy for women with acute myocardial infarction; the audit included 181 consecutive women admitted to one hospital with this diagnosis in one calendar year. The set standard was that, barring any contraindication, all postmenopausal women with acute myocardial infarction should be prescribed hormone replacement therapy before discharge from hospital. The evidence base of this standard derives from more than 30 epidemiological and clinical studies and a large body of biological data. Only 4.7% of the women were current users of hormone replacement therapy and the set standard was met in only 3% of eligible nonusers. Professionals caring for women who have had a myocardial infarction need to consider hormone replacement therapy as a secondary prophylaxis of myocardial infarction. Gynaecologists should liaise with colleagues in other specialties and general practice to ensure that information on the nongynaecological benefits of hormone replacement therapy is widely disseminated.

Comparison of the process of care of acute severe asthma in adults admitted to hospital before and 1 yr after the publication of national guidelines

This study set out to assess the effect of publication of the British Guidelines on Asthma Management on the processes and outcomes of the inpatient care of acute severe asthma in the U.K. A criterion-based audit of all acute asthma admissions during August and September 1990 (immediately before) and in 1991 (1 yr after publication of the Guidelines) using eight criteria of process and outcome was performed. Thirty-six teaching and district general hospitals in England, Scotland and Wales took part. In total, 766 patients admitted in 1990, and 900 patients admitted in 1991, were studied. The 1990 and 1991 cohorts were very similar demographically and had asthma of comparable severity. Respiratory physicians achieved similar high performance rates of between 75 and 91% for seven of the eight criteria for both years. Respiratory physicians were significantly more likely to provide patients with a written management plan in 1991. General physicians' performance was significantly lower in both years, but overall there was a very small, but just significant, improvement in their performance in 1991. Some hospitals performed consistently well in both years. It is concluded that respiratory physicians consistently provide better asthma care than general physicians. Though statistically significant, the small degree of improvement was disappointing. Possible reasons include: insufficient time for the Guidelines to be incorporated into practice; inaccessibility of the Guidelines to general physicians; failure to accept responsibility for implementing the good practice reflected in the Guidelines; and an explicit need for strategies to implement the Guidelines beyond publication in a widely-read general medical journal.

Criterion-based audit of epilepsy in both community and hospital settings in seven districts

Criterion-based audit of epilepsy in both community and hospital settings in seven districts