Proficiency Testing Performance Research Articles

Proficiency Testing Performance in Physician’s Office, Clinic, and Small Hospital Laboratories, 1994–2004

Background CLIA 88 was fully implemented for profiency testing (PT) by 1994, but, to date, few studies have examined the long-term impact of PT on performance in laboratories. Methods Failure rates for selected chemistry, hematology, and microbiology analytes were monitored periodically from 1994–2004 to evaluate proficiency test performance. Results Failure rates for chemistry and hematology analytes declined significantly during the 10-year period. Failure rates for microbiology analytes also declined but remained above 5% in 2004 for positive genital/GC cultures, positive urine cultures, and Gram stains. Performance of chemistry and hematology tests improved significantly during the study period. Microbiology analytes also showed significant improvement, but certain tests remained problematic. Conclusion The data indicates that statistics for unsatisfactory laboratory performance may fail to detect significant problems.

Proficiency Testing Performance in Physician's Office, Clinic, and Small Hospital Laboratories, 1994-2004

CLIA 88 was fully implemented for profiency testing (PT) by 1994, but, to date, few studies have examined the long-term impact of PT on performance in laboratories. Failure rates for selected chemistry, hematology, and microbiology analytes were monitored periodically from 1994–2004 to evaluate proficiency test performance. Failure rates for chemistry and hematology analytes declined significantly during the 10-year period. Failure rates for microbiology analytes also declined but remained above 5% in 2004 for positive genital/GC cultures, positive urine cultures, and Gram stains. Performance of chemistry and hematology tests improved significantly during the study period. Microbiology analytes also showed significant improvement, but certain tests remained problematic. The data indicates that statistics for unsatisfactory laboratory performance may fail to detect significant problems.

The Effect of Instrumental Music Participation and Socioeconomic Status on Ohio Fourth-, Sixth-, and Ninth-Grade Proficiency Test Performance

This study is a comparison of the Ohio Proficiency Test (OPT) results of instrumental music students and their noninstrumental classmates according to socioeconomic status (SES) over time. Subjects ( N= 15,431) were students in the Columbus Public Schools in Ohio, whose fourth-, sixth-, and ninth-grade OPT results were compared with others of like SES on the subjects of citizenship, math, science, and reading. Results show that instrumental students outperformed noninstrumental students in every subject and at every grade level. Instrumental students at both levels of SES held higher scores than their noninstrumental classmates from the fourth grade, suggesting that instrumental music programs attract higher scorers from the outset of instruction. Results also show a pattern of increased achievement by lower SES instrumental students, who surpassed their higher SES noninstrumental classmates by the ninth grade in all subjects. September 15, 2005 March 20, 2006

The Quality of Laboratory Testing Today

To assess the analytic quality of laboratory testing in the United States, we obtained proficiency testing survey results from several national programs that comply with Clinical Laboratory Improvement Amendments (CLIA) regulations. We studied regulated tests (cholesterol, glucose, calcium, fibrinogen, and prothrombin time) and nonregulated tests (international normalized ratio [INR], glycohemoglobin, and prostate-specific antigen [PSA]). Quality was assessed on the sigma scale with a benchmark for minimum process performance of 3 sigma and a goal for world-class quality of 6 sigma. Based on the CLIA criteria for acceptable performance in proficiency testing (allowable total errors [TEa]), the national quality of cholesterol testing (TEa = 10%) estimated sigma values as 2.9 to 3.0; glucose (TEa = 10%), 2.9 to 3.3; calcium (TEa = 1.0 mg/dL), 2.8 to 3.0; prothrombin time (TEa = 15%), 1.8; INR (TEa = 20%), 2.4 to 3.5; fibrinogen (TEa = 20%), 1.8 to 3.2; glycohemoglobin (TEa = 10%), 1.9 to 2.6; and PSA (TEa = 10%), 1.2 to 1.8. The analytic quality of laboratory tests requires improvement in measurement performance and more intensive quality control monitoring than the CLIA minimum of 2 levels per day.

The Effect of Instrumental Music Participation and Socioeconomic Status on Ohio Fourth-, Sixth-, and Ninth-Grade Proficiency Test Performance

The Effect of Instrumental Music Participation and Socioeconomic Status on Ohio Fourth-, Sixth-, and Ninth-Grade Proficiency Test Performance

Proficiency testing performance: a case study with modeling

Objectives: Previous literature has approached proficiency testing (PT) performance by defining the minimum levels, and combinations of imprecision and bias, necessary to meet PT requirements. In this case report, current PT performance was assessed and modeling performed to prioritize our quality improvement efforts. Methods: A total of 1006 chemistry challenge results from Ontario’s Laboratory Proficiency Testing Program (LPTP, now QMPLS) performed on 69 tests during 1999 and 2000 were used for this retrospective analysis. Peer group means, all method means and results from reference labs were used for comparison. QMPLS flagging and recommended performance criteria were compiled, and modeling performed to predict different levels of performance. Results: Our internal imprecision is <5% for 72% of our 69 tests; however, only 20% of our tests had a CV/PT <25%. Of the 1006 challenges performed, 136 (13.5%) results were outside PT limits, 55 (5.5%) results were flagged, and 12 requests were received from QMPLS seeking clarification on 24 (2.4%) results. Follow-up identified 9 (38%) nonanalytical errors, 8 (33%) method bias errors, 4 (17%) random errors, 2 poor methods, and one with no error identified. Modeling predicted flagging rates of 2.4% using QMPLS recommended precision performance, 1.6% using our current internal imprecision, 2.2% or 7.0% if we included an overall 20% or 50% relative bias rate with our current imprecision levels, or 15.0% when an estimate of our actual bias for each analyte was considered along with our current imprecision levels. Conclusions: If imprecision were the only cause of PT errors, our flagging rate for this study period would be 1.6%, and we would need to formally investigate 8 results a year. In practice, strict application of the QMPLS PT criteria would result in 68 investigations annually; however, judicial review of the results before request for clarification significantly reduced this number to 12 investigations (of which 38% were nonanalytical errors). At the present time bias is a significant cause of poor PT performance in a variety of assays. Individual laboratories need to address the problem of bias, and ultimately so do manufacturers. It would be helpful if PT programs also acknowledged this necessary evolution in both their criteria and processes.

Factor transformation to produce statistics describing the uncertainty of analytical data.

Analytical uncertainty produced by random error has a positively skewed distribution and accuracy and precision have non-linear scales. Compared with conventional statistics, factor transformation of the data allows more appropriate interpretation of results and facilitates graphical inspection of data. Statistics are compared for practical examples of performance in proficiency tests and of repeatability and reproducibility in collaborative studies. Factor transformation is shown to be applicable to wide ranges of analyte concentration and measurement precision.

Causes of Unsatisfactory Performance in Proficiency Testing

Proficiency testing (PT) provides a measure of the effectiveness of laboratory quality assurance programs. Test reports are released from processes that the laboratory judges to be in conformance with quality specifications; an evaluation of unsatisfactory performance (UNSAT) by a PT provider is an unexpected outcome for the laboratory. An understanding of the root cause(s) of testing errors provides an opportunity for the continuous improvement of laboratory services. We used participant data from the New York State Department of Health PT program to characterize the quality of testing in the toxicology specialty. Outcomes from laboratory investigations into causes of UNSAT and information on quality control practices collected from all program participants were used to identify the root causes of error. Two classes of error were encountered: spurious test results caused by lapses in standard operating procedures and instrument malfunctions (300 per million assays) and common-cause analytic error (7000 per million assays or 0.7% rate of UNSAT). Causes of spurious results included inaccurate mathematical correction for specimen dilution, misinterpretation of instrument codes, and instrument sampling errors. Calibration drift was most frequently cited as the common-cause analytic error. Approximately one-half of the laboratories used an allowable error for the quality control of analytical systems that exceeded the threshold error specified by manufacturers for stable instrument performance. The causes of spurious results suggest the need for ongoing competency testing of analysts where analyst intervention is required in an otherwise automated process, and for continued diligence in mistake-proofing instrument design. The intrinsic quality of laboratory testing is unlikely to improve until the allowable error in quality control is consistent with manufacturer specifications for stable system performance.

Adding Value to Proficiency Testing Programs

The Clinical Laboratory Improvement Amendments of 1988 (CLIA 88) (1) have caused great changes not only in US laboratories but in clinical laboratories throughout the world. CLIA’s requirements for pre- and postanalytical control were harbingers of more complete approaches to the implementation of quality systems in clinical laboratories, such as the quality model just introduced by the National Committee for Clinical Laboratory Standards (2). CLIA’s maximum allowable error specifications for proficiency testing (the so-called CLIA limits) are being used by manufacturers to help set analytical performance goals for new laboratory analyzers. These specifications have ignited a transatlantic debate (3) regarding the formulation of analytical goals based strictly on patient variation vs the CLIA goals, which were based on a host of variables, including physician surveys, patient variation, analytical performance, and even prior proficiency testing (PT) performance. The enactment of CLIA 88 unraveled the comfortable existence of a few large PT providers who primarily served hospital laboratories. In the early 1990s, thousands of previously unregulated physician office laboratories were added to the PT pool. Many of these laboratories eventually subscribed to newly established PT providers whose missions were more closely aligned to the participants’ medical organizations, e.g., the American Academy of Family Physicians and the American Society of Internal Medicine. As of 1998, there were 20 different CLIA-certified PT programs (4). The presumably decreased profitability of the PT business has led to a commoditization of PT products and less effort expended in the design and manufacture of PT specimens for analytes that are not directly regulated by CLIA. Another result, coincident with the enactment of CLIA 88, was the development of PT programs that serve a narrow spectrum of users, such as those using instruments of a specific manufacturer. Finally, the requirement to subscribe to CLIA-approved PT programs has eroded the …

The correlation of laboratory performance in proficiency testing with other QA characteristics

Although it seems self-evident that proficiency testing (PT) and accreditation can be expected to improve quality, their relative benefits remain uncertain as does their efficacy. The study reported here examines the following issues: (a) Why do laboratories take part in PT schemes? (b) How does participation in PT fit in with a laboratory's overall quality assurance (QA) system? (c) Is there a link between a laboratory's performance in specific PT and it's QA system? (d) How does PT performance change with time and how do laboratories respond to poor performance? The overall conclusion is that there is no evidence from the present study that laboratories with third-party assessment (accreditation and certification) perform any better in PT than laboratories without. The validity of this conclusion and its significance for the future design and operation of such schemes requires further investigation. In particular, study is required of the degree to which good performance in open PT correlates with blind PT performance, where laboratories are not aware that the samples being analysed are part of a quality assessment exercise.

Requirements for accreditation by the College of American Pathologists Laboratory Accreditation Program.

The College of American Pathologists Laboratory Accreditation Program expects a participant laboratory or laboratory section to be able to demonstrate that it is in compliance with the Standards for Laboratory Accreditation. The program expects laboratories to demonstrate that they are continually taking steps to identify and correct deficient areas and improve performance, in compliance with the Clinical Laboratory Improvement Amendments of 1988 regulatory requirements, particularly those pertaining to proficiency testing performance, and participating as inspectors in the accreditation process.

Is accreditation useful for quality improvement?

The Laboratory of the Government Chemist (UK) and the Institute for Reference Measurements and Materials (Belgium) evaluated the correlation between accreditation and performance in proficiency tests. It was concluded that accreditation does not have the expected positive effect on the quality of laboratory results. In this journal discussions conducted on this subject during the CITAC workshop at Pittcon in 1998, were published. No satisfactory explanation for this phenomenon was put forward. In this article, it is proposed that the main effect of accreditation is a decrease of intralaboratory spread. The effect on the trueness of laboratory results is not significant due to the lack of certified reference materials and the low frequency of participation in proficiency tests. Proficiency tests cannot be used to find a correlation between accreditation status and quality without changing the set-up of the proficiency test.

Variation in proficiency testing performance by testing site.

Congress enacted the Clinical Laboratory Improvement Amendments of 1988 (CLIA) to promote uniform quality and standards among all testing sites in the United States. The performance indicators specified in the legislation are proficiency testing (PT) performance and periodic inspections. To evaluate variation in PT performance by type of testing facility during the first year of compulsory participation under CLIA. All 1994 PT score data electronically reported to the Health Care Financing Administration as a component of compliance with the CLIA regulations were obtained. Over 1.2 million PT event scores from 17058 unique testing sites were sorted into 2 groups based on the type of testing facility: hospitals and independent laboratories (HI) and all other testing sites (AOT). Satisfactory and unsatisfactory performance rates for HI and AOT for each analyte and/or test, according to the criteria specified by the CLIA regulations. The aggregate rates of satisfactory event performance for all regulated analytes, tests, and specialties were 97% and 91% for the HI and AOTgroups, respectively. The aggregate odds ratio for unsatisfactory PT event performance for the AOT group compared with the HI group was 2.89, with a range of 2.19 to 7.51 for the individual analytes. There was a consistent difference in PT performance during the first full year of compulsory PT under the CLIA regulations based on the type of testing facility performing the analysis. Traditional testing sites achieved higher rates of satisfactory performance than newly regulated, alternative testing sites.

The comparability of sample preparation techniques for the determination of metals in sediments

Satisfactory performance in interlaboratory proficiency tests for the determination of metals in sediments does not necessarily demonstrate that laboratories' data are of adequate comparability. This paper reports a study of the comparability of sample preparation techniques (i.e. drying grinding, sieving etc., prior to sample digestion) for the determination of metals in marine sediments. The results indicate that the particular requirements of the task in hand may need to be emphasised, if adequate comparability is to be achieved. Recommendations for quality control in sample preparation are made.

Sample filtration as a source of error in the determination of trace metals in marine waters.

Adequate performance in interlaboratory proficiency tests using filtered, pre-treated or pre-digested test materials does not necessarily demonstrate that laboratories' data are of adequate comparability. Sample handling can be an important source of error which is not examined by routine proficiency tests. This paper reports a study of sample filtration as a source error in the determination of trace metals in marine water samples. The results indicate that that current practice may need to be reviewed if important contamination errors are to be controlled.

Quality (r)evolution - guidelines and practical implementation of quality management: the U.S. perspective

All clinical laboratories in the United States were required for the first time in 1992 to comply with universal minimum regulations: the Clinical Laboratory Improvement Amendments of 1988 (CLIA'88). The CLIA'88 regulations precipitated revolutionary and evolutionary changes based in both total quality management and continuous quality improvement principles. The regulations specify minimum requirements for personnel, quality control, and proficiency testing. Under quality assurance, the requirements are more individualized and allow laboratories to meet the needs of their customers while being in compliance with the regulations. Biannual inspection is integral to CLIA'88. However, laboratories can choose other federally approved, "deemed" professional organizations, with regulations equivalent to CLIA'88. Selection of one of these organizations allows for some flexibility in the application of the requirements, and eliminates having federal CLIA'88 inspectors perform the inspection. Since CLIA's inception in 1992, inspectors have noted a decline in the number of deficiencies cited. An improved level of proficiency testing performance also indicates that better quality test results are being provided to clinicians. The quality evolution and revolution will continue in the United States, and a major impetus will be health care reform's mandate to reduce cost.

Laboratory Testing in Previously Unregulated Laboratories: Washington State’s Experience

Having implemented its own set of regulations in October 1990, Washington State has data to assess laboratory testing quality in previously unregulated laboratories and to gauge its program’s impact on laboratory testing. In a two-part study, investigators from the Washington Department of Health’s Office of Laboratory Quality Assurance evaluated initial inspection findings and proficiency testing performance for 194 laboratories not previously regulated; 101 of these laboratories were evaluated after a second inspection. On initial inspection, significant differences among personnel categories in the areas of deficiencies cited, proficiency testing participation, and rates of discontinued testing were found. By the second inspection, testing quality indicators had greatly improved, with no significant differences among categories of testing personnel.

Proficiency test performance as a predictor of accuracy of routine patient testing for theophylline

Proficiency testing (PT) is pivotal in assessing laboratory qualifications for certification and licensure. PT is expected to typify routine assay performance and determine whether the laboratory is producing clinically useful test results. Conventional schemes use mail-distributed test specimens and are often criticized as measuring the best possible laboratory performance, principally because of special practices associated with processing PT specimens. We used on-site proficiency tests and split samples to evaluate the ability of conventional PT schemes to accurately characterize routine laboratory performance. Using 412 assays of theophylline, performed routinely by 200 laboratories and subsequently in a reference laboratory, we found that the predictive value of PT performance in assessing quality of routine testing was high (100% for predicting substandard reliability of routine patient testing and 94% for excluding substandard reliability of patient testing). The imprecision of interlaboratory PT results was equivalent whether testing was observed (hand-carried specimens) or unobserved (mail-distributed specimens). Many methods used for determining theophylline concentration in serum were highly automated, closed, and precise analytical systems. The performance characteristics of these analytical systems are not easily manipulated by the analyst for purposes of improving PT outcome, and PT by use of mail-distributed test specimens is effective for assessing intralaboratory performance.

Analyzing oral proficiency test performance in general and specific purpose contexts

To investigate whether a field-specific oral proficiency test, constructed by manipulating test method facets, would be a better predictor of field-specific performance than a general purpose oral proficiency test, 31 Chinese chemistry graduate students were given three English tests: the field specific test, the general purpose test and a chemistry teaching performance test. Results suggested that when raters of the performance test were asked to recommend specifically whether or not a subject should be allowed to actually teach chemistry in a lab or classroom, the field-specific test was a better predictor than the general purpose test. The paper contains a theoretical discussion of field-specific language testing and guidelines for the construction of oral proficiency tests in specific purpose contexts.

Charts of operational process specifications ("OPSpecs charts") for assessing the precision, accuracy, and quality control needed to satisfy proficiency testing performance criteria

"Operational process specifications" have been derived from an analytical quality-planning model to assess the precision, accuracy, and quality control (QC) needed to satisfy Proficiency Testing (PT) criteria. These routine operating specifications are presented in the form of an "OPSpecs chart," which describes the operational limits for imprecision and inaccuracy when a desired level of quality assurance is provided by a specific QC procedure. OPSpecs charts can be used to compare the operational limits for different QC procedures and to select a QC procedure that is appropriate for the precision and accuracy of a specific measurement procedure. To select a QC procedure, one plots the inaccuracy and imprecision observed for a measurement procedure on the OPSpecs chart to define the current operating point, which is then compared with the operational limits of candidate QC procedures. Any QC procedure whose operational limits are greater than the measurement procedure's operating point will provide a known assurance, with the percent chance specified by the OPSpecs chart, that critical analytical errors will be detected. OPSpecs charts for a 10% PT criterion are presented to illustrate the selection of QC procedures for measurement procedures with different amounts of imprecision and inaccuracy. Normalized OPSpecs charts are presented to permit a more general assessment of the analytical performance required with commonly used QC procedures.