Estimates Of Test Performance Research Articles

Accuracy of Tests for Diagnosis of Animal Tuberculosis: Moving Away from the Golden Calf (and towards Bayesian Models)

The last decades have seen major efforts to develop new and improved tools to maximize our ability to detect tuberculosis-infected animals and advance towards the objective of disease control and ultimately eradication. Nevertheless, there is still uncertainty regarding test performance due to the wide range of specificity and especially sensitivity estimates published in the scientific literature. Here, we performed a systematic review of the literature on studies that evaluated the performance of tuberculosis diagnostic tests used in animals through Bayesian Latent Class Models (BLCMs), which do not require the application of a (fallible) reference procedure to classify animals as infected with tuberculosis or not. BLCM-based sensitivity and specificity estimates deviated from those obtained using a reference procedure for certain antemortem tests: an overall lower sensitivity of skin tests and serology and a higher sensitivity of interferon-gamma (IFN-γ) assays was reported. In the case of postmortem diagnostic tests, sensitivity estimates from BLCMs were similar to estimates from studies based on other methodologies. For specificity, the range of BLCM-based estimates was narrower than those based on a reference test, reaching values close to 100% (but lower in the case of IFN-γ assays). In conclusion, Bayesian methods have been increasingly applied for the evaluation of tuberculosis diagnostic tests in animals, yielding results that differ (sometimes substantially) from previously reported test performance in the literature, particularly for in vivo tests and sensitivity estimates. Newly developed models that allow adjustment for relevant factors (e.g., age, breed, region, and herd size) can contribute to the generation of more unbiased estimates of test performance. Nevertheless, although BLCMs for tuberculosis do not require the use of an imperfect reference procedure and are therefore not influenced by its limited performance, they require careful implementation, and transparent systematic reporting should be the norm.

A systematic review and meta-analysis of cell-free DNA testing for detection of fetal sex chromosome aneuploidy.

The aim was to determine the accuracy of cell-free DNA testing (cfDNA) for detecting sex chromosome aneuploidies (SCA) in singleton pregnancies. A systematic review and meta-analysis was performed to assess cfDNA accuracy for prenatal detection of 45,X, 47,XXY, 47,XXX and 47,XYY. Inclusion was restricted to studies published between January 2010 and December 2021 reporting both cfDNA and confirmatory diagnostic test results. For 45,X, the sensitivity was 98.8% (95%CI 94.6%-100%), specificity 99.4% (95%CI 98.7%-99.9%) and positive predictive value (PPV) 14.5% (95%CI 7.0%-43.8%). For 47,XXY, the sensitivity was 100% (95%CI 99.6%-100%), specificity 100% (95%CI 99.9%-100%) and PPV 97.7% (95%CI 78.6%-100%). For 47,XXX, the sensitivity was 100% (95%CI 96.9%-100%), specificity 99.9% (95%CI 99.7%-100%) and PPV 61.6% (95%CI 37.6%-95.4%). For 47,XYY, the sensitivity was 100% (95%CI 91.3%-100%), specificity 100% (95% CI 100%-100%) and PPV 100% (95%CI 76.5%-100%). All four SCAs had estimated negative predictive values (NPV) exceeding 99.99%, though false negatives were reported. This analysis suggests that cfDNA is a reliable screening test for SCA, though both false negatives and false positives were reported. These estimates of test performance are derived from pregnancies at high pretest risk for aneuploidy, limiting the generalisability to average risk pregnancies.

Emerging Multi-Cancer Early Detection Technologies


 Horizon Scan reports provide brief summaries of information regarding new and emerging health technologies; these technologies are identified through the CADTH Horizon Scanning Service as topics of potential interest to health care decision-makers in Canada. This report is not an endorsement or assessment of any test or technology.
 This Horizon Scan summarizes the available information regarding the emerging technology of liquid biopsy–based, multi-cancer early detection tests for cancer screening. This Horizon Scan focuses specifically on the Galleri (GRAIL Inc.) and CancerSEEK (Exact Sciences) tests, which are further along in the development cycle and are being assessed in different international clinical studies.
 Multi-cancer early detection technologies aim to provide a new approach to complement traditional cancer screening programs. These tests examine genetic signals within blood samples with next-generation sequencing and computational algorithms to assess the presence and type of different cancers. Research to date has focused on describing results from training and validation studies that have provided initial estimates of test performance and modelling studies estimating the potential impact on cancer incidence.
 This Horizon Scan also highlights some issues for health care decision-makers to consider about the technology relating to real-world test performance, the potential benefits and harms of screening with multi-cancer early detection tests, and the disruptiveness to health systems they could pose. Ongoing review of clinical trials and the emerging evidence base can help inform health systems in Canada about their potential role within cancer-control initiatives.

Cost-Effectiveness of a Proteomic Test for Preterm Birth Prediction.

BackgroundPreterm birth (PTB) carries increased risk of short- and long-term health problems as well as higher healthcare costs. Current strategies using clinically accepted maternal risk factors (prior PTB, short cervix) can only identify a minority of singleton PTBs.ObjectiveWe modeled the cost-effectiveness of a risk-screening-and-treat strategy versus usual care for commercially insured pregnant US women without clinically accepted PTB risk factors. The risk-screening-and-treat strategy included use of a novel PTB prognostic blood test (PreTRM®) in the 19th–20th week of pregnancy, followed by treatment with a combined regimen of multi-component high-intensity-case-management and pharmacologic interventions for the remainder of the pregnancy for women assessed as higher-risk by the test, and usual care in women without higher risk.MethodsWe built a cost-effectiveness model using a combined decision-tree/Markov approach and a US payer perspective. We modeled 1-week cycles of pregnancy from week 19 to birth (preterm or term) and assessed costs throughout the pregnancy, and further to 12-months post-delivery in mothers and 30-months in infants. PTB rates and costs were based on >40,000 mothers and infants from the HealthCore Integrated Research Database® with birth events in 2016. Estimates of test performance, treatment effectiveness, and other model inputs were derived from published literature.ResultsIn the base case, the risk-screening-and-treat strategy dominated usual care with an estimated 870 fewer PTBs (20% reduction) and $54 million less in total cost ($863 net savings per pregnant woman). Reductions were projected for neonatal intensive care admissions (10%), overall length-of-stay (7%), and births <32 weeks (33%). Treatment effectiveness had the strongest influence on cost-effectiveness estimates. The risk-screening-and-treat strategy remained dominant in the majority of probabilistic sensitivity analysis simulations and model scenarios.ConclusionUse of a novel prognostic test during pregnancy to identify women at risk of PTB combined with evidence-based treatment is estimated to reduce total costs while preventing PTBs and their consequences.

PIH7 Cost-Effectiveness of a Proteomic Test for Preterm Birth Prediction

Preterm birth (PTB) carries increased risk of health problems for infants as well as higher healthcare costs for both infants and mothers. We evaluated the cost-effectiveness of a risk-screening-and-treat strategy, compared to usual care, for a population of commercially-insured pregnant US women without known risk factors for PTB. The strategy included a novel PTB prognostic test (PreTRM®) in the 19th-20th week of pregnancy and treatment with vaginal progesterone and high-intensity case management for the remainder of the pregnancy for women assessed as high-risk. A decision-tree with Markov nodes representing 1-week cycles from week 19 of pregnancy to birth (preterm or full-term) was developed, using a payer’s perspective and time horizon from pregnancy start to 12-months post-delivery in mothers and 30-months from birth in infants. PTB rates and costs were based on real-world cohorts of >40,000 mothers and infants with birth events in 2016, as identified in administrative claims from the HealthCore Integrated Research Database®. Estimates of test performance and treatment effectiveness were derived from published literature. Uncertainty was explored via scenario, one-way, and probabilistic sensitivity analysis (PSA). In the base-case analysis, the risk-screening-and-treat strategy dominated usual care with 870 fewer PTBs (20% reduction) and $54 million less in total cost ($863 net savings per pregnant woman). Reductions were also seen for neonatal-intensive-care-unit admissions (10%), overall length-of-stay (7%), and births <32 weeks gestation (33%). Treatment effectiveness had the most influence on cost-effectiveness estimates per one-way sensitivity analysis, followed by infant care costs and test costs. The risk-screening-and-treat strategy was dominant in the majority of PSA simulations and model scenarios. Use of a novel prognostic test during pregnancy to identify women at risk of PTB combined with evidence-based treatment can reduce total costs and prevent preterm deliveries and their consequences in a representative population of commercially-insured US women.

Deep learning analysis provides accurate COVID-19 diagnosis on chest computed tomography

IntroductionComputed Tomography is an essential diagnostic tool in the management of COVID-19. Considering the large amount of examinations in high case-load scenarios, an automated tool could facilitate and save critical time in the diagnosis and risk stratification of the disease. MethodsA novel deep learning derived machine learning (ML) classifier was developed using a simplified programming approach and an open source dataset consisting of 6868 chest CT images from 418 patients which was split into training and validation subsets. The diagnostic performance was then evaluated and compared to experienced radiologists on an independent testing dataset. Diagnostic performance metrics were calculated using Receiver Operating Characteristics (ROC) analysis. Operating points with high positive (>10) and low negative (<0.01) likelihood ratios to stratify the risk of COVID-19 being present were identified and validated. ResultsThe model achieved an overall accuracy of 0.956 (AUC) on an independent testing dataset of 90 patients. Both rule-in and rule out thresholds were identified and tested. At the rule-in operating point, sensitivity and specificity were 84.4 % and 93.3 % and did not differ from both radiologists (p > 0.05). At the rule-out threshold, sensitivity (100 %) and specificity (60 %) differed significantly from the radiologists (p < 0.05). Likelihood ratios and a Fagan nomogram provide prevalence independent test performance estimates. ConclusionAccurate diagnosis of COVID-19 using a basic deep learning approach is feasible using open-source CT image data. In addition, the machine learning classifier provided validated rule-in and rule-out criteria could be used to stratify the risk of COVID-19 being present.

Evaluation of the methodological quality of studies of the performance of diagnostic tests for bovine tuberculosis using QUADAS

There has been little assessment of the methodological quality of studies measuring the performance (sensitivity and/or specificity) of diagnostic tests for animal diseases. In a systematic review, 190 studies of tests for bovine tuberculosis (bTB) in cattle (published 1934–2009) were assessed by at least one of 18 reviewers using the QUADAS (Quality Assessment of Diagnostic Accuracy Studies) checklist adapted for animal disease tests. VETQUADAS (VQ) included items measuring clarity in reporting (n = 3), internal validity (n = 9) and external validity (n = 2). A similar pattern for compliance was observed in studies of different diagnostic test types. Compliance significantly improved with year of publication for all items measuring clarity in reporting and external validity but only improved in four of the nine items measuring internal validity (p < 0.05). 107 references, of which 83 had performance data eligible for inclusion in a meta-analysis were reviewed by two reviewers. In these references, agreement between reviewers’ responses was 71% for compliance, 32% for unsure and 29% for non-compliance. Mean compliance with reporting items was 2, 5.2 for internal validity and 1.5 for external validity. The index test result was described in sufficient detail in 80.1% of studies and was interpreted without knowledge of the reference standard test result in only 33.1%. Loss to follow-up was adequately explained in only 31.1% of studies. The prevalence of deficiencies observed may be due to inadequate reporting but may also reflect lack of attention to methodological issues that could bias the results of diagnostic test performance estimates. QUADAS was a useful tool for assessing and comparing the quality of studies measuring the performance of diagnostic tests but might be improved further by including explicit assessment of population sampling strategy.

MANAGEMENT OF ENDOCRINE DISEASE: Imaging for the diagnosis of malignancy in incidentally discovered adrenal masses: a systematic review and meta-analysis.

ObjectiveAdrenal masses are incidentally discovered in 5% of CT scans. In 2013/2014, 81 million CT examinations were undertaken in the USA and 5 million in the UK. However, uncertainty remains around the optimal imaging approach for diagnosing malignancy. We aimed to review the evidence on the accuracy of imaging tests for differentiating malignant from benign adrenal masses.DesignA systematic review and meta-analysis was conducted.MethodsWe searched MEDLINE, EMBASE, Cochrane CENTRAL Register of Controlled Trials, Science Citation Index, Conference Proceedings Citation Index, and ZETOC (January 1990 to August 2015). We included studies evaluating the accuracy of CT, MRI, or 18F-fluoro-deoxyglucose (FDG)-PET compared with an adequate histological or imaging-based follow-up reference standard.ResultsWe identified 37 studies suitable for inclusion, after screening 5469 references and 525 full-text articles. Studies evaluated the accuracy of CT (n=16), MRI (n=15), and FDG-PET (n=9) and were generally small and at high or unclear risk of bias. Only 19 studies were eligible for meta-analysis. Limited data suggest that CT density >10HU has high sensitivity for detection of adrenal malignancy in participants with no prior indication for adrenal imaging, that is, masses with ≤10HU are unlikely to be malignant. All other estimates of test performance are based on too small numbers.ConclusionsDespite their widespread use in routine assessment, there is insufficient evidence for the diagnostic value of individual imaging tests in distinguishing benign from malignant adrenal masses. Future research is urgently needed and should include prospective test validation studies for imaging and novel diagnostic approaches alongside detailed health economics analysis.

BAYESIAN ANALYSIS TO EVALUATE TESTS FOR THE DETECTION OF MYCOBACTERIUM BOVIS INFECTION IN FREE-RANGING WILD BISON (BISON BISON ATHABASCAE) IN THE ABSENCE OF A GOLD STANDARD.

We estimated the sensitivity and specificity of the caudal-fold skin test (CFT), the fluorescent polarization assay (FPA), and the rapid lateral-flow test (RT) for the detection of Mycobacterium bovis in free-ranging wild wood bison (Bison bison athabascae), in the absence of a gold standard, by using Bayesian analysis, and then used those estimates to forecast the performance of a pairwise combination of tests in parallel. In 1998-99, 212 wood bison from Wood Buffalo National Park (Canada) were tested for M. bovis infection using CFT and two serologic tests (FPA and RT). The sensitivity and specificity of each test were estimated using a three-test, one-population, Bayesian model allowing for conditional dependence between FPA and RT. The sensitivity and specificity of the combination of CFT and each serologic test in parallel were calculated assuming conditional independence. The test performance estimates were influenced by the prior values chosen. However, the rank of tests and combinations of tests based on those estimates remained constant. The CFT was the most sensitive test and the FPA was the least sensitive, whereas RT was the most specific test and CFT was the least specific. In conclusion, given the fact that gold standards for the detection of M. bovis are imperfect and difficult to obtain in the field, Bayesian analysis holds promise as a tool to rank tests and combinations of tests based on their performance. Combining a skin test with an animal-side serologic test, such as RT, increases sensitivity in the detection of M. bovis and is a good approach to enhance disease eradication or control in wild bison.

Contrast-enhanced ultrasound using SonoVue® (sulphur hexafluoride microbubbles) compared with contrast-enhanced computed tomography and contrast-enhanced magnetic resonance imaging for the characterisation of focal liver lesions and detection of liver metastases: a systematic review and cost-effectiveness analysis

BACKGROUND: Medical imaging techniques are important in the management of many patients with liver disease. Unenhanced ultrasound examinations sometimes identify focal abnormalities in the liver that may require further investigation, primarily to distinguish liver cancers from benign abnormalities. One important factor in selecting an imaging test is the ability to provide a rapid diagnosis. Options for additional imaging investigations include computed tomography (CT) and/or magnetic resonance imaging (MRI) and biopsy when the diagnosis remains uncertain. CT and MRI usually require referral with associated waiting time and are sometimes contraindicated. The use of contrast agents may improve the ability of ultrasound to distinguish between liver cancer and benign abnormalities and, because it can be performed at the same appointment as unenhanced ultrasound, more rapid diagnoses may be possible. OBJECTIVE: To compare the clinical effectiveness and cost-effectiveness of contrast-enhanced ultrasound (CEUS) using SonoVue((R)) with that of contrast-enhanced computed tomography (CECT) and contrast-enhanced magnetic resonance imaging (CEMRI) for the assessment of adults with focal liver lesions (FLLs) in whom previous liver imaging is inconclusive. DATA SOURCES: Eight bibliographic databases including MEDLINE, EMBASE, Cochrane Database of Systematic Reviews and Database of Abstracts of Reviews of Effects were searched from 2000 to September/October 2011. Research registers and conference proceedings were also searched. REVIEW METHODS: Systematic review methods followed published guidance. Risk of bias was assessed using a modified version of the QUADAS-2 tool. Results were stratified by clinical indication for imaging (characterisation of FLLs detected on ultrasound surveillance of cirrhosis patients, detection of liver metastases, characterisation of incidentally detected FLLs, assessment of treatment response). For incidental FLLs, pooled estimates of sensitivity and specificity, with 95% CIs, were calculated using a random-effects model. For other clinical indications a narrative summary was used. The cost-effectiveness of CEUS was modelled separately for the three main clinical applications considered [characterisation of FLLs detected on ultrasound surveillance of cirrhosis patients, detection of liver metastases in patients with colorectal cancer (CRC), characterisation of incidentally detected FLLs]. RESULTS: Of the 854 references identified, 19 (describing 18 studies) were included in the review. Hand searching of conference proceedings identified a further three studies. Twenty of the 21 studies included in the systematic review were diagnostic test accuracy studies. Studies in cirrhosis patients reported varying estimates of test performance. There was no consistent evidence of a significant difference in performance between imaging modalities. It was unclear whether or not CEUS alone is adequate to rule out hepatocellular carcinoma (HCC) for FLLs of < 30 mm; one study indicated that CEUS may be better at ruling out HCC for FLLs of 11-30 mm [very small FLLs (< 10 mm) excluded]. There was no consistent evidence of a difference in test performance between imaging modalities for the detection of metastases; CEUS alone may be adequate to rule out liver metastases in colorectal cancer. In patients with incidentally detected FLLs, the pooled estimates of sensitivity for any malignancy using CEUS and CECT were 95.1% and 94.6%, respectively, and the corresponding specificity estimates were 93.8% and 93.1% respectively. One study comparing CEUS with CEMRI reported similar sensitivity and lower specificity for both modalities. In the surveillance of cirrhosis, CEUS was as effective as but pound379 less costly than CECT. CEMRI was pound1063 more costly than CEUS and gained 0.022 QALYs. In the detection of liver metastases from CRC, CEUS cost pound1 more than CECT, and at a lifetime time horizon they yielded equal QALYs. CEMRI was dominated by CECT. In the characterisation of incidentally detected FLLs, CEUS was slightly more effective than CECT and CEMRI (by 0.0002 QALYs and 0.0026 QALYs respectively) and less costly (by pound52 and pound131 respectively). LIMITATIONS: There were a number of methodological issues specific to the studies included in this review. The main indication for liver imaging in the populations considered is likely to be to rule out primary liver cancer or metastases. Therefore, patient-level analyses of test performance are of particular interest. Some of the studies included in this review reported per-patient analyses; however, no study clearly stated how results were defined (e.g. was the presence of any positive lesion regarded as a positive test for the whole patient). In addition,

Abstract 259: Myocardial Perfusion Imaging has Markedly Lower Sensitivity after Accounting for Verification Bias

Introduction Estimates of myocardial perfusion imaging (MPI) performance are often derived from convenience cohorts of patients undergoing coronary angiography, who are selectively referred for this invasive test because of high-risk clinical characteristics, high-risk test results, or both. While clinically appropriate, this approach distorts estimates of MPI's test performance, since sensitivity and specificity are only assessed in patients who are referred for angiography. This verification bias, sometimes called “work-up” or “referral” bias, compromises physicians' ability to appropriately use noninvasive testing since test performance data may be inaccurate. We sought to examine the impact of verification bias on MPI's performance, to provide clinicians with a more accurate assessment of its diagnostic characteristics. Hypothesis We hypothesized that the bias-adjusted performance of MPI will show significantly lower sensitivity and significantly higher specificity compared to published estimates derived from convenience cohorts undergoing invasive angiography. Methods We used a comprehensive meta-analysis of MPI performance (14 studies totaling 3,500 patients based on angiographic referral) to assess the reported (biased) sensitivity and specificity of MPI. We then computed unbiased sensitivity and specificity rates using published methods, combining the meta-analysis estimates with other published data giving angiography referral rates for patients with both positive and negative MPI results (48% and 6%, respectively). Finally, we validated the unbiased performance estimates using a recent multi-center study (Coronary Obstruction Detection by Molecular Personalized Gene Expression [COMPASS]) of MPI that was not subject to verification bias, since all subjects had angiographic referral to use as the gold standard. Results The meta-analysis reported (biased) MPI sensitivity was 81% and specificity was 65%. After appropriate correction for verification bias, the unbiased estimate of sensitivity was 35% (95% CI = 27% to 45%) and specificity was 94% (95% CI = 91% to 96%). These estimates were very similar to the observed values in the unbiased COMPASS trial (N=431 patients): core laboratory MPI sensitivity was 36% (95% CI = 24% to 50%) and specificity was 90% (95% CI = 87% to 93%). Conclusions In conclusion, after correction for verification bias or when estimated in a population without bias, MPI is markedly less sensitive and more specific than most published estimates would indicate. The biased estimates of test performance would tend to encourage referral to angiography compared to corrected estimates. Future studies of non-invasive imaging modalities in patients with suspected or diagnosed CHD should report performance estimates that are either corrected for verification bias or use unbiased cohorts, to provide an accurate assessment of test performance for clinical decision making.

Diagnostic performance of serum cobalamin tests: a systematic review and meta-analysis

Serum cobalamin (cbl, vitamin B(12)) tests are routinely ordered for investigating conditions potentially amenable to cbl supplementation. This study aimed to systematically assess the evidence of diagnostic accuracy for serum cbl tests across patient subgroups. Seven medical databases were searched (1990 to November 2009). Studies were included that compared serum cbl to a reference standard (all reference standards employed). Study quality was assessed using QUADAS. Summary estimates of test performance were determined using the bivariate model and hierarchical summary receiver operating characteristic curves (HSROC). Of 2878 identified studies, 54 were included. Studies rated poorly against QUADAS criteria. Positive (PLR) and negative likelihood ratios (NLR) were 2.72 [95% confidence interval (CI) 1.95, 3.81] and 0.59 (0.49, 0.72), respectively (studies employing methylmalonic acid as the referent). In studies employing a clinical reference standard, PLR was 3.33 (0.92, 12.10) and NLR 0.34 (0.13, 0.89). Test performance did not vary by clinical indication, test method or age. This review was limited by the quality of the evidence base and lack of a gold standard. From the available evidence, diagnosis of conditions amenable to cbl supplementation on the basis of serum cbl level alone cannot be considered a reliable approach to investigating suspected vitamin deficiency.

Decision-Analytic Modeling to Evaluate Benefits and Harms of Medical Tests: Uses and Limitations

The clinical utility of medical tests is measured by whether the information they provide affects patient-relevant outcomes. To a large extent, effects of medical tests are indirect in nature. In principle, a test result affects patient outcomes mainly by influencing treatment choices. This indirectness in the link between testing and its downstream effects poses practical challenges to comparing alternate test-and-treat strategies in clinical trials. Keeping in mind the broader audience of researchers who perform comparative effectiveness reviews and technology assessments, the authors summarize the rationale for and pitfalls of decision modeling in the comparative evaluation of medical tests by virtue of specific examples. Modeling facilitates the interpretation of test performance measures by connecting the link between testing and patient outcomes, accounting for uncertainties and explicating assumptions, and allowing the systematic study of tradeoffs and uncertainty. The authors discuss challenges encountered when modeling test-and-treat strategies, including but not limited to scarcity of data on important parameters, transferring estimates of test performance across studies, choosing modeling outcomes, and obtaining summary estimates for test performance data.

Adjusting for verification bias in diagnostic test evaluation: A Bayesian approach

Obtaining accurate estimates of the performance of a diagnostic test for some population of patients might be difficult when the sample of subjects used for this purpose is not representative for the whole population. Thus, in the motivating example of this paper a test is evaluated by comparing its results with those given by a gold standard procedure, which yields the disease status verification. However, this procedure is invasive and has a non-negligible risk of serious complications. Moreover, subjects are selected to undergo the gold standard based on some risk factors and the results of the test under study. The test performance estimates based on the selected sample of subjects are biased. This problem was presented in previous studies under the name of verification bias. The current paper introduces a Bayesian method to adjust for this bias, which can be regarded as a missing data problem. In addition, it addresses the case of non-ignorable verification bias. The proposed Bayesian estimation approach provides test performance estimates that are consistent with the results obtained using likelihood-based approach. In addition, the paper studies how valuable the statistical findings are from the perspective of clinical decision making.

Foot-and-mouth disease non-structural protein serology in cattle: Use of a Bayesian framework to estimate diagnostic sensitivity and specificity of six ELISA tests and true prevalence in the field

The diagnostic performance of six foot-and-mouth disease (FMD) assays for detection of antibodies to the non-structural proteins (NSP) of the FMD virus (FMDV) was estimated using a Bayesian analysis on field sera from cattle of unknown infection status originating from post-FMDV outbreak situations in Israel and Zimbabwe. Estimations of the disease prevalence in both populations were also obtained. The diagnostic sensitivity estimates did not differ between both field studies, although overall Bayesian estimates were markedly higher than those previously reported based on sera from comparable experimentally infected (vaccinated) cattle populations. All NSP-based assays demonstrated a lower diagnostic specificity when applied to the Zimbabwean sera compared to both published specificities and similar Bayesian specificity estimates derived for the Israeli dataset. In Israel, the disease prevalence was estimated at 23.9% (95% credibility interval: 19.5-28.8%), whereas 65.4% (59.0-72.5%) was found in Zimbabwe. The need for reliable diagnostic test performance estimates and the benefits of Bayesian analysis in obtaining them are also addressed.

Accuracy of magnetic resonance imaging for the diagnosis of multiple sclerosis: systematic review

Objective To determine the accuracy of magnetic resonance imaging criteria for the early diagnosis of multiple sclerosis in patients with suspected disease.Design Systematic review.Data sources 12 electronic databases, citation searches,...

Newborn screening for congenital heart defects: a systematic review and cost-effectiveness analysis

To provide evidence to inform policy decisions about the most appropriate newborn screening strategy for congenital heart defects, identifying priorities for future research that might reduce important uncertainties in the evidence base for such decisions. Electronic databases. Groups of parents and health professionals. A systematic review of the published medical literature concerning outcomes for children with congenital heart defects was carried out. A decision analytic model was developed to assess the cost-effectiveness of alternative screening strategies for congenital heart defects relevant to the UK. A further study was then carried out using a self-administered anonymous questionnaire to explore the perspectives of parents and health professionals towards the quality of life of children with congenital heart defects. The findings from a structured review of the medical literature regarding parental experiences were linked with those from a focus group of parents of children with congenital heart defects. Current newborn screening policy comprises a clinical examination at birth and 6 weeks, with specific cardiac investigations for specified high-risk children. Routine data are lacking, but under half of affected babies, not previously identified antenatally or because of symptoms, are identified by current newborn screening. There is evidence that screen-positive infants do not receive timely management. Pulse oximetry and echocardiography, in addition to clinical examination, are alternative newborn screening strategies but their cost-effectiveness has not been adequately evaluated in a UK setting. In a population of 100,000 live-born infants, the model predicts 121 infants with life-threatening congenital heart defects undiagnosed at screening, of whom 82 (68%) and 83 (69%) are detected by pulse oximetry and screening echocardiography, respectively, but only 39 (32%) by clinical examination alone. Of these, 71, 71 and 34, respectively, receive a timely diagnosis. The model predicts 46 (0.5%) false-positive screening diagnoses per 100,000 infants with clinical examination, 1168 (1.3%) with pulse oximetry and 4857 (5.4%) with screening echocardiography. The latter includes infants with clinically non-significant defects. Total programme costs are predicted of pound 300,000 for clinical examination, pound 480,000 for pulse oximetry and pound 3.54 million for screening echocardiography. The additional cost per additional timely diagnosis of life-threatening congenital heart defects ranges from pound 4900 for pulse oximetry to pound 4.5 million for screening echocardiography. Including clinically significant congenital heart defects gives an additional cost per additional diagnosis of pound 1500 for pulse oximetry and pound 36,000 for screening echocardiography. Key determinants for cost-effectiveness are detection rates for pulse oximetry and screening echocardiography. Parents and health professionals place similar values on the quality of life outcomes of children with congenital heart defects and both are more averse to neurological than to cardiac disability. Adverse psychosocial effects for parents are focused around poor management and/or false test results. Early detection through newborn screening potentially can improve the outcome of congenital heart defects; however the current programme performs poorly, and lacks monitoring of quality assurance, performance management and longer term outcomes. Pulse oximetry is a promising alternative newborn screening strategy but further evaluation is needed to obtain more precise estimates of test performance and to inform optimal timing, diagnostic and management strategies. Although screening echocardiography is associated with the highest detection rate, it is the most costly strategy and has a 5% false-positive rate. Improving antenatal detection of congenital heart defects increases the cost per timely postnatal diagnosis afforded by any newborn screening strategy but does not alter the relative effects of the strategies. An improvement of timely management of screen positive infants is essential. Further research is required to refine the detection rate and other aspects of pulse oximetry, to evaluate antenatal screening strategies more directly, and to investigate the psychosocial effects of newborn screening for congenital heart defects.

Spectrum bias or spectrum effect? Subgroup variation in diagnostic test evaluation.

Diagnostic tests must be evaluated in a clinically relevant population. However, test performance often varies across population subgroups. Spectrum bias, a term commonly used to describe this heterogeneity, is typically thought to occur when diagnostic test performance varies across patient subgroups and a study of that test's performance does not adequately represent all subgroups. Yet subgroup variation is not a bias if appropriate analyses are conducted. Failure to recognize and address heterogeneity will lead to estimates of test performance that are not generalizable to the relevant clinical populations. Heterogeneity can be addressed with relatively simple stratification procedures, limited primarily by the sample size and the precision of the estimates. This paper proposes the use of the term spectrum effect, rather than spectrum bias, and outlines strategies for using stratified sensitivity and specificity estimates, likelihood ratios, and receiver-operating characteristic curves. Investigators of diagnostic tests should consider the potential for spectrum effect seriously and should address heterogeneity in their analyses. Furthermore, clinicians should consider study samples carefully to determine whether results are generalizable to their specific patient population.

A Bayesian approach to estimating the performance of a bovine virus diarrhoea virus (BVDV) antibody ELISA bulk-tank milk test

We investigated the operational performance (sensitivity and specificity) of a bovine virus diarrhoea virus (BVDV) antibody ELISA bulk-tank milk test for predicting the herd BVDV antibody status in young stock (as a relatively precise indicator of active BVDV infection). The study was based on results from the annual screenings under the Norwegian bovine virus diarrhoea (BVD) control and eradication program, lasting from 1993 to 1997. Empirical information from these annual screenings was the basis for prior assumptions about the true prevalence of young-stock-positive herds. Assumptions about prior distributions for sensitivity and specificity were based on the literature. Improved posterior test performance estimates were achieved applying a Bayesian approach using Gibbs sampling simulation. The simulations were run separately for each year, and yielded median values for sensitivity of 87% at the cut-off used in the BVD program. The posterior distributions were wide indicating much uncertainty in these estimates. The specificity estimates ranged from 79 to 92% and had narrower posterior estimates. The estimates differed by year. When running the same simulation procedures at a lower cut-off — after altering the sensitivity and specificity priors — the median sensitivity estimates increased to about 95%; the median specificity ranged from 71 to 83%. Due to low prevalence, the Bayesian method lacked power to assess the test sensitivity. A technically simpler descriptive graphing procedure (based on empirical information) provided equally useful insight into the bulk-tank milk test performance.

A comparative meta-analysis on the variability in test performance among FDA-licensed enzyme immunosorbent assays for HIV antibody testing

Background. Although independently published studies have compared diagnostic test performance among various manufactured enzyme immunosorbent assays (EIAs) used in HIV antibody testing, none have attempted to formally synthesize such results through a comparative meta-analysis. In particular, no estimates of post-FDA approval test performance, in terms of sensitivity and specificity, and their associated variability within each manufacturer, has been reported in the literature, along with an analysis of the relative differences in manufacturer test performance in practice (after FDA approval). Methods. Retrieval of studies was done using several searching strategies, while retrieval of manufacturer information was done through package inserts and direct contacts. Comparisons of HIV antibody test performance across manufacturers and within a single manufacturer were made based on 16 estimates (from 11 articles) and 33 estimates (from 19 articles), respectively. A generalized linear model, based upon Bayes estimates of sensitivity and specificity, was used to assess the impact of several study-level covariates on the performance of these EIAs, with overall estimates of manufacturer test performance and associated variability obtained based on generalized estimating equations. Results. Estimates of test performance were obtained across studies, with a significant (P < 0.01) difference between manufacturers. The test performance of each manufacturer significantly interacted (P < 0.05) with the following study-level covariates: type of population screened, year of diagnostic testing and study quality. Among a single manufacturer, Abbott, significant improvement in estimates of test sensitivity (P < 0.01) and specificity (P < 0.01) was observed with each newly produced antibody kit. Conclusion. Estimates on the relative differences in test performance within each manufacturer may be used for guiding decisions on the choice of EIA test kit for HIV antibodies, given the type of population screened, as well as cost and time considerations. In addition, the results of this meta-analysis may be used in modeling HIV prevalence when used as prior information within a Bayesian framework or for standardizing test results among various manufacturers.