Verification Bias Research Articles

Risk of Bias in Liver Imaging Reporting and Data System Studies Using QUADAS-2.

Purpose: Use a tailored version of the Quality Assessment of Diagnostic Accuracy Studies tool to evaluate risk of bias and applicability across LIRADS related publications. Method: A tailored QUADAS-2 tool was created through consensus approach to assess risk of bias and applicability across 37 LI-RADS related publications. Studies were selected from 2017 to 2022 using the assistance of experienced hospital librarians to search for studies evaluating the diagnostic accuracy of CT, MRI, or contrast-enhanced ultrasound for HCC using LI-RADS through multiple different databases. QUADAS-2 assessments were performed in duplicate and independently by 2 authors with experience using the QUADAS-2 tool. Disagreements were resolved with a third expert reviewer. Consensus QUADAS-2 assessments were tabulated for each domain. Results: Using the tailored QUADAS-2 tool, 31 of the 37 included LI-RADS studies were assessed as high risk of bias, and 9 out of 37 studies demonstrated concerns for applicability. Patient selection (21 out of 37 studies) and flow/timing (24 out of 37 studies) domains demonstrated the highest risk of bias. 6 out of 37 studies in the index domain demonstrated high risk of bias. 2 out of 37 studies showed high risk of bias in the reference standard domain. Conclusion: A significant proportion of LI-RADS research is at risk of bias with concerns for applicability. Identifying risk of bias in such research is essential to recognize limitations of a study that may affect the validity of the results. Areas for improvement in LI-RADS research include reducing selection bias, avoiding inappropriate exclusions, and decreasing verification bias.

Comparison of parent or caregiver-completed development screening tools with Bayley Scales of Infant Development: a systematic review and meta-analysis

BackgroundParent/caregiver-completed developmental testing (PCDT) is integral to developmental care in children; however, there is limited information on its accuracy. In this systematic review, we compared the diagnostic accuracy of PCDT...

The Sydney system for lymph node FNA biopsy cytopathology: A detailed analysis of recent publications and meta-analysis and a proposal for the components of an ideal prospective study of a cytopathology reporting system.

The Sydney system for fine-needle aspiration biopsy of lymph nodes has five categories, stressing the role of correlation of cytopathology with clinical, ultrasound, and ancillary findings to achieve diagnosis. The five categories constitute a hierarchical system with increasing risk of malignancy from benign to atypical, suspicious, and malignant categories, which informs recommendations for further workup to achieve a final diagnosis as possible. This article analyzes 10 publications using the Sydney system and a meta-analysis of nine of these studies. The primary goal of the analysis is to ascertain the causes of the large ranges in risk of malignancy for the "atypical" and "inadequate" compared to "benign," "suspicious," and "malignant" categories, which were comparable to well-established reporting systems. Research protocols are proposed to improve future studies. PubMed literature search from January 2021 to December 2023 identified studies evaluating performance of the Sydney system. Ten studies showed heterogeneity with clinical setting, study design, ultrasound use and rapid on-site evaluation, operator, cutoff points for "positive" cases, with inherent partial verification biases, resulting in a wide range of risk of malignancy, specificity, and sensitivity values. Analysis shows the large range is due to heterogeneity of the studies, which suffer from biases and variable statistical analysis that are ultimately included in any meta-analysis, detracting from the usefulness of the risk of malignancy derived by the meta-analysis. Components for ideal analyses of reporting systems are presented.

Optimal two-phase sampling for comparing correlated areas under the ROC curves of two screening tests in the presence of verification bias

ABSTRACT The accuracy of a screening test is often measured by the area under the receiver characteristic (ROC) curve (AUC) of a screening test. Two-phase designs have been widely used in diagnostic studies for estimating one single AUC and comparing two AUCs where the screening test results are measured for a large sample (Phase one sample) while the disease status is only verified for a subset of Phase one sample (Phase two sample) by a gold standard. In this paper, we consider the optimal two-phase sampling design for comparing the performance of two ordinal screening tests in classifying disease status. Specifically, we derive an analytical variance formula for the AUC difference estimator and use it to find the optimal sampling probabilities that minimize the variance formula for the AUC difference estimator. According to the proposed optimal two-phase design, the strata with the levels of two tests far apart from each other should be over-sampled while the strata with the levels of two tests close to each other should be under-sampled. Simulation results indicate that two-phase sampling under optimal allocation (OA) achieves a substantial amount of variance reduction, compared with two-phase sampling under proportional allocation (PA). Furthermore, in comparison with a one-phase random sampling, two-phase sampling under OA or PA has a clear advantage in reducing the variance of AUC difference estimator when the variances of the two screening test results in the disease population differ greatly from their counterparts in non-disease population.

Simultaneous alleviation of verification and reference standard biases in a community-based tuberculosis screening study using Bayesian latent class analysis.

Estimation of prevalence and diagnostic test accuracy in tuberculosis (TB) prevalence surveys suffer from reference standard and verification biases. The former is attributed to the imperfect reference test used to bacteriologically confirm TB disease. The latter occurs when only the participants screening positive for any TB-compatible symptom or chest X-ray abnormality are selected for bacteriological testing (verification). Bayesian latent class analysis (LCA) alleviates the reference standard bias but suffers verification bias in TB prevalence surveys. This work aims to identify best-practice approaches to simultaneously alleviate the reference standard and verification biases in the estimates of pulmonary TB prevalence and diagnostic test performance in TB prevalence surveys. We performed a secondary analysis of 9869 participants aged ≥15 years from a community-based multimorbidity screening study in a rural district of KwaZulu-Natal, South Africa (Vukuzazi study). Participants were eligible for bacteriological testing using Xpert Ultra and culture if they reported any cardinal TB symptom or had an abnormal chest X-ray finding. We conducted Bayesian LCA in five ways to handle the unverified individuals: (i) complete-case analysis, (ii) analysis assuming the unverified individuals would be negative if bacteriologically tested, (iii) analysis of multiply-imputed datasets with imputation of the missing bacteriological test results for the unverified individuals using multivariate imputation via chained equations (MICE), and simultaneous imputation of the missing bacteriological test results in the analysis model assuming the missing bacteriological test results were (iv) missing at random (MAR), and (v) missing not at random (MNAR). We compared the results of (i)-(iii) to the analysis based on a composite reference standard (CRS) of Xpert Ultra and culture. Through simulation with an overall true prevalence of 2.0%, we evaluated the ability of the models to alleviate both biases simultaneously. Based on simulation, Bayesian LCA with simultaneous imputation of the missing bacteriological test results under the assumption that the missing data are MAR and MNAR alleviate the reference standard and verification biases. CRS-based analysis and Bayesian LCA assuming the unverified are negative for TB alleviate the biases only when the true overall prevalence is <3.0%. Complete-case analysis produced biased estimates. In the Vukuzazi study, Bayesian LCA with simultaneous imputation of the missing bacteriological test results under the MAR and MNAR assumptions produced overall PTB prevalence of 0.9% (95% Credible Interval (CrI): 0.6-1.9) and 0.7% (95% CrI: 0.5-1.1) respectively alongside realistic estimates of overall diagnostic test sensitivity and specificity with substantially overlapping 95% CrI. The CRS-based analysis and Bayesian LCA assuming the unverified were negative for TB produced 0.7% (95% CrI: 0.5-0.9) and 0.7% (95% CrI: 0.5-1.2) overall PTB prevalence respectively with realistic estimates of overall diagnostic test sensitivity and specificity. Unlike CRS-based analysis, Bayesian LCA of multiply-imputed data using MICE mitigates both biases. The findings demonstrate the efficacy of these advanced techniques in alleviating the reference standard and verification biases, enhancing the robustness of community-based screening programs. Imputing missing values as negative for bacteriological tests is plausible under realistic assumptions.

Internal validation of Automated Visual Evaluation (AVE) on smartphone images for cervical cancer screening in a prospective study in Zambia.

Visual inspection with acetic acid (VIA) is a low-cost approach for cervical cancer screening used in most low- and middle-income countries (LMICs) but, similar to other visual tests, is subjective and requires sustained training and quality assurance. We developed, trained, and validated an artificial-intelligence-based "Automated Visual Evaluation" (AVE) tool that can be adapted to run on smartphones to assess smartphone-captured images of the cervix and identify precancerous lesions, helping augment VIA performance. Prospective study. Eight public health facilities in Zambia. A total of 8204 women aged 25-55. Cervical images captured on commonly used low-cost smartphone models were matched with key clinical information including human immunodeficiency virus (HIV) and human papillomavirus (HPV) status, plus histopathology analysis (where applicable), to develop and train an AVE algorithm and evaluate its performance for use as a primary screen and triage test for women who are HPV positive. Area under the receiver operating curve (AUC); sensitivity; specificity. As a general population screening tool for cervical precancerous lesions, AVE identified cases of cervical precancerous and cancerous (CIN2+) lesions with high performance (AUC = 0.91, 95% confidence interval [CI] = 0.89-0.93), which translates to a sensitivity of 85% (95% CI = 81%-90%) and specificity of 86% (95% CI = 84%-88%) based on maximizing the Youden's index. This represents a considerable improvement over naked eye VIA, which as per a meta-analysis by the World Health Organization (WHO) has a sensitivity of 66% and specificity of 87%. For women living with HIV, the AUC of AVE was 0.91 (95% CI = 0.88-0.93), and among those testing positive for high-risk HPV types, the AUC was 0.87 (95% CI = 0.83-0.91). These results demonstrate the feasibility of utilizing AVE on images captured using a commonly available smartphone by nurses in a screening program, and support our ongoing efforts for moving to more broadly evaluate AVE for its clinical sensitivity, specificity, feasibility, and acceptability across a wider range of settings. Limitations of this study include potential inflation of performance estimates due to verification bias (as biopsies were only obtained from participants with visible aceto-white cervical lesions) and due to this being an internal validation (the test data, while independent from that used to develop the algorithm was drawn from the same study).

Sample size calculation for comparing two screening tests when the gold standard is missing at random.

Sample size formulas have been proposed for comparing two sensitivities (specificities) in the presence of verification bias under a paired design. However, the existing sample size formulas involve lengthy calculations of derivatives and are too complicated to implement. In this paper, we propose alternative sample size formulas for each of three existing tests, two Wald tests and one weighted McNemar's test. The proposed sample size formulas are more intuitive and simpler to implement than their existing counterparts. Furthermore, by comparing the sample sizes calculated based on the three tests, we can show that the three tests have similar sample sizes even though the weighted McNemar's test only use the data from discordant pairs whereas the two Wald tests also use the additional data from accordant pairs.

The performance of single and combination test strategies using visual inspection, cytology, high-risk HPV DNA and HPV16/18 to screen South African women with and without HIV-infection

BackgroundCervical cancer screening strategies should ideally be informed by population-specific data. Strategies recommended for secondary prevention, are often inadequately studied in populations with high cervical disease burdens. This report describes the test performance measured against CIN2 + /CIN3 + histology in HIV-positive women (HPW) and HIV-negative women (HNW) with the aim to determine the most effective strategies to identify South African women at risk.MethodsPrimary screening using visual inspection, cytology and HPV DNA (cobas®) was performed in two South African provinces on 456 HPW and 639 HNW participating in the multicentric DiaVACCS trial. Histology was obtained for 91.7% screen-positive and 42.7% screen-negative participants, and unavailable histology was determined by multiple imputation to adjust for verification bias. Cross-sectional test performance was calculated for single and combination test strategies with and without intermediate risk categories using different cut-offs. Minimum acceptability for sensitivity and specificity, treatment and follow-up numbers were considered to evaluate strategies.ResultsThe only single test to reach acceptability in HPW was cytology (LSIL) [sensitivity 71.2%; specificity 90.5%; treatment 33.4%]; in HNW only HPV (hr) qualified [sensitivity 68.2%; specificity 85.2%; treatment 23.5%]. The universally best performing strategy which also resulted in smaller treatment numbers without intermediate risk group was primary HPV(hr), with treatment of both HPV(16/18) and cytology (ASCUS +) [HPW: sensitivity 73.6%; specificity 89.7%; treatment 34.7%. HNW: sensitivity 59.1%; specificity 93.6%; treatment 13.9%].DNA testing for hrHPV (any) and hrHPV (16/18) was the best universally acceptable strategy with an intermediate risk category (early follow-up) in HPW [sensitivity 82.1%; specificity 96.4%; treatment 17.1%; follow-up 31.4%] and HNW [sensitivity 68.2%; specificity 96.7%; treatment 7.6%; follow-up 15.9%]. In comparison, using both HPV (16/18) and cytology (ASCUS +) as secondary tests in hrHPV positive women, decreased follow-up [HPW 13.8%, HNW 9.6%], but increased treatment [HPW 34.7%, HNW 13.9%].ConclusionUsing hrHPV (any) as primary and both HPV16/18 and cytology as secondary tests, was universally acceptable without an intermediate risk group. Strategies with follow-up groups improved screening performance with smaller treatment numbers, but with effective management of the intermediate risk group as prerequisite.

Colorectal Cancer Screening by Fecal Immunochemical Tests (FIT): Considerations on Sampling and Markers (Hb and Hb/Hp Complex) of Fecal Occult Blood (FOB).

Formal demonstration of the efficacy of colorectal cancer (CRC) screening by fecal immunochemical tests (FITs) in reducing CRC incidence and mortality is still missing. The aim of this study was to analyze the impact of sampling and FIT marker in the recently implemented CRC screening program in Finland. Because only the index test [FIT hemoglobin (Hb)]-positive subjects are verified by the reference test (colonoscopy), the new screening program is subject to verification bias that precludes estimating the diagnostic accuracy (DA) indicators. A previously published study (5) with 100% biopsy verification of colonoscopy referral subjects (called validation cohort, n=300) was used to derive these missing DA estimates. Two points of concern were addressed: i) only one-day sample tested, and ii) only the Hb component (but not Hb/Hp complex) was analyzed by FIT. The estimated DA of one-sample testing for Hb in the screening setting had a very low sensitivity (SE) (12.5%; 95%CI=12.3-12.7) for adenomas, with AUC=0.560 (for CRC, AUC=0.950). Testing three samples for Hb improved SE to 19.4% (95%CI=19.1-19.7%) but had little effect on overall DA (AUC=0.590). For adenomas, one-sample testing for Hb and Hb/Hp complex provided higher SE than three-sample testing for Hb (SE 20.6%; 95%CI=20.3-21.0), and the best SE was reached when two samples were tested for Hb and Hb/Hp complex (SE 47.5%; 95%CI=46.9-48.1%) (AUC=0.730). The strategy of the current CRC screening could be significantly improved by testing two consecutive samples by Hb and Hb/Hp complex, instead of stand-alone Hb testing of one sample.

Diagnostic accuracy of MRI, CT, and [18F]FDG-PET-CT in detecting lymph node metastases in clinically early-stage cervical cancer — a nationwide Dutch cohort study

ObjectivesImaging is increasingly used to assess lymph node involvement in clinically early-stage cervical cancer. This retrospective study aimed to evaluate the diagnostic accuracy of MRI, CT, and [18F]FDG-PET-CT.MethodsWomen with International Federation of Gynaecology and Obstetrics (FIGO) 2009 stage IA2-IIA cervical cancer and pretreatment imaging between 2009 and 2017 were selected from the Netherlands Cancer Registry. Patient-based and region-based (i.e. pelvic and common iliac) nodal status was extracted from radiology reports. Pathology results were considered the reference standard for calculating accuracy indices. Multiple imputation was used for missing pathology to limit verification bias risk.ResultsNodal assessment was performed in 1676 patients with MRI, 926 with CT, and 379 with [18F]FDG-PET-CT, with suspicious nodes detected in 17%, 16%, and 48%, respectively. [18F]FDG-PET-CT was used to confirm MRI/CT results in 95% of patients. Pathology results were imputed for 30% of patients. [18F]FDG-PET-CT outperformed MRI and CT in detecting patient-based nodal metastases with sensitivities of 80%, 48%, and 40%, and AUCs of 0.814, 0.706, and 0.667, respectively, but not in specificity: 79%, 92%, and 92%. Region-based analyses showed similar indices in the pelvic region, but worse performance in the common iliac region with AUCs of 0.575, 0.554, and 0.517, respectively.Conclusions[18F]FDG-PET-CT outperformed MRI and CT in detecting nodal metastases, which may be related to its use as a verification modality. However, MRI and CT had the highest specificity. As MRI is generally performed routinely to assess local and regional spread of cervical cancer, [18F]FDG-PET-CT can be used to confirm suspicious nodes.Critical relevance statementAccurate assessment of the nodal status in clinically early-stage cervical cancer is essential for tumour staging, treatment decision making and prognosis.Key points• The accuracy of MRI, CT or [18F]FDG-PET-CT for nodal staging in early cervical cancer is a subject of discussion.• Overall, [18F]FDG-PET-CT outperformed MRI, followed by CT, when used as a verification modality.• Staging with MRI and the addition of [18F]FDG-PET-CT to verify high-risk cases seems to be a good approach.Graphical

Accuracy of screening tests for cervical precancer in women living with HIV in low-resource settings: a paired prospective study in Lusaka, Zambia

ObjectiveThis study aimed to provide evidence to improve cervical screening for women living with HIV (WLHIV). We assessed the accuracy of screening tests that can be used in low-resource settings...

Risk of Bias Assessment and Risk Minimisation Strategies in COVID-19 Diagnostic Test Accuracy Study

It is paramount to assess the risk of biases in may arise from diagnostic test accuracy (DTA) study as it will affect the accuracy and validity of the tests. These biases can be found in published researches and here we look at COVID-19 DTA studies. The evaluation of bias risk in diagnostic research is mainly performed using QUADAS-2. The aim of this review was to determine potential selection and information biases in diagnostic test accuracy studies and strategies to minimize risk of biases. Literature review related to diagnostic test accuracy study is identified through an online search of databases namely PubMed, ScienceDirect, Research Gate, Google Scholar, and official government websites range. Six potential biases in four QUADAS-2 domains are identified in COVID-19 diagnostic test accuracy study which are 1) spectrum bias in patient selection; 2) interpretation bias in index test; 3) differential misclassification bias and nondifferential misclassification bias in reference standard; and 4) partial verification bias and differential verification bias in patient flow. The identified biases exert effects on accuracy of COVID-19 diagnostic tests. Six strategies are recommended to reduce these biases, hence, improving the accuracy of COVID-19 diagnostic tests. The best diagnostic test can give benefits to the population in the mass screening program during COVID-19.

Discriminant Validity and Time‐effectiveness of A Stepwise Dementia Case‐finding Approach in an Asian Elderly Community

AbstractBackgroundTo investigate the discriminant validity and time‐effectiveness of a stepwise dementia case‐finding approach in a community‐based Singaporean older adult population.MethodParticipants who completed the Progressive Forgetfulness Question (PFQ) and the Abbreviated Mental Test (AMT) were invited to phase II and administered the Mini‐Mental State Examination (MMSE), Montreal Cognitive Assessment (MoCA), and a formal neuropsychological battery. Participants were diagnosed by cognitive performance and DSM‐IV criteria into No Cognitive Impairment (NCI), Cognitive Impairment‐No Dementia (CIND) mild (≤2 cognitive domains impaired), CIND moderate (&gt;2 domains impaired) and dementia. Receiver Operating Characteristic were conducted for different cognitive instruments (AMT, MoCA, MMSE) between PFQ = Yes and PFQ = No groups. All discriminant indices including sensitivity, specificity, positive (PPV), negative predictive values (NPV) and accuracy were calculated. Bayesian correction methods were used to adjust the verification bias. Screening time saving and number of subjects further evaluated was estimated in two scenarios with or without the PFQ.ResultThe PFQ showed an NPV of 94.2% for excluding dementia‐free cases after adjustment. After excluding PFQ = No participants, all cognitive tools achieved optimal NPV (&gt;99%). Furthermore, the number of people requiring further evaluation decreased by 129 (42.7%), 58 (36.5%), 25 (14.3%) subjects, corresponding to 48.2%, 48.2%, and 47.8% of screening time being saved, when the PFQ was conducted prior to the MoCA, MMSE, and AMT, respectively.ConclusionUsing a single‐question assessment as the first step of a case‐finding approach, followed up a cognitive test such as the MoCA, could minimize time and resources for further investigation in the community.

Verification Bias Correction in Endometrial Abnormalities in Infertile Women Referred to Royan Institute Using Statistical Methods.

Verification bias is a common bias in the diagnostic accuracy of diagnostic tests and occurs when a number of individuals do not perform the gold standard test. In this study, we review the correcting methods of verification bias. In a cross-sectional study in 2020, 567 infertile women who were referred to Royan Research Institute were evaluated. The ultrasound is the performed test and the gold standard are hysteroscopy for some, and pathology for other abnormalities. For correcting verification bias conventional, Begg and Greens, Zhou, and logistic regression methods were used. In the gold standard hysteroscopy test, the sensitivity (SEN) and specificity (SPEC) obtained in conventional, Begg and Greens, Zhou, and logistics Regression methods were (50%, 90.3%), (48%, 96%), (22%, 77%), (50%, 90%), and (72.8, 77) respectively. Furthermore, the area under the curve (AUC) index and kappa statistics were calculated as 70.2%, and 43.6% respectively. In the pathology gold standard test, the SEN and SPEC for the conventional methods, Begg and Greens, Zhou and logistics regression were (67.7%, 86.7%), (66%, 88%), (29%, 70%), (66.9%, 87.6%), and (73%, 83.9%) respectively. Also, the AUC index and kappa statistics were 77%, and 55% respectively. In the study on endometrial abnormalities in infertile women, assuming that the missing data mechanism is random, the amount of bias in calculating SEN and SPEC is very low in the diagnostic tests calculated before and after correction, using Begg and Greens and logistic regression method. But Zhou's method gives rather large biased estimates.

A Comprehensive Review on the Diagnosis of Knee Injury by Deep Learning-Based Magnetic Resonance Imaging.

The continual improvement in the field of medical diagnosis has led to the monopoly of using deep learning (DL)-based magnetic resonance imaging (MRI) for the diagnosis of knee injury related to meniscal injury,ligament injury including the cruciate ligaments, collateral ligaments and medial patella-femoral ligament, and cartilage injury. The present systematic review was done by PubMed and Directory of Open Access Journals (DOAJ), wherein we finalised 24 studies conducted on the accuracy of DL MRI studies for knee injury identification. The studies showed an accuracy of 72.5% to 100% indicating that DL MRI holds an equivalent performance as humans in decision-making and management of knee injuries. This further opens up future exploration for improving MRI-based diagnosis keeping in mind the limitations of verification bias and data imbalance in ground truth subjectivity.

Sonoelastography for Testicular Tumor Identification: A Systematic Review and Meta-Analysis of Diagnostic Test Accuracy.

The objective of this review was to summarize the applications of sonoelastography in testicular tumor identification and inquire about their test performances. Two authors independently searched English journal articles and full conference papers from CINAHL, Embase, IEEE Xplore®, PubMed, Scopus, and Web of Science from inception and organized them into a PIRO (patient, index test, reference test, outcome) framework. Eleven studies (n = 11) were eligible for data synthesis, nine of which (n = 9) utilized strain elastography and two (n = 2) employed shear-wave elastography. Meta-analyses were performed on the distinction between neoplasm (tumor) and non-neoplasm (non-tumor) from four study arms and between malignancy and benignity from seven study arms. The pooled sensitivity of classifying malignancy and benignity was 86.0% (95%CI, 79.7% to 90.6%). There was substantial heterogeneity in the classification of neoplasm and non-neoplasm and in the specificity of classifying malignancy and benignity, which could not be addressed by the subgroup analysis of sonoelastography techniques. Heterogeneity might be associated with the high risk of bias and applicability concern, including a wide spectrum of testicular pathologies and verification bias in the reference tests. Key technical obstacles in the index test were manual compression in strain elastography, qualitative observation of non-standardized color codes, and locating the Regions of Interest (ROI), in addition to decisions in feature extractions. Future research may focus on multiparametric sonoelastography using deep learning models and ensemble learning. A decision model on the benefits-risks of surgical exploration (reference test) could also be developed to direct the test-and-treat strategy for testicular tumors.

Direct estimation of volume under the ROC surface with verification bias

ABSTRACT In practice, the receiver operating characteristic (ROC) curve of a diagnostic test is widely used to show the performance of the test for discriminating two-class events. The area under the ROC curve (AUC) is proposed as an index for the assessment of the diagnostic accuracy of the test under consideration. Due to ethical and cost considerations associated with application of gold standard (GS) tests, only a subset of the patients initially tested have verified disease status. Statistical evaluation of the test performance based only on test results from subjects with verified disease status are typically biased. Various AUC estimation methods for tests with verification biased data have been developed over the last few decades. In this article, we develop new direct estimation methods for the volume under the ROC surface (VUS) by extending the AUC estimation methods for two-class diagnostic tests to three-class diagnostic tests in the presence of verification bias. The proposed methods will provide a comprehensive guide to deal with the verification bias in three-class diagnostic test accuracy studies and lead to a better choice of diagnostic tests.

Is Your Machine Better Than You? You May Never Know

Artificial intelligence systems are increasingly demonstrating their capacity to make better predictions than human experts. Yet recent studies suggest that professionals sometimes doubt the quality of these systems and overrule machine-based prescriptions. This paper explores the extent to which a decision maker (DM) supervising a machine to make high-stakes decisions can properly assess whether the machine produces better recommendations. To that end, we study a setup in which a machine performs repeated decision tasks (e.g., whether to perform a biopsy) under the DM’s supervision. Because stakes are high, the DM primarily focuses on making the best choice for the task at hand. Nonetheless, as the DM observes the correctness of the machine’s prescriptions across tasks, the DM updates the DM’s belief about the machine. However, the DM is subject to a so-called verification bias such that the DM verifies the machine’s correctness and updates the DM’s belief accordingly only if the DM ultimately decides to act on the task. In this setup, we characterize the evolution of the DM’s belief and overruling decisions over time. We identify situations under which the DM hesitates forever whether the machine is better; that is, the DM never fully ignores but regularly overrules it. Moreover, the DM sometimes wrongly believes with positive probability that the machine is better. We fully characterize the conditions under which these learning failures occur and explore how mistrusting the machine affects them. These findings provide a novel explanation for human–machine complementarity and suggest guidelines on the decision to fully adopt or reject a machine.This paper was accepted by Elena Katok, special issue on the human–algorithm connection.Supplemental Material: The online appendix is available at https://doi.org/10.1287/mnsc.2023.4791 .

MRI vs. CT for pancreatic adenocarcinoma vascular invasion: comparative diagnostic test accuracy systematic review and meta-analysis.

To perform a systematic review comparing the diagnostic accuracy of MRI vs. CT for assessing pancreatic ductal adenocarcinoma (PDAC) vascular invasion. MEDLINE, EMBASE, Cochrane Central, and Scopus were searched until December 2021 for diagnostic accuracy studies comparing MRI vs. CT to evaluate vascular invasion of pathologically confirmed PDAC in the same patients. Findings on resection or exploratory laparotomy were the preferred reference standard. Data extraction, risk of bias, and applicability assessment were performed by two authors using the Quality Assessment of Diagnostic Accuracy Studies-Comparative Tool. Bivariate random-effects meta-analysis and meta-regression were performed with 95% confidence intervals (95% CI). Three studies were included assessing 474 vessels without vascular invasion and 65 with vascular invasion in 107 patients. All patients were imaged using MRI at ≥ 1.5T and a pancreatic protocol CT. No difference was shown between MRI and CT for diagnosing PDAC vascular invasion: MRI/CT sensitivity (95% CI) were 71% (47-87%)/74% (56-86%), and specificity were 97% (94-99%)/97% (94-98%). Sources of bias included selection bias from only a subset of CT patients undergoing MRI and verification bias from patients with unresectable disease not confirmed on surgery. No patients received neoadjuvant therapy prior to staging. Based on limited data, no difference was observed between MRI and pancreatic protocol CT for PDAC vascular invasion assessment. MRI may be an adequate substitute for pancreatic protocol CT in some patients, particularly those who have already had a single-phase CT. Larger and more recent cohort studies at low risk of bias, including patients who have received neoadjuvant therapy, are needed. Abdominal MRI performed similarly to pancreatic protocol CT at assessing pancreatic ductal adenocarcinoma vascular invasion, suggesting local staging is adequate in some patients using MRI. More data are needed using larger, more recent cohorts including patients with neoadjuvant treatment. • Based on limited data, no difference was found between MRI and pancreatic protocol CT sensitivity and specificity for diagnosing PDAC vascular invasion (p = 0.81, 0.73 respectively). • Risk of bias could be reduced in future PDAC MRI vs CT comparative diagnostic test accuracy research by ensuring all enrolled patients undergo both imaging modalities being compared in random order and regardless of the findings on either modality. • More studies are needed that directly compare the diagnostic performance of MRI and CT for PDAC staging after neoadjuvant therapy.

Partial Verification Bias Correction Using Inverse Probability Bootstrap Sampling for Binary Diagnostic Tests.

In medical care, it is important to evaluate any new diagnostic test in the form of diagnostic accuracy studies. These new tests are compared to gold standard tests, where the performance of binary diagnostic tests is usually measured by sensitivity (Sn) and specificity (Sp). However, these accuracy measures are often biased owing to selective verification of the patients, known as partial verification bias (PVB). Inverse probability bootstrap (IPB) sampling is a general method to correct sampling bias in model-based analysis and produces debiased data for analysis. However, its utility in PVB correction has not been investigated before. The objective of this study was to investigate IPB in the context of PVB correction under the missing-at-random assumption for binary diagnostic tests. IPB was adapted for PVB correction, and tested and compared with existing methods using simulated and clinical data sets. The results indicated that IPB is accurate for Sn and Sp estimation as it showed low bias. However, IPB was less precise than existing methods as indicated by the higher standard error (SE). Despite this issue, it is recommended to use IPB when subsequent analysis with full data analytic methods is expected. Further studies must be conducted to reduce the SE.