Reference For Diagnosis Research Articles

Revolution or risk?-Assessing the potential and challenges of GPT-4V in radiologic image interpretation.

ChatGPT-4 Vision (GPT-4V) is a state-of-the-art multimodal large language model (LLM) that may be queried using images. We aimed to evaluate the tool's diagnostic performance when autonomously assessing clinical imaging studies. A total of 206 imaging studies (i.e., radiography (n = 60), CT (n = 60), MRI (n = 60), and angiography (n = 26)) with unequivocal findings and established reference diagnoses from the radiologic practice of a large university hospital were accessed. Readings were performed uncontextualized, with only the image provided, and contextualized, with additional clinical and demographic information. Responses were assessed along multiple diagnostic dimensions and analyzed using appropriate statistical tests. With its pronounced propensity to favor context over image information, the tool's diagnostic accuracy improved from 8.3% (uncontextualized) to 29.1% (contextualized, first diagnosis correct) and 63.6% (contextualized, correct diagnosis among differential diagnoses) (p ≤ 0.001, Cochran's Q test). Diagnostic accuracy declined by up to 30% when 20 images were re-read after 30 and 90 days and seemed unrelated to the tool's self-reported confidence (Spearman's ρ = 0.117 (p = 0.776)). While the described imaging findings matched the suggested diagnoses in 92.7%, indicating valid diagnostic reasoning, the tool fabricated 258 imaging findings in 412 responses and misidentified imaging modalities or anatomic regions in 65 images. GPT-4V, in its current form, cannot reliably interpret radiologic images. Its tendency to disregard the image, fabricate findings, and misidentify details, especially without clinical context, may misguide healthcare providers and put patients at risk. Question Can Generative Pre-trained Transformer 4 Vision (GPT-4V) interpret radiologic images-with and without clinical context? Findings GPT-4V performed poorly, demonstrating diagnostic accuracy rates of 8% (uncontextualized), 29% (contextualized, most likely diagnosis correct), and 64% (contextualized, correct diagnosis among differential diagnoses). Clinical relevance The utility of commercial multimodal large language models, such as GPT-4V, in radiologic practice is limited. Without clinical context, diagnostic errors and fabricated findings may compromise patient safety and misguide clinical decision-making. These models must be further refined to be beneficial.

Impact of Patient-reported Symptom Information on the Interpretation of MRI of the Lumbar Spine.

Background Distinguishing lumbar pain generators from incidental findings at MRI can be difficult. Dictated reports may become lists of findings that cannot be ranked in order of diagnostic importance. Purpose To determine whether patient-reported symptom information can improve the interpretation of lumbar spine MRI by using the spine specialist as the reference standard. Materials and Methods This prospective, single-center, multireader study analyzed 240 participants who completed pre-MRI symptom questionnaires between May 2022 and February 2023. At the time of clinical MRI reporting, radiologists recorded pain generators in consecutive participants, creating two study groups by alternating interpretations with versus without symptom questionnaire results (SQR). Diagnostic certainty was recorded using a numeric scale of 0 to 100. Types, levels, and sides of pain generators were compared with reference diagnoses by calculating Cohen κ values with 95% CIs. Participant characteristics and diagnostic certainties were compared using the Wilcoxon rank sum, Pearson χ2, or Kruskal-Wallis test. Interrater agreement was analyzed. Results There was no difference in age (P = .69) or sex (P = .60) between participants using SQR (n = 120; mean age, 61.0 years; 62 female) and not using SQR (n = 120; mean age, 62.5 years; 67 female). When radiologists were compared with specialists, agreements on pain generators were almost perfect for interpretations using SQR (type: κ = 0.82 [95% CI: 0.74,0.89]; level: κ = 0.88 [95% CI: 0.80, 0.95]; side: κ = 0.84 [95% CI: 0.75, 0.92]), but only fair to moderate for interpretations not using SQR (type: κ = 0.26 [95% CI: 0.15, 0.36]; level: κ = 0.51 [95% CI: 0.39, 0.63]; side: κ = 0.30 [95% CI: 0.18, 0.42]) (all P < .001). Diagnostic certainty was higher for MRI interpretations using SQR (mean, 80.4 ± 14.9 [SD]) than MRI interpretations not using SQR (60.5 ± 17.7) (P < .001). Interrater agreements were substantial (κ = 0.65-0.78) for MRI interpretations using SQR but only fair to moderate (κ = 0.24-0.49) for MRI interpretations not using SQR (all P < .001). Conclusion Patient-reported symptom information enabled radiologists to achieve nearly perfect diagnostic agreement with clinical experts. © RSNA, 2024 See also the editorial by Isikbay and Shah in this issue.

Contrast-enhanced ultrasound appearance of adrenal hemorrhage after orthotopic liver transplantation: a retrospective study.

This study aimed to identify the incidence of adrenal hemorrhage (AH) after OLT and to summarize the ultrasound (US) and contrast-enhanced ultrasound (CEUS) characteristics. Patients with adrenal lesions after OLT at our hospital were retrospectively reviewed between January 2010 and November 2023. The reference diagnosis was defined on the basis of surgical data, computed tomography scans, and magnetic resonance imaging with at least 12 months of follow-up. The incidence of AH and the US and CEUS characteristics after OLT were analyzed and compared with those of adrenal metastases. A total of 23 patients (1.2%) with AH and 7 patients (0.35%) with suprarenal metastases were assessed. Compared with metastases, hematomas had more inhomogeneous echotextures (57% vs. 0.00%, P = 0.010), hypoechoic or mixed-echoic patterns (96% vs. 71%, P = 0.022), and anechoic areas (52% vs. 0.00%, P = 0.024), and their echotextures varied more over time (65% vs. 0.14%, P = 0.031). CEUS was performed on 12 patients with AH and 2 patients with metastases. A "jet-like" contrast superflux was observed in one actively bleeding hematoma, whereas no enhancement was observed in any static hematoma (100%). However, adrenal metastases had a contrast-enhanced appearance in the early arterial phase, followed by fast washout in the late phase (100%), and the difference was statistically significant (P < 0.001). The sonographic characteristics of AH after OLT vary over time. CEUS is recommended when adrenal lesions are detected, as CEUS can differentiate AH from metastases.

Prevalence of human papilloma virus infection and risk of cervical intraepithelial neoplasia among female sex workers in Mumbai, India.

Cervical cancer, mostly caused by human papilloma virus (HPV), has disproportionately high incidence in developing countries. HPV infection being essentially a sexually transmitted infection, high-risk behaviour women with multiple sexual contacts like female sex workers (FSWs) are at higher risk of co-infection with HPV and of developing cervical precancer and cancer. This study aimed to determine the prevalence and determinants of HPV infection and cervical intraepithelial neoplasia (CIN) among FSWs in Mumbai, India. A cross-sectional study was conducted among 448 FSWs, between the ages of 18-50 years, by collaborating with local non-government organizations working for the health and welfare of FSW communities at sexually transmitted diseases clinics. All FSWs were screened for HPV DNA by hybrid capture II followed by reference diagnosis of colposcopy and/or cervical biopsy. The prevalence of HPV DNA positivity was 35.5% and CIN was 2.2%. Factors significantly associated with HPV DNA positivity were age group younger than 30 years odds ratio (OR = 2.098, 95% confidence interval (CI) 1.408-3.127), Illiteracy (OR = 2.015, 95% CI 1.305-3.112), being single (OR = 2.409, 95% CI 1.558-3.724), less than 18 years of age at time of initiating work as FSW (OR = 3.718, 95% CI 3.718-2.392), having more than five clients per day (OR = 2.078, 95% CI 1.301-3.318), been working as a FSW for more than 5 years (OR = 2.321, 95% CI 1.455-3.701), not using barrier contraception methods (OR = 5.155, 95% CI 3.395-7.827) and having no exposure to human immunodeficiency virus (HIV)/acquired immunodeficiency syndrome (AIDS) education program (OR = 29.153, 95% CI 15.385-55.240). FSWs with a positive HPV DNA test are substantially more likely to have CIN compared to those with a negative test (OR = 7.6, 95% CI 1.59-36.25). The prevalence of HPV infection and CIN was high among FSWs. FSWs with a positive HPV DNA test had a seven times higher risk of developing CIN. The persistence of HPV infection is expected to significantly raise the risk of cervical cancer in the future. It is suggested to have an integrated approach towards cervical cancer screening and HIV/AIDS control activities.

A Digital Clock Drawing Test on Tablet for the Diagnosis of Neurocognitive Disorders in Older Adults.

The diagnosis of neurocognitive diseases is an important health issue for patients, families and healthcare professionals. The need to develop rapid, high-performance screening tools would improve access to care. The Clock Drawing Test (CDT) is widely used and validated with the older adults, and its digital version is becoming increasingly widespread. We propose to confirm its validity in a population of old patients hospitalized in a geriatric unit, and secondly to verify its performance in comparison with the reference diagnosis made by a specialized team in a memory consultation. CDTs were collected from older hospitalized patients, both in paper form and digitally on a touchscreen tablet. The results show good agreement between the paper and digital versions (kappa coefficient = 0.81). Sensitivity and specificity of the digital CDT were 0.84 and 0.59 respectively for the diagnosis of major cognitive disorders. The corresponding values were 0.72 and 0.59 for the diagnosis of mild neurocognitive disorders. User questionnaires indicate that older participants find the digital tablet easy to use. However, they prefer to use paper, even if they are open to learning how to use the tablet.

Assessments of arterial and venous phase radiodensity does not improve carotid near-occlusion diagnostics

The hypothesis of this study was that evaluation of radiodensity assessment beyond a carotid stenosis in arterial and/or venous phase can be used to separate near-occlusion and conventional ≥ 50% stenosis. We prospectively included participants with ≥ 50% carotid stenosis with inclusion preference for cases with extracranial internal carotid artery (ICA) asymmetry. All participants were examined with a research biphasic computed tomography angiography (CTA) protocol (arterial and venous phase). Reference diagnosis was set by interpretation on CTA and radiodensity difference between ipsilateral and contralateral ICA (c-corrected) or vertebral (v-corrected) was compared. We included 93 participants, 62 with near-occlusion and 31 with conventional ≥ 50% stenosis. Just beyond the stenosis, median c-corrected radiodensity was − 20 Hounsfield units (HU) among near-occlusions and − 1 HU among conventional ≥ 50% stenoses (p < 0.001) in the arterial phase. For the venous phase, these findings were + 17 HU and + 3 HU (p = 0.007). Similar group differences were seen for v-correction. No parameter had good diagnostic performance, area under the curve ≤ 0.82. With specificity set at ≥ 95%, detected near-occlusions were foremost those with large side-to-side differences in distal ICA-diameter. Carotid near-occlusions can have reduced radiodensity beyond the stenosis in arterial phases and increased radiodensity in venous phases compared to a reference artery—which was not clearly seen for conventional stenoses. However, these radiodensity findings are best seen in near-occlusion cases that are not diagnostically challenging, while they work poorly as additional diagnostic aids.

Multiparametric Cardiac Magnetic Resonance Imaging to Discriminate Endomyocardial Biopsy-Proven Chronic Myocarditis From Healed Myocarditis

BackgroundDetecting ongoing inflammation in myocarditis patients has prognostic relevance, but there are limited data on the detection of chronic myocarditis and its differentiation from healed myocarditis. ObjectivesThis study sought to assess the performance of cardiac magnetic resonance (CMR) for the detection of ongoing inflammation and the discrimination of chronic myocarditis from healed myocarditis. MethodsConsecutive patients with persistent symptoms (>30 days) suggestive of myocarditis were prospectively enrolled from a single tertiary center. All patients underwent a multiparametric 1.5-T CMR protocol including biventricular strain, T1/T2 mapping, and late gadolinium enhancement (LGE). Endomyocardial biopsy was chosen for the reference standard diagnosis. ResultsAmong 452 consecutive patients, 103 (median age: 50 years; 66 men) had evaluable CMR and cardiopathologic reference diagnosis: 53 (51%) with chronic lymphocytic myocarditis and 50 (49%) with healed myocarditis. T2 mapping as a single parameter showed the best accuracy in detecting chronic myocarditis, if abnormal in ≥3 segments (92%; 95% CI: 85-97), and provided the best discrimination from healed myocarditis, as defined by the area under the receiver-operating characteristic curve (0.87 [95% CI: 0.79-0.93]; P < 0.001), followed by radial peak systolic strain rate of the left ventricle (0.86) and the right ventricle (0.84); T1 mapping (0.64), extracellular volume fraction (0.62), and LGE (0.57). Specificity increased when T2 mapping was combined with elevation of either troponin or C-reactive protein. ConclusionsA multiparametric CMR protocol allows detection of ongoing myocardial inflammation and discrimination of chronic myocarditis from healed myocarditis, with segmental T2 mapping and biventricular strain analysis showing higher diagnostic accuracy compared with T1 mapping, extracellular volume fraction, and LGE. The use of biomarkers (troponin or C-reactive protein) may improve specificity.

Double testing with lateral flow antigen test devices for COVID-19: does a second test in quick succession add value?

Background/ObjectivesWe investigated if performing two lateral flow device (LFD) tests, LFD2 immediately after LFD1, could improve diagnostic sensitivity or specificity for detecting severe acute respiratory syndrome-related coronavirus 2 (SARS-CoV-2) antigen. Study DesignIndividuals aged ≥16 years attending UK community testing sites (February–May 2021) performed two successive LFD tests and provided a nose-and-throat sample for a polymerase chain reaction (PCR) test. Using the PCR result as the reference diagnosis, we assessed whether improvements could be achieved in sensitivity (by counting a positive result in either LFD as a positive overall test result) or specificity (by using LFD2 as confirmatory test). ResultsOverall, 2231 participants were included with 159 (7 %) having a positive PCR test. Of 2223 participants who completed both LFD tests, LFD results were highly concordant both with each other and with PCR tests (>97 %). The proportion of discord LFD results decreased significantly over the study period. Combined LFD usage achieved a sensitivity of 68.6 %, versus 67.1 % for either LFD individually. The specificity increased from 99.5 % to 99.8 % when using LFD2 as confirmatory test. Observed increases in sensitivity and specificity were not statistically significant. Void results were recorded for 31 (1.4 %) LFD1s, 19 (0.9 %) LFD2s and 6 (0.3 %) combined LFD tests. ConclusionsLFD tests were highly reproducible even when they were performed by untrained users following only written instructions and without supervision. While performing two LFD tests of the same type in quick succession marginally increased sensitivity or specificity, statistically significant improvements were not detected in our study.

Conventional PCR-based versus next-generation sequencing-based approach for T-cell receptor γ gene clonality assessment in mature T-cell lymphomas: A phase 3 diagnostic accuracy study.

Clonality assessment is currently the major molecular analysis utilized to support the diagnosis of suspicious lymphoid malignancies. Clonal rearrangements of the V-J segments of T-cell receptor G chain locus (TCRγ or TRG) have been observed in almost all types of T neoplasms, such as T-cell-related non-Hodgkin lymphomas and leukemias. At present, the gold standard for clonality evaluation is multiplex polymerase chain reaction (PCR), plus subsequent capillary electrophoresis/heteroduplex analyses, and/or Sanger sequencing. This approach overcomes the problem with the conventional Southern blot hybridization and is more efficient, simple, fast, and reproducible. In the recent years, the new next-generation sequencing (NGS) technologies provided alternative techniques for the analysis of antigen receptors genes, which presented several advantages, such as increased efficiency, specificity (SP), sensitivity (ST), resolution, and objectivity of the results, leading to a better classification, stratification, and monitoring of lymphoid malignancies. Nonetheless, these technologies are still far from being the new gold standard since further studies are warranted to prove their utility. The present study aimed to assess the diagnostic accuracy of these two methods by comparing a commercial NGS-based assay for the evaluation of TRG locus with the gold standard PCR-based one, to fulfill the requirements of a phase 3 diagnostic accuracy study. We assessed the TRG gene rearrangements in 72 cases using the conventional and highly-validated PCR-based assay proposed by EuroClonality consortium, an alternative commercial PCR-based assay, namely, IdentiClone® TCR Gamma Gene Rearrangement Assay 2.0, and a commercial NGS-based assay, that is, Invivoscribe LymphoTrack® Dx MiSeq® (both by Invivoscribe Technologies Inc., San Diego, CA, USA), to determine the diagnostic accuracy of the latter, and compare them with reference diagnoses made based on observation of clinical manifestations, cytohistological, and immunohistochemical analyses. Statistical values were calculated using the Oxford CATmaker software package. Using standardized criteria of interpretation, the obtained results showed a diagnostic accuracy of 90.3% (correspondence in 65 out of 72 cases) of the test under investigation, with a ST of 86%, a SP of 95%, a positive predicting value of 94%, and a negative predicting value of 88%, demonstrating that it had high efficiency and reliability in detecting clonal TRG gene rearrangements in T-cell non-Hodgkin lymphomas. This diagnostic accuracy study yielded comparable results using a validated PCR-based approach and a new NGS-based one. Subsequent studies and cost-effectiveness evaluation are needed to put the NGS-based clonality assessment into routine diagnostic practice.

Do patients with inflammatory bowel disease really know what other immune-mediated inflammatory diseases they are diagnosed with?

The association of inflammatory bowel disease (IBD) with other immune-mediated inflammatory diseases (IMIDs) in the same patient is well known. We aimed to evaluate the degree of knowledge that patients with IBD have regarding the coexistence of other IMIDs and to analyze the factors associated with the concordance between self-reported and confirmed medical information. Patients with IBD at a tertiary hospital answered a questionnaire on the presence of 54 IMIDs (self-reported diagnosis), and their IMID diagnosis was confirmed in their medical records (reference diagnosis). Agreement between the self-reported IMID and the IMID according to medical records was evaluated. The association between concordance and different predictors was evaluated using logistic regression models. A total of 1,620 patients were included. Six hundred and twenty-six (39%) patients were diagnosed with at least one IMID, and 177 (11%) with two or more. Overall agreement between patients´ self-report and medical records was k:0.61. When we grouped IMIDs according to affected organs or systems, agreement on rheumatic IMIDs was moderate (k:0.58), whereas agreement on cutaneous (k:0.66), endocrine (k: 0.74) and ocular (k:0.73) IMIDs was substantial. Among patients who had IMIDs, the factor associated with greater concordance was female gender, while lower concordance was associated with a lower educational level and the fact that the IMID had been diagnosed at the same time or later than IBD. The knowledge that patients with IBD have regarding the coexistence of other IMIDs is poor, especially in rheumatic IMIDs.

Performance of ChatGPT in classifying periodontitis according to the 2018 classification of periodontal diseases

ObjectivesThis study assessed the ability of ChatGPT, an artificial intelligence(AI) language model, to determine the stage, grade, and extent of periodontitis based on the 2018 classification.Materials and methodsThis study used baseline digital data of 200 untreated periodontitis patients to compare standardized reference diagnoses (RDs) with ChatGPT findings and determine the best criteria for assessing stage and grade. RDs were provided by four experts who examined each case. Standardized texts containing the relevant information for each situation were constructed to query ChatGPT. RDs were compared to ChatGPT's responses. Variables influencing the responses of ChatGPT were evaluated.ResultsChatGPT successfully identified the periodontitis stage, grade, and extent in 59.5%, 50.5%, and 84.0% of cases, respectively. Cohen’s kappa values for stage, grade and extent were respectively 0.447, 0.284, and 0.652. A multiple correspondence analysis showed high variance between ChatGPT’s staging and the variables affecting the stage (64.08%) and low variance between ChatGPT’s grading and the variables affecting the grade (42.71%).ConclusionsThe present performance of ChatGPT in the classification of periodontitis exhibited a reasonable level. However, it is expected that additional improvements would increase its effectiveness and broaden its range of functionalities (NCT05926999).Clinical relevanceDespite ChatGPT's current limitations in accurately classifying periodontitis, it is important to note that the model has not been specifically trained for this task. However, it is expected that with additional improvements, the effectiveness and capabilities of ChatGPT might be enhanced.

The accuracy of bedside attention tests for delirium detection in older outpatients with cognitive impairment

BackgroundTo determine the test accuracy of four observations, three explorations and four short tests commonly used in psychiatric examinations to detect attention disorders related to delirium in older outpatients. MethodWe performed a test accuracy study among 86 older patients referred for cognitive screening to an outpatient clinic of a psychiatric institution. Eleven index tests were applied without knowledge of the reference standard. A geriatrician or psychiatrist blinded to the results of the index tests determined the reference diagnosis with DSM-IV-TR criteria. Observations were lack of adequate/ regular eye-contact, dozing off during conversation, dozing off when not stimulated, and losing the point of story. Explored was lack of concentration during activities, difficulty following conversation, and being easily distracted. Serial 7s, WORLD spelled backward, months of the year backward (MOTYB) and counting back from 20 to 0 were applied. Test accuracy was calculated for the single tests and combinations. ResultsDozing off during conversation scored high specificity (97.9 %) as did dozing off when not stimulated (97.9 %), but both had low sensitivity (< 27.9 %). Test accuracy of the exploration questions was low. Serial 7s and WORLD spelled backwards had high sensitivity (92.7 %, respectively 80.5 %) but low specificity (< 51.1 % %). The best combination was MOTYB with WORLD spelled backward (sensitivity of 63.4 %, specificity of 77.8 %). ConclusionsNo single observation, exploration or short test of attention was found to have both high sensitivity and specificity. Probably, attention deficits in delirium are best detected with a combination of observations and short tests as part of a comprehensive psychiatric examination.

Comparing the accuracy of two machine learning models in detection and classification of periapical lesions using periapical radiographs.

Previous deep learning-based studies were mainly conducted on detecting periapical lesions; limited information in classification, such as the periapical index (PAI) scoring system, is available. The study aimed to apply two deep learning models, Faster R-CNN and YOLOv4, in detecting and classifying periapical lesions using the PAI score from periapical radiographs (PR) in three different regions of the dental arch: anterior teeth, premolars, and molars. Out of 2658 PR selected for the study, 2122 PR were used for training, 268 PR were used for validation and 268 PR were used for testing. The diagnosis made by experienced dentists was used as the reference diagnosis. The Faster R-CNN and YOLOv4 models obtained great sensitivity, specificity, accuracy, and precision for detecting periapical lesions. No clear difference in the performance of both models among these three regions was found. The true prediction of Faster R-CNN was 89%, 83.01% and 91.84% for PAI 3, PAI 4 and PAI 5 lesions, respectively. The corresponding values of YOLOv4 were 68.06%, 50.94%, and 65.31%. Our study demonstrated the potential of YOLOv4 and Faster R-CNN models for detecting and classifying periapical lesions based on the PAI scoring system using periapical radiographs.

Flow cytometric analysis of peripheral blood neutrophil myeloperoxidase expression in myelodysplastic neoplasms (MPO-MDS-Valid): protocol for a multicentre diagnostic accuracy study

IntroductionMany patients referred for suspicion of myelodysplastic neoplasm (MDS) are subjected to unnecessary discomfort from bone marrow aspiration, due to the low disease prevalence in this population. Flow cytometric analysis...

Comparing performance of primary care clinicians in the interpretation of SPIROmetry with or without Artificial Intelligence Decision support software (SPIRO-AID): a protocol for a randomised controlled trial

IntroductionSpirometry is a point-of-care lung function test that helps support the diagnosis and monitoring of chronic lung disease. The quality and interpretation accuracy of spirometry is variable in primary care....

Rapid Endoscopic Diagnosis of Benign Ulcerative Colorectal Diseases With an Artificial Intelligence Contextual Framework

Benign ulcerative colorectal diseases (UCDs) such as ulcerative colitis, Crohn's disease, ischemic colitis, and intestinal tuberculosis share similar phenotypes with different etiologies and treatment strategies. To accurately diagnose closely related diseases like UCDs, we hypothesize that contextual learning is critical in enhancing the ability of the artificial intelligence models to differentiate the subtle differences in lesions amidst the vastly divergent spatial contexts. White-light colonoscopy datasets of patients with confirmed UCDs and healthy controls were retrospectively collected. We developed a Multiclass Contextual Classification (MCC) model that can differentiate among the mentioned UCDs and healthy controls by incorporating the tissue object contexts surrounding the individual lesion region in a scene and spatial information from other endoscopic frames (video-level) into a unified framework. Internal and external datasets were used to validate the model's performance. Training datasets included 762 patients, and the internal and external testing cohorts included 257 patients and 293 patients, respectively. Our MCC model provided a rapid reference diagnosis on internal test sets with a high averaged area under the receiver operating characteristic curve (image-level: 0.950 and video-level: 0.973) and balanced accuracy (image-level: 76.1% and video-level: 80.8%), which was superior to junior endoscopists (accuracy: 71.8%, P < .0001) and similar to experts (accuracy: 79.7%, P= .732). The MCC model achieved an area under the receiver operating characteristic curve of 0.988 and balanced accuracy of 85.8% using external testing datasets. These results enable this model to fit in the routine endoscopic workflow, and the contextual framework to be adopted for diagnosing other closely related diseases.

Health equity assessment of machine learning performance (HEAL): a framework and dermatology AI model case study

Artificial intelligence (AI) has repeatedly been shown to encode historical inequities in healthcare. We aimed to develop a framework to quantitatively assess the performance equity of health AI technologies and to illustrate its utility via a case study. Here, we propose a methodology to assess whether health AI technologies prioritise performance for patient populations experiencing worse outcomes, that is complementary to existing fairness metrics. We developed the Health Equity Assessment of machine Learning performance (HEAL) framework designed to quantitatively assess the performance equity of health AI technologies via a four-step interdisciplinary process to understand and quantify domain-specific criteria, and the resulting HEAL metric. As an illustrative case study (analysis conducted between October 2022 and January 2023), we applied the HEAL framework to a dermatology AI model. A set of 5420 teledermatology cases (store-and-forward cases from patients of 20 years or older, submitted from primary care providers in the USA and skin cancer clinics in Australia), enriched for diversity in age, sex and race/ethnicity, was used to retrospectively evaluate the AI model's HEAL metric, defined as the likelihood that the AI model performs better for subpopulations with worse average health outcomes as compared to others. The likelihood that AI performance was anticorrelated to pre-existing health outcomes was estimated using bootstrap methods as the probability that the negated Spearman's rank correlation coefficient (i.e., "R") was greater than zero. Positive values of R suggest that subpopulations with poorer health outcomes have better AI model performance. Thus, the HEAL metric, defined as p (R>0), measures how likely the AI technology is to prioritise performance for subpopulations with worse average health outcomes as compared to others (presented as a percentage below). Health outcomes were quantified as disability-adjusted life years (DALYs) when grouping by sex and age, and years of life lost (YLLs) when grouping by race/ethnicity. AI performance was measured as top-3 agreement with the reference diagnosis from a panel of 3 dermatologists per case. Across all dermatologic conditions, the HEAL metric was 80.5% for prioritizing AI performance of racial/ethnic subpopulations based on YLLs, and 92.1% and 0.0% respectively for prioritizing AI performance of sex and age subpopulations based on DALYs. Certain dermatologic conditions were significantly associated with greater AI model performance compared to a reference category of less common conditions. For skin cancer conditions, the HEAL metric was 73.8% for prioritizing AI performance of age subpopulations based on DALYs. Analysis using the proposed HEAL framework showed that the dermatology AI model prioritised performance for race/ethnicity, sex (all conditions) and age (cancer conditions) subpopulations with respect to pre-existing health disparities. More work is needed to investigate ways of promoting equitable AI performance across age for non-cancer conditions and to better understand how AI models can contribute towards improving equity in health outcomes. Google LLC.

Detection of acute pulmonary embolism using native repeated magnetic resonance imaging acquisitions under free-breathing and without respiratory or cardiac gating. A diagnostic accuracy study

ObjectivesComputed tomography pulmonary angiography (CTPA) is the gold standard diagnostic method for patients with suspected pulmonary embolism (PE), but it has its drawbacks, including exposure to ionizing radiation and iodinated contrast agent. The present study aims to evaluate the diagnostic performance of our in-house developed non-contrast MRI protocol for PE diagnosis in reference to CTPA. Methods107 patients were included, all of whom underwent MRI immediately before or within 36 hours after CTPA. Additional cases examined only with MRI and a negative result were added to reach a PE prevalence of approximately 20%. The protocol was a non-contrast 2D steady-state free precession (SSFP) sequence under free-breathing, without respiratory or cardiac gating, and repeated five times to capture the vessels at different breathing/cardiac phases. The MRIs were blinded and read by two radiologists and the results were compared to CTPA. ResultsOf the 243 patients included, 47 were positive for PE. Readers 1 and 2 demonstrated 89% and 87% sensitivity, 100% specificity, 98% accuracy and Cohen’s kappa of 0.88 on patient level. In the per embolus comparison, readers 1 and 2 detected, 60 and 59/61 (98, 97%) proximal, 101 and 94/113 (89, 83%) segmental, and 5 and 2/32 (16, 6%) subsegmental emboli, resulting in 81 and 75% sensitivity respectively. ConclusionThe repeated 2D SSFP can reliably be used for the diagnosis of acute PE at the proximal and segmental artery levels.

Perfluorobutane (Sonazoid) Contrast-enhanced ultrasound to diagnose hepatocellular carcinoma: A systematic review and meta-analysis.

This meta-analysis aimed to assess the precision of Sonazoid contrast-enhanced ultrasound (CEUS) to distinguish hepatocellular carcinoma (HCC) from focal liver lesions (FLLs). The Cochrane Library, Embase, PubMed, and Web of Science databases were systematically searched and checked for studies using Sonazoid CEUS to characterize HCC. A comprehensive meta-analysis was conducted, involving data pooling, subgroup analyses, meta-regression, and investigation of publication bias. The meta-analysis included fourteen studies. The overall diagnostic accuracy for characterizing HCC was as follows (all ranges show the 95% confidence interval): pooled sensitivity of 0.87 (0.80-0.92), pooled specificity of 0.95 (0.91-0.97), and a diagnostic odds ratio of 121 (61-241). The overall weighted area under the curve was 0.97 (0.95-0.98). The pooled sensitivity, specificity, and diagnostic odds ratio for Sonazoid and Sonovue were 0.75 (0.63- 0.84), 0.97 (0.86-0.99), 82 (15-445); and 0.64 (0.51-0.76), 0.98 (0.91-0.99), 72 (17-311), respectively. The sources of heterogeneity were identified as the study location, prevailing risk factor, reference diagnosis standard, criteria of Sonazoid CUES, and the proportion of cases of HCC. We observed no potential publication bias. Sonazoid CEUS is efficient to distinguish HCC from FLLs, with good sensitivity and specificity. It is comparable to Sonovue CEUS to diagnose HCC.

Validating the accuracy of deep learning for the diagnosis of pneumonia on chest x-ray against a robust multimodal reference diagnosis: a post hoc analysis of two prospective studies.

Artificial intelligence (AI) seems promising in diagnosing pneumonia on chest x-rays (CXR), but deep learning (DL) algorithms have primarily been compared with radiologists, whose diagnosis can be not completely accurate. Therefore, we evaluated the accuracy of DL in diagnosing pneumonia on CXR using a more robust reference diagnosis. We trained a DL convolutional neural network model to diagnose pneumonia and evaluated its accuracy in two prospective pneumonia cohorts including 430 patients, for whom the reference diagnosis was determined a posteriori by a multidisciplinary expert panel using multimodal data. The performance of the DL model was compared with that of senior radiologists and emergency physicians reviewing CXRs and that of radiologists reviewing computed tomography (CT) performed concomitantly. Radiologists and DL showed a similar accuracy on CXR for both cohorts (p ≥ 0.269): cohort 1, radiologist 1 75.5% (95% confidence interval 69.1-80.9), radiologist 2 71.0% (64.4-76.8), DL 71.0% (64.4-76.8); cohort 2, radiologist 70.9% (64.7-76.4), DL 72.6% (66.5-78.0). The accuracy of radiologists and DL was significantly higher (p ≤ 0.022) than that of emergency physicians (cohort 1 64.0% [57.1-70.3], cohort 2 63.0% [55.6-69.0]). Accuracy was significantly higher for CT (cohort 1 79.0% [72.8-84.1], cohort 2 89.6% [84.9-92.9]) than for CXR readers including radiologists, clinicians, and DL (all p-values < 0.001). When compared with a robust reference diagnosis, the performance of AI models to identify pneumonia on CXRs was inferior than previously reported but similar to that of radiologists and better than that of emergency physicians. The clinical relevance of AI models for pneumonia diagnosis may have been overestimated. AI models should be benchmarked against robust reference multimodal diagnosis to avoid overestimating its performance. NCT02467192 , and NCT01574066 . • We evaluated an openly-access convolutional neural network (CNN) model to diagnose pneumonia on CXRs. • CNN was validated against a strong multimodal reference diagnosis. • In our study, the CNN performance (area under the receiver operating characteristics curve 0.74) was lower than that previously reported when validated against radiologists' diagnosis (0.99 in a recent meta-analysis). • The CNN performance was significantly higher than emergency physicians' (p ≤ 0.022) and comparable to that of board-certified radiologists (p ≥ 0.269).