Generalisability Of Model Research Articles

Diagnostic performance of deep learning for infectious keratitis: a systematic review and meta-analysis

Infectious keratitis (IK) is the leading cause of corneal blindness globally. Deep learning (DL) is an emerging tool for medical diagnosis, though its value in IK is unclear. We aimed to assess the diagnostic accuracy of DL for IK and its comparative accuracy with ophthalmologists. In this systematic review and meta-analysis, we searched EMBASE, MEDLINE, and clinical registries for studies related to DL for IK published between 1974 and July 16, 2024. We performed meta-analyses using bivariate models to estimate summary sensitivities and specificities. This systematic review was registered with PROSPERO (CRD42022348596). Of 963 studies identified, 35 studies (136,401 corneal images from >56,011 patients) were included. Most studies had low risk of bias (68.6%) and low applicability concern (91.4%) in all domains of QUADAS-2, except the index test domain. Against the reference standard of expert consensus and/or microbiological results (seven external validation studies; 10,675 images), the summary estimates (95% CI) for sensitivity and specificity of DL for IK were 86.2% (71.6-93.9) and 96.3% (91.5-98.5). From 28 internal validation studies (16,059 images), summary estimates for sensitivity and specificity were 91.6% (86.8-94.8) and 90.7% (84.8-94.5). Based on seven studies (4007 images), DL and ophthalmologists had comparable summary sensitivity [89.2% (82.2-93.6) versus 82.2% (71.5-89.5); P=0.20] and specificity [(93.2% (85.5-97.0) versus 89.6% (78.8-95.2); P=0.45]. DL models may have good diagnostic accuracy for IK and comparable performance to ophthalmologists. These findings should be interpreted with caution due to the image-based analysis that did not account for potential correlation within individuals, relatively homogeneous population studies, lack of pre-specification of DL thresholds, and limited external validation. Future studies should improve their reporting, data diversity, external validation, transparency, and explainability to increase the reliability and generalisability of DL models for clinical deployment. NIH, Wellcome Trust, MRC, Fight for Sight, BHP, and ESCRS.

Artificial intelligence in cardiology: a machine learning model for supraventricular tachycardia discrimination

Abstract Background Distinguishing atrioventricular nodal reentrant tachycardia (AVNRT) from orthodromic atrioventricular reentrant tachycardia (AVRT) presents a diagnostic challenge, with clinical and electrocardiogram (ECG) features playing a crucial role in differential diagnosis. Purpose The aim of this study was to devise and validate a predictive algorithm that utilises basic clinical and ECG parameters to accurately classify AVNRT versus AVRT. Furthermore, this algorithm was integrated into a user-friendly, web-based tool to support clinical decision-making. Methods We conducted a prospective analysis of 704 patients (median age: 53.6 years; 59.7% female) with electrophysiologically confirmed AVNRT (n=535) or AVRT (n=169). A comprehensive evaluation was performed using a structured questionnaire tailored to arrhythmia-specific symptoms. The study utilised advanced machine learning techniques for classification, employing feature selection methods to enhance predictive accuracy. The analysis included 19 demographic, clinical, and ECG variables, adopting an 80-20% split for training and internal validation. The algorithm's performance was evaluated against a model comprising all variables (Full Model), expert ECG interpretations (ExpertECG), objective ECG criteria (ECG model), and a baseline dummy classifier, with efficacy assessed by the Area Under the Receiver Operating Characteristic curve (AUC) and the Area Under the Precision-Recall Curve (AUPRC). Results The optimised model, utilising the XGBoost algorithm with six critical variables (initial age at first episode, biological sex, presence of neck palpitations, pseudoR wave in lead V1/pseudoS wave in inferior leads, visibility of retrograde P-waves, and tachycardia cycle length), exhibited superior performance, achieving an AUC of 0.916 and an AUPRC of 0.823. This model significantly outperformed other comparative models in terms of predictive accuracy (Figure 1). Implementation of the model into a web-based interface was accomplished using Shiny, enhancing its accessibility for clinical application (Figure 2). Conclusions This study illustrates the efficacy of employing machine learning algorithms to utilise bedside clinical and ECG variables for accurately differentiating between AVNRT and AVRT. The creation of a web-based calculator marks a significant advancement, empowering clinicians, especially non-specialists, with a potent tool for providing personalised diagnostic insights. Future research should focus on external validation to ascertain the model's generalisability and its applicability in broader clinical settings.Figure 1Figure 2

Multi-vendor validation of automated left ventricular analysis from 3D echocardiography using artificial intelligence

Abstract Background Accurate echocardiographic assessment of left ventricular (LV) function is dependent on both image acquisition and analysis. Although 3D echocardiography (3DE) is increasingly recommended for volume quantification, manually derived volumes are limited by operator subjectivity, variable image quality, and the lengthy duration of analysis. Consequently, several automated approaches based on artificial intelligence (AI) have been proposed to segment the LV cavity from 3DE and have demonstrated better accuracy and reproducibility compared to human experts. While the utility of AI models in the clinic depends on their ability to generalise to different populations and ultrasound vendors, the external validation of developed models remains limited. Purpose This study evaluates the ability of an existing AI model trained using data from a single vendor to accurately estimate routine 3DE LV indices in an independent external dataset acquired using equipment from a different vendor. Methods A previously published and internally validated deep learning model for segmentation of the LV cavity and myocardium from 3DE was used to automatically derive LV indices including end-diastolic volume (EDV), end-systolic volume (ESV), LV mass, and ejection fraction (EF) [1]. A second independent dataset consisting of paired 3DE and CMR images from 65 marathon runners from a different centre (using equipment from a different vendor) was subsequently used for external validation. Images from both datasets were subjectively scored from 1 (poor) to 5 (excellent) as a measure of image quality. Agreement with CMR as the reference was assessed using a two-way mixed effects intraclass correlation coefficient (ICC), supplemented by Bland-Altman analysis, to determine the model bias (between 3DE and CMR) and 95% limits of agreement (LOA). To assess whether performance could be improved by inclusion of multi-vendor data during training, the initial model was fine-tuned using a small subset (n=9) of images from the second centre. Results The mean image quality score from the second dataset was 2.5 (± 1.1), which was significantly lower than the mean of 3.4 (± 1.0) from the first dataset. Despite vendor and image quality differences, the initial model produced good agreement with CMR in EDV, ESV, and LV mass but only poor agreement in EF (Fig 1). Fine tuning greatly improved agreement in EF with reduced bias and narrower 95% LOA (Fig 2). Conclusion With the addition of a small number of cases for fine-tuning, AI models for automated LV analysis can achieve good performance analysing 3DE images acquired from a different vendor and population. Advancing model generalisability through more sophisticated data augmentation strategies may enable truly vendor-agnostic automated LV analysis from 3DE.

Prediction of pre-eclampsia with machine learning approaches: Leveraging important information from routinely collected data

BackgroundGlobally, pre-eclampsia (PE) is a leading cause of maternal and perinatal morbidity and mortality. PE prediction using routinely collected data has the advantage of being widely applicable, particularly in low-resource settings. Early intervention for high-risk women might reduce PE incidence and related complications. We aimed to replicate our machine learning (ML) published work predicting another maternal condition (gestational diabetes) to (1) predict PE using routine health data, (2) identify the optimal ML model, and (3) compare it with logistic regression approach. MethodsData were from a large health service network with 48,250 singleton pregnancies between January 2016 and June 2021. Supervised ML models were employed. Maternal clinical and medical characteristics were the feature variables (predictors), and a 70/30 data split was used for training and testing the model. Predictive performance was assessed using area under the curve (AUC) and calibration plots. Shapley value analysis assessed the contribution of feature variables. ResultsThe random forest approach provided excellent discrimination with an AUC of 0.84 (95% CI: 0.82–0.86) and highest prediction accuracy (0.79); however, the calibration curve (slope of 1.21, 95% CI 1.13–1.30) was acceptable only for a threshold of 0.3 or less. The next best approach was extreme gradient boosting, which provided an AUC of 0.77 (95% CI: 0.76–0.79) and well-calibrated (slope of 0.93, 95% CI 0.85–1.01). Logistic regression provided good discrimination performance with an AUC of 0.75 (95% CI: 0.74–0.76) and perfect calibration. Nulliparous, pre-pregnancy body mass index, previous pregnancy with prior PE, maternal age, family history of hypertension, and pre-existing hypertension and diabetes were the top-ranked features in Shapley value analysis. ConclusionTwo ML models created the highest-performing prediction using routinely collected data to identify women at high risk of PE, with acceptable discrimination. However, to confirm this result and also examine model generalisability, external validation studies are needed in other settings, utilising standardised prognostic factors.

Machine learning in paediatric haematological malignancies: a systematic review of prognosis, toxicity and treatment response models.

Machine Learning (ML) has demonstrated potential in enhancing care in adult oncology. However, its application in paediatric haematological malignancies is still emerging, necessitating a comprehensive review of its capabilities and limitations in this area. A literature search was conducted through Ovid. Studies included focused on ML models in paediatric patients with haematological malignancies. Studies were categorised into thematic groups for analysis. Twenty studies, primarily on leukaemia, were included in this review. Studies were organised into thematic categories such as prognoses, treatment responses and toxicity predictions. Prognostic studies showed AUC scores between 0.685 and 0.929, indicating moderate-high predictive accuracy. Treatment response studies demonstrated AUC scores between 0.840 and 0.875, reflecting moderate accuracy. Toxicity prediction studies reported high accuracy with AUC scores from 0.870 to 0.927. Only five studies (25%) performed external validation. Significant heterogeneity was noted in ML tasks, reporting formats, and effect measures across studies, highlighting a lack of standardised reporting and challenges in data comparability. The clinical applicability of these ML models remains limited by the lack of external validation and methodological heterogeneity. Addressing these challenges through standardised reporting and rigorous external validation is needed to translate ML from a promising research tool into a reliable clinical practice component. Key message: Machine Learning (ML) significantly enhances predictive models in paediatric haematological cancers, offering new avenues for personalised treatment strategies. Future research should focus on developing ML models that can integrate with real-time clinical workflows. Addition to literature: Provides a comprehensive overview of current ML applications and trends. It identifies limitations to its applicability, including the limited diversity in datasets, which may affect the generalisability of ML models across different populations. Encourages standardisation and external validation in ML studies, aiming to improve patient outcomes through precision medicine in paediatric haematological oncology.

Ancestry-aligned polygenic scores combined with conventional risk factors improve prediction of cardiometabolic outcomes in African populations

BackgroundCardiovascular diseases (CVD) are a major health concern in Africa. Improved identification and treatment of high-risk individuals can reduce adverse health outcomes. Current CVD risk calculators are largely unvalidated in African populations and overlook genetic factors. Polygenic scores (PGS) can enhance risk prediction by measuring genetic susceptibility to CVD, but their effectiveness in genetically diverse populations is limited by a European-ancestry bias. To address this, we developed models integrating genetic data and conventional risk factors to assess the risk of developing cardiometabolic outcomes in African populations.MethodsWe used summary statistics from a genome-wide association meta-analysis (n = 14,126) in African populations to derive novel genome-wide PGS for 14 cardiometabolic traits in an independent African target sample (Africa Wits-INDEPTH Partnership for Genomic Research (AWI-Gen), n = 10,603). Regression analyses assessed relationships between each PGS and corresponding cardiometabolic trait, and seven CVD outcomes (CVD, heart attack, stroke, diabetes mellitus, dyslipidaemia, hypertension, and obesity). The predictive utility of the genetic data was evaluated using elastic net models containing multiple PGS (MultiPGS) and reference-projected principal components of ancestry (PPCs). An integrated risk prediction model incorporating genetic and conventional risk factors was developed. Nested cross-validation was used when deriving elastic net models to enhance generalisability.ResultsOur African-specific PGS displayed significant but variable within- and cross- trait prediction (max.R2 = 6.8%, p = 1.86 × 10−173). Significantly associated PGS with dyslipidaemia included the PGS for total cholesterol (logOR = 0.210, SE = 0.022, p = 2.18 × 10−21) and low-density lipoprotein (logOR = − 0.141, SE = 0.022, p = 1.30 × 10−20); with hypertension, the systolic blood pressure PGS (logOR = 0.150, SE = 0.045, p = 8.34 × 10−4); and multiple PGS associated with obesity: body mass index (max. logOR = 0.131, SE = 0.031, p = 2.22 × 10−5), hip circumference (logOR = 0.122, SE = 0.029, p = 2.28 × 10−5), waist circumference (logOR = 0.013, SE = 0.098, p = 8.13 × 10−4) and weight (logOR = 0.103, SE = 0.029, p = 4.89 × 10−5). Elastic net models incorporating MultiPGS and PPCs significantly improved prediction over MultiPGS alone. Models including genetic data and conventional risk factors were more predictive than conventional risk models alone (dyslipidaemia: R2 increase = 2.6%, p = 4.45 × 10−12; hypertension: R2 increase = 2.6%, p = 2.37 × 10−13; obesity: R2 increase = 5.5%, 1.33 × 10−34).ConclusionsIn African populations, CVD and associated cardiometabolic trait prediction models can be improved by incorporating ancestry-aligned PGS and accounting for ancestry. Combining PGS with conventional risk factors further enhances prediction over traditional models based on conventional factors. Incorporating data from target populations can improve the generalisability of international predictive models for CVD and associated traits in African populations.

Modelling the nicotine pharmacokinetic profile for e-cigarettes using real time monitoring of consumers’ physiological measurements and mouth level exposure

Pharmacokinetic (PK) studies can provide essential information on abuse liability of nicotine and tobacco products but are intrusive and must be conducted in a clinical environment. The objective of the study was to explore whether changes in plasma nicotine levels following use of an e-cigarette can be predicted from real time monitoring of physiological parameters and mouth level exposure (MLE) to nicotine before, during, and after e-cigarette vaping, using wearable devices. Such an approach would allow an -effective pre-screening process, reducing the number of clinical studies, reducing the number of products to be tested and the number of blood draws required in a clinical PK study Establishing such a prediction model might facilitate the longitudinal collection of data on product use and nicotine expression among consumers using nicotine products in their normal environments, thereby reducing the need for intrusive clinical studies while generating PK data related to product use in the real world.An exploratory machine learning model was developed to predict changes in plasma nicotine levels following the use of an e-cigarette; from real time monitoring of physiological parameters and MLE to nicotine before, during, and after e-cigarette vaping. This preliminary study identified key parameters, such as heart rate (HR), heart rate variability (HRV), and physiological stress (PS) that may act as predictors for an individual’s plasma nicotine response (PK curve). Relative to baseline measurements (per participant), HR showed a significant increase for nicotine containing e-liquids and was consistent across sessions (intra-participant). Imputing missing values and training the model on all data resulted in 57% improvement from the original’learning’ data and achieved a median validation R2 of 0.70.The study is in its exploratory phase, with limitations including a small and non-diverse sample size and reliance on data from a single e-cigarette product. These findings necessitate further research for validation and to enhance the model's generalisability and applicability in real-world settings. This study serves as a foundational step towards developing non-intrusive PK models for nicotine product use.

Proteomic prediction of diverse incident diseases: a machine learning-guided biomarker discovery study using data from a prospective cohort study

Broad-capture proteomic technologies have the potential to improve disease prediction, enabling targeted prevention and management, but studies have so far been limited to very few selected diseases and have not evaluated predictive performance across multiple conditions. We aimed to evaluate the potential of serum proteins to improve risk prediction over and above health-derived information and polygenic risk scores across a diverse set of 24 outcomes. We designed multiple case-cohorts nested in the EPIC-Norfolk prospective study, from participants with available serum samples and genome-wide genotype data, with more than 32 974 person-years of follow-up. Participants were middle-aged individuals (aged 40-79 years at baseline) of European ancestry who were recruited from the general population of Norfolk, England, between March, 1993 and December, 1997. We selected participants who developed one of ten less common diseases within 10 years of follow-up; we also subsampled a randomly drawn control subcohort, which also served to investigate 14 more common outcomes (n>70), including all-cause premature mortality (death before the age of 75 years; case numbers 71-437; controls 608-1556). Individuals were excluded from the current study owing to failed genotyping or proteomic quality control, relatedness, or missing information on age, sex, BMI, or smoking status. We used a machine learning framework to derive sparse predictive protein models for the onset of the the 23 individual diseases and all-cause premature mortality, and to derive a single common sparse multimorbidity signature that was predictive across multiple diseases from 2923 serum proteins. Participants who developed one of ten less common diseases within 10 years of follow-up included 482 women and 507 men, with a mean age at baseline of 64·56 years (8·08). The random subcohort included 990 women and 769 men, with a mean age of 58·79 years (9·31). As few as five proteins alone outperformed polygenic risk scores for 17 of 23 outcomes (median dfference in concordance index [C-index] 0·13 [0·10-0·17]) and improved predictive performance when added over basic patient-derived information models for seven outcomes, achieving a median C-index of 0·82 (IQR 0·77-0·82). This included diseases with poor prognosis such as lung cancer (C-index 0·85 [+/- cross-validation error 0·83-0·87]), for which we identified unreported biomarkers such as C-X-C motif chemokine ligand 17. A sparse multimorbidity signature of ten proteins improved prediction across seven outcomes over patient-derived information models, achieving performances (median C-index 0·81 [IQR 0·80-0·82]) similar to those of disease-specific signatures. We show the value of broad-capture proteomic biomarker discovery studies across multiple diseases of diverse causes, pointing to those that might benefit the most from proteomic approaches, and the potential to derive common sparse biomarker panels for prediction of multiple diseases at once. This framework could enable follow-up studies to explore the generalisability of proteomic models and to benchmark these against clinical assays, which are required to understand the translational potential of these findings. Medical Research Council, Health Data Research UK, UK Research and Innovation-National Institute for Health and Care Research, Cancer Research UK, and Wellcome Trust.

Eight Principles for Intelligence Sharing: A Holistic and Strategic Approach

This paper reviews the strategic use of warning intelligence to pre-emptively address threats in complex geopolitical scenarios through rapid intelligence sharing. Specifically, the paper reviews the question How, based on research and experience, can a set of principles be applied by states to enhance situational awareness and tackle threat actors through a holistic and collaborative approach to intelligence sharing? The paper examines historical and contemporary alliances like the Five Eyes and reviews a Signals Intelligence Alliance as a case in point, highlighting the importance of collaborative approaches to enhance situational awareness and tackle threat actors. The study, grounded in the philosophical paradigm of interpretivism, adheres to the principles for transparent science when researchers use tools such as large-language-models as grammar editors or research assistants. The paper also acknowledges limitations such as the generalisability of the SIGINT model and the need for continuous adaptation of intelligence sharing practices. The results discuss policy, process, and people challenges to intelligence sharing. The paper concludes that successful intelligence sharing should follow eight general principles. Future research directions include exploring the impact of emerging technologies, human aspects of intelligence sharing, and context-based intelligence sharing alliances.

Neural equivalent circuit models: Universal differential equations for battery modelling

Current battery modelling methodologies including equivalent circuital modelling and electrochemical modelling do not maintain accuracy over diverse operating conditions of current rates, depth-of-discharge and temperatures. To address this limitation, this article proposes the Universal Differential Equations (UDE) framework from scientific machine learning (SciML) as a methodology to generate battery models with improved generalisability. The effectiveness of UDE in enhancing generalisability is demonstrated through a specific battery modelling example. The approach starts with the Thermal Equivalent Circuital Model with Diffusion (TECMD), a state-of-the-art battery model, which is then enhanced through the integration of neural networks into its state equations, resulting in the Neural-TECMD; a UDE model. Additionally, a two-stage UDE parameterisation method is introduced, combining collocation-based pretraining with mini-batch training. The parameterisation method enables the neural networks in the Neural-TECMD to efficiently learn battery dynamics from multiple time series data sets, covering a wide operating spectrum. Consequently, the Neural-TECMD model offers accurate predictions over broader operating conditions, thus enhancing model generalisability. The Neural-TECMD model was validated using 20 data sets covering current rates of 0 to 2C and temperatures from 0 to 45 °C. This validation revealed substantial improvements in accuracy, with an average of 34.51% decrease in RMSE for voltage and a 24.94% decrease for temperature predictions compared to the standard TECMD model.

Identification and validation of gestational diabetes subgroups by data-driven cluster analysis

Aims/hypothesisGestational diabetes mellitus (GDM) is a heterogeneous condition. Given such variability among patients, the ability to recognise distinct GDM subgroups using routine clinical variables may guide more personalised treatments. Our main aim was to identify distinct GDM subtypes through cluster analysis using routine clinical variables, and analyse treatment needs and pregnancy outcomes across these subgroups.MethodsIn this cohort study, we analysed datasets from a total of 2682 women with GDM treated at two central European hospitals (1865 participants from Charité University Hospital in Berlin and 817 participants from the Medical University of Vienna), collected between 2015 and 2022. We evaluated various clustering models, including k-means, k-medoids and agglomerative hierarchical clustering. Internal validation techniques were used to guide best model selection, while external validation on independent test sets was used to assess model generalisability. Clinical outcomes such as specific treatment needs and maternal and fetal complications were analysed across the identified clusters.ResultsOur optimal model identified three clusters from routinely available variables, i.e. maternal age, pre-pregnancy BMI (BMIPG) and glucose levels at fasting and 60 and 120 min after the diagnostic OGTT (OGTT0, OGTT60 and OGTT120, respectively). Cluster 1 was characterised by the highest OGTT values and obesity prevalence. Cluster 2 displayed intermediate BMIPG and elevated OGTT0, while cluster 3 consisted mainly of participants with normal BMIPG and high values for OGTT60 and OGTT120. Treatment modalities and clinical outcomes varied among clusters. In particular, cluster 1 participants showed a much higher need for glucose-lowering medications (39.6% of participants, compared with 12.9% and 10.0% in clusters 2 and 3, respectively, p<0.0001). Cluster 1 participants were also at higher risk of delivering large-for-gestational-age infants. Differences in the type of insulin-based treatment between cluster 2 and cluster 3 were observed in the external validation cohort.Conclusions/interpretationOur findings confirm the heterogeneity of GDM. The identification of subgroups (clusters) has the potential to help clinicians define more tailored treatment approaches for improved maternal and neonatal outcomes.Graphical

Synthetic Data for Deep Learning in Computer Vision &amp; Medical Imaging: A Means to Reduce Data Bias

Deep-learning (DL) performs well in computer-vision and medical-imaging automated decision-making applications. A bottleneck of DL stems from the large amount of labelled data required to train accurate models that generalise well. Data scarcity and imbalance are common problems in imaging applications that can lead DL models towards biased decision making. A solution to this problem is synthetic data. Synthetic data is an inexpensive substitute to real data for improved accuracy and generalisability of DL models. This survey reviews the recent methods published in relation to the creation and use of synthetic data for computer-vision and medical-imaging DL applications. The focus will be on applications that utilised synthetic data to improve DL models by either incorporating an increased diversity of data that is difficult to obtain in real life, or by reducing a bias caused by class imbalance. Computer-graphics software and generative networks are the most popular data generation techniques encountered in the literature. We highlight their suitability for typical computer-vision and medical-imaging applications, and present promising avenues for research to overcome their computational and theoretical limitations.

Improving cross-site generalisability of vision-based solar forecasting models with physics-informed transfer learning

Forecasting solar energy from cloud cover observations is crucial to truly anticipate future changes in power supply. On an intra-hour timescale, ground-level sky cameras located near a solar site offer the most valuable source of information on incoming clouds. In the literature, the analysis of these hyperlocal cloud cover observations for solar modelling is increasingly performed by deep learning algorithms trained and tested on years’ worth of local data. However, this approach is not suitable for industrial applications since solar energy producers cannot wait for years of local data collection to start generating reliable solar forecasts. However, they might own relevant multi-location data collected from other solar sites over time. This study thus explores the capability of such algorithms to generalise beyond their training location in two data scarce conditions: zero-shot learning (i.e. direct application of a trained model to a new location without local fine-tuning) and few-shot learning (i.e. calibration of a pre-trained model based on very limited local data such as a day of observations). Zero-shot learning results show that using local clear-sky models to normalise output variables (e.g. solar irradiance or solar energy production values) facilitates cross-dataset transfer learning. Compared to previous methods, the resulting forecast skill increases by close to 25% in cloudy conditions and by more than 700% in clear-sky conditions. An additional gain is observed when local data collected in overcast weather conditions are used for model calibration via few-shot learning. The corresponding neural networks trained in data scarce conditions achieve comparable performance to expert local models based on years of training data. These promising results shed light on the potential of large-scale and multi-location sky image datasets to improve the generalisation skills of solar forecasting algorithms.

Benchmarking proton RBE models

Objective. To biologically optimise proton therapy, models which can accurately predict variations in proton relative biological effectiveness (RBE) are essential. Current phenomenological models show large disagreements in RBE predictions, due to different model assumptions and differences in the data to which they were fit. In this work, thirteen RBE models were benchmarked against a comprehensive proton RBE dataset to evaluate predictions when all models are fit using the same data and fitting techniques, and to assess the statistical robustness of the models. Approach. Model performance was initially evaluated by fitting to the full dataset, and then a cross-validation approach was applied to assess model generalisability and robustness. The impact of weighting the fit and the choice of biological endpoint (either single or multiple survival levels) was also evaluated. Main results. Fitting the models to a common dataset reduced differences between their predictions, however significant disagreements remained due to different underlying assumptions. All models performed poorly under cross-validation in the weighted fits, suggesting that some uncertainties on the experimental data were significantly underestimated, resulting in over-fitting and poor performance on unseen data. The simplest model, which depends linearly on the LET but has no tissue or dose dependence, performed best for a single survival level. However, when fitting to multiple survival levels simultaneously, more complex models with tissue dependence performed better. All models had significant residual uncertainty in their predictions compared to experimental data. Significance. This analysis highlights that poor quality of error estimation on the dose response parameters introduces substantial uncertainty in model fitting. The significant residual error present in all approaches illustrates the challenges inherent in fitting to large, heterogeneous datasets and the importance of robust statistical validation of RBE models.

AI competitions as infrastructures of power in medical imaging

ABSTRACT This article examines how platform-based AI competitions structure power relations in medical imaging research. It focuses on two leading platforms, Kaggle and Grand Challenge, which provide organisational as well as infrastructural support to run AI competitions. In dialogue with critical AI and platform studies research, we investigate how such competitions are organised – under which infrastructural conditions and by whom – and how this shapes processes of model production and evaluation. To address these concerns, we have collected data from 118 medical image AI competitions on Kaggle and Grand Challenge, organised between January 2017 and May 2022. In addition, a variety of platform boundary resources – platform documentation, competition descriptions, dataset descriptions, and competition leaderboards – have been gathered. The analysis of these materials shows, first, that platforms direct the AI development process by requiring substantial financial resources, defining which institutions can host a competition and under which conditions. Second, competition organisers define dataset diversity and the generalisability of models. As most datasets are constructed with data from hospitals in North America, Western Europe and China, the application of models to different geographical contexts is potentially limited. Finally, competition participants influence model development through the institutional, demographic, and disciplinary contexts in which they operate. Overall, the examination demonstrates the importance of critically interrogating the entire medical AI research pipeline, including the definition of research problems, the construction of datasets as well as model production and evaluation.

Multi-temporal change detection of asbestos roofing: A hybrid object-based deep learning framework with post-classification structure

The detection of changes in asbestos roofing is vital due to the significant health risks associated with asbestos exposure. Utilising remote sensing for this task is promising, yet it encounters challenges like misregistration and variations in illumination within temporal images. This study presents a novel object-based deep learning framework with a post-classification structure designed specifically to detect changes in asbestos roofing. The method employs a hybrid deep learning model that utilises DenseNet121 to extract spatial features and integrates with either LSTM (DenseNet-LSTM) or conv1D (DenseNet-Conv1D) to capture temporal features. Using VHR aerial imagery, image patch sequences were generated for each roofing object across seven temporal points. After training the model, labels were assigned to each image patch in the sequence through binary classification. In post-processing, sequence alignment based on Hamming distance was used to enhance label assignments by aligning them with a pre-determined set of valid label sequences. Finally, the temporal change map was generated by comparing label transitions across temporal points. The results highlighted the model's robust capability to address inconsistencies across temporal points without relying on preprocessing techniques to manage such temporal discrepancies. The DenseNet-LSTM and DenseNet-Conv1D models demonstrated comparable classification performance in asbestos roofing detection, achieving average overall accuracies of 95.8% and 96.0%, respectively. Furthermore, the model's generalisability was underscored by a test on an independent dataset, which produced up to 94% detection accuracy for asbestos class. While the framework has limitation in detecting painted asbestos roofing and partial changes, it offers a reproducible approach for change detection tasks that target a specific roofing type, rather than detecting changes in all roofing types within remote sensing imagery.

Impact of brain parcellation on prediction performance in models of cognition and demographics.

Brain connectivity analysis begins with the selection of a parcellation scheme that will define brain regions as nodes of a network whose connections will be studied. Brain connectivity has already been used in predictive modelling of cognition, but it remains unclear if the resolution of the parcellation used can systematically impact the predictive model performance. In this work, structural, functional and combined connectivity were each defined with five different parcellation schemes. The resolution and modality of the parcellation schemes were varied. Each connectivity defined with each parcellation was used to predict individual differences in age, education, sex, executive function, self-regulation, language, encoding and sequence processing. It was found that low-resolution functional parcellation consistently performed above chance at producing generalisable models of both demographics and cognition. However, no single parcellation scheme showed a superior predictive performance across all cognitive domains and demographics. In addition, although parcellation schemes impacted the graph theory measures of each connectivity type (structural, functional and combined), these differences did not account for the out-of-sample predictive performance of the models. Taken together, these findings demonstrate that while high-resolution parcellations may be beneficial for modelling specific individual differences, partial voluming of signals produced by the higher resolution of the parcellation likely disrupts model generalisability.

Understanding Episode Hardness in Few-Shot Learning.

Achieving generalization for deep learning models has usually suffered from the bottleneck of annotated sample scarcity. As a common way of tackling this issue, few-shot learning focuses on "episodes", i.e. sampled tasks that help the model acquire generalizable knowledge onto unseen categories - better the episodes, the higher a model's generalisability. Despite extensive research, the characteristics of episodes and their potential effects are relatively less explored. A recent paper discussed that different episodes exhibit different prediction difficulties, and coined a new metric "hardness" to quantify episodes, which however is too wide-range for an arbitrary dataset and thus remains impractical for realistic applications. In this paper therefore, we for the first time conduct an algebraic analysis of the critical factors influencing episode hardness supported by experimental demonstrations, that reveal episode hardness to largely depend on classes within an episode, and importantly propose an efficient pre-sampling hardness assessment technique named Inverse-Fisher Discriminant Ratio (IFDR). This enables sampling hard episodes at the class level via class-level (cl) sampling scheme that drastically decreases quantification cost. Delving deeper, we also develop a variant called class-pair-level (cpl) sampling, which further reduces the sampling cost while guaranteeing the sampled distribution. Finally, comprehensive experiments conducted on benchmark datasets verify the efficacy of our proposed method. Codes are available at: https://github.com/PRIS-CV/class-level-sampling.

Elucidating salient site-specific functional connectivity features and site-invariant biomarkers in schizophrenia via deep neural networks

Schizophrenia is a highly heterogeneous disorder and salient functional connectivity (FC) features have been observed to vary across study sites, warranting the need for methods that can differentiate between site-invariant FC biomarkers and site-specific salient FC features. We propose a technique named Semi-supervised learning with data HaRmonisation via Encoder-Decoder-classifier (SHRED) to examine these features from resting state functional magnetic resonance imaging scans gathered from four sites. Our approach involves an encoder-decoder-classifier architecture that simultaneously performs data harmonisation and semi-supervised learning (SSL) to deal with site differences and labelling inconsistencies across sites respectively. The minimisation of reconstruction loss from SSL was shown to improve model performance even within small datasets whilst data harmonisation often led to lower model generalisability, which was unaffected using the SHRED technique. We show that our proposed model produces site-invariant biomarkers, most notably the connection between transverse temporal gyrus and paracentral lobule. Site-specific salient FC features were also elucidated, especially implicating the paracentral lobule for our local dataset. Our examination of these salient FC features demonstrates how site-specific features and site-invariant biomarkers can be differentiated, which can deepen our understanding of the neurobiology of schizophrenia.

Development of a novel robot-assisted vocabulary learning system using pure synthetic data

This study investigates the application of deep learning methods trained on synthetic data for the robust detection of vocabulary flashcards, an essential aspect of robot-assisted language learning (RALL). Despite its importance, flashcard detection has not been extensively researched in RALL systems, and it poses significant challenges due to the extensive data collection and annotation needed, especially given the large number of classes involved. These models need to generalize well to different real-world environments. To address these issues, a novel robotic platform designed for flashcard-based vocabulary learning is proposed, supported by a synthetic data generation pipeline using high dynamic range images (HDRIs) and synthetic human actors. The proposed pipeline offers an efficient data generation method and significantly enhances model generalisability to various environments. The proposed method was evaluated with five object detection models based on several challenging real-world datasets, each containing more than 200 class labels. The object detection models trained based on the proposed synthetic datasets (HDRI and HDRI+Humans) demonstrated outstanding performance, achieving median mean average precision (mAP) scores of 0.797 and 0.778. The proposed method achieves this performance without needing to be trained with real data. A comprehensive analysis comparing the proposed method with other synthetic data generation techniques is presented, and its potential for improving vocabulary flashcard detection in RALL systems is highlighted. The data and models are available at https://github.com/aimlab-mu/aimrobot.