Pre-trained Weights Research Articles

Cross-Modal self-supervised vision language pre-training with multiple objectives for medical visual question answering

Medical Visual Question Answering (VQA) is a task that aims to provide answers to questions about medical images, which utilizes both visual and textual information in the reasoning process. The absence of large-scale annotated medical VQA datasets presents a formidable obstacle to training a medical VQA model from scratch in an end-to-end manner. Existing works have been using image captioning dataset in the pre-training stage and fine-tuning to downstream VQA tasks. Following the same paradigm, we use a collection of public medical image captioning datasets to pre-train multimodality models in a self-supervised setup, and fine-tune to downstream medical VQA tasks. In the work, we propose a method that featured with Cross-Modal pre-training with Multiple Objectives (CMMO), which includes masked image modelling, masked language modelling, image-text matching, and image-text contrastive learning. The proposed method is designed to associate the visual features of medical images with corresponding medical concepts in captions, for learning aligned vision and language feature representations, and multi-modal interactions. The experimental results reveal that our proposed CMMO method outperforms state-of-the-art methods on three public medical VQA datasets, showing absolute improvements of 2.6%, 0.9%, and 4.0% on the VQA-RAD, PathVQA, and SLAKE dataset, respectively. We also conduct comprehensive ablation studies to validate our method, and visualize the attention maps which show a strong interpretability. The code and pre-trained weights will be released at https://github.com/pengfeiliHEU/CMMO.

Impact of Pre-trained Weights on MobileNet Architectures for Animal Classification

This study aims to address the problem of improving animal image classification accuracy using different versions of MobileNet. Accurate animal classification plays a vital role in biodiversity protection, environmental monitoring, and agriculture. The research is significant because existing studies focus on specific models and datasets, leaving a gap in the comparative performance analysis of MobileNet versions. To address this issue, MobileNet V1, V2, and V3 models were utilized, both with and without ImageNet pre-trained weights. The models were trained on a dataset composed of 30,179 images from two sources, covering 13 animal categories. The experiment involved training the models over 10 epochs using a standard configuration of the TensorFlow framework, with accuracy serving as the primary evaluation metric.The results showed that MobileNet V3Large, with pre-trained weights, achieved the highest accuracy (97.43%), outperforming V1 and V2. Using pre-trained weights consistently enhanced performance, as models without pre-training exhibited lower accuracy and slower convergence. This study contributes by providing a comprehensive comparison of MobileNet versions in animal classification tasks, demonstrating the importance of pre-training and model architecture optimization for achieving high accuracy in image classification.

Imaging phenotype evaluation from digital breast tomosynthesis data: A preliminary study

Background:Digital breast tomosynthesis (DBT) has been widely adopted as a supplemental imaging modality for diagnostic evaluation of breast cancer and confirmation studies. In this study, a deep learning-based method for characterizing breast tissue patterns in DBT data is presented. Methods:A set of 5388 2D image patches was produced from 230 right mediolateral oblique, 259 left mediolateral oblique, 18 right craniocaudal, and 15 left craniocaudal single-breast DBT studies, using slice-wise annotations of abnormalities and normal tissue. We implemented a patch classifier to predict samples according to two differing scenarios and train it using the patch dataset. First, tissue samples were classified into the following classes: malignant, benign, and normal breast tissue. Second, tissue samples were classified into the following classes: malignant mass, benign mass, benign architectural distortion, malignant architectural distortion, and normal breast tissue. We employed transfer learning and initialized the model base layers with existing pre-trained weights obtained from Globally-Aware Multiple Instance Classifier. Results:High class-wise recall values of 0.8906, 0.8541 and 0.7345 and specificities 0.9558, 0.9575 and 0.8830 were obtained for normal, benign, and malignant classification, respectively. More intricate classification yielded class-wise recall values of 0.8708, 0.8299, 0.9444 and 0.5723 and specificities 0.9406, 0.9833, 0.8943 and 0.9652 for benign mass, normal, malignant architectural distortion, and malignant mass, respectively. However, benign architectural distortion was confused with benign mass and malignant architectural distortion. Conclusions:Combining the proposed phenotype classifier with the commonly used malignant-benign-normal classification enables a more detailed assessment of digital breast tomosynthesis images.

Digital Grading Company: Deep Neural Networks for Predictive Analytics for Alternative Asset Portfolio Management

In this study, we investigate the application of deep neural networks for predictive analytics in the realm of alternative asset portfolio management, focusing on Trading Card Games (TCGs) like Pokemon. Our primary objective is to understand the optimal trading cards for investment and to determine the appropriate timing for divestment and retention of these assets. Drawing on the successful implementation of machine learning predictive analytics in traditional financial markets, we aim to extend these techniques to alternative investments. To achieve this, we employ a two-pronged approach using a comprehensive dataset of historical sales values for Pokemon TCGs spanning five years, comprising over 200 million unlabeled data points. The first approach utilizes a semi-supervised sequence learning method, where a Long Short-Term Memory (LSTM) model is initially trained as a sequence autoencoder. The pre-trained model's weights are then used in a supervised phase to fine-tune the predictions for future trading card values. The second approach involves heuristic pretraining, which leverages market-derived indicators from existing literature to bridge the gap between traditional feature engineering and deep learning methods. Our technology stack is designed to handle end-to-end data collection, analytics generation, and execution of recommended trading strategies. Simulated backtesting results demonstrate that our Digital Grading Company (DGC) portfolio model outperforms traditional indices, achieving a 357% return compared to the 61% return of the S&P 500 Index over the same period. Real-time analysis further indicates that our model consistently identifies profitable trading opportunities in the TCG market using hourly data and proprietary algorithmic trade logic, executing trades 24/7. The DGC Predictive Portfolio, based on ungraded Pokemon TCG trading cards, also shows superior performance when compared to the Nasdaq 100 Index (Invesco QQQ ETF). Our analytics framework not only predicts the direction of price movement but also provides insights into the magnitude of changes, enabling a more dynamic and responsive trading strategy. The findings suggest that the integration of deep learning models with heuristic indicators has the potential to revolutionize portfolio management in alternative investments, paving the way for more accurate and profitable decision-making in niche markets. This research highlights the promising role of machine learning in enhancing alternative asset management strategies, suggesting future applications in other asset classes. We conclude that deep neural networks, when combined with data-driven heuristics, can significantly enhance trading outcomes, making them valuable tools for investors in the evolving landscape of alternative investments.

Advances in Deep Learning for Skin Cancer Diagnosis

The most prevalent type of cancer worldwide is known as skin cancer. Early detection is critical because if left undiagnosed in the primary stage, it might be fatal. Although there are differences within the class and high inter-class similarities, it is too difficult to distinguish with the naked eye. Owing to the disease's global prevalence, a number of deep learning based automated systems were created thus far to help doctors identify skin lesions early on. Using pre-trained ImageNet weights and fine-tuning the Convolutional Neural Networks (CNNs), we trained VGG19 on the HAM10000 dataset. The optimal performance was observed with FT. The model that was created, which yielded an accuracy that was greater overall than the one used in transfer learning, was 82.4±1.9 %. By offering a second opinion and supporting the clinician's diagnosis, this performance could lower morbidity and treatment costs.

Near Infrared Spectroscopy (NIRS) Model Based on Transfer Learning

This study compares three transfer learning strategies for adapting convolutional neural network (CNN) models in near-infrared spectroscopy to new instruments. Strategy 1 freezes pre-trained weights except for the output layer; Strategy 2 fine-tunes all weights with new data; Strategy 3 adds new fully connected layers to the pre-trained model. Using tablet spectra to predict active pharmaceutical ingredient content, results show that fine-tuning all network weights (Strategy 2) yields the best performance. The study also finds that moderate training sample sizes balance model accuracy and resource expenditure. Transfer learning is effective for near-infrared model transfer.

Accelerating multi-coil MR image reconstruction using weak supervision.

Deep-learning-based MR image reconstruction in settings where large fully sampled dataset collection is infeasible requires methods that effectively use both under-sampled and fully sampled datasets. This paper evaluates a weakly supervised, multi-coil, physics-guided approach to MR image reconstruction, leveraging both dataset types, to improve both the quality and robustness of reconstruction. A physics-guided end-to-end variational network (VarNet) is pretrained in a self-supervised manner using a 4 under-sampled dataset following the self-supervised learning via data undersampling (SSDU) methodology. The pre-trained weights are transferred to another VarNet, which is fine-tuned using a smaller, fully sampled dataset by optimizing multi-scale structural similarity (MS-SSIM) loss in image space. The proposed methodology is compared with fully self-supervised and fully supervised training. Reconstruction quality improvements in SSIM, PSNR, and NRMSE when abundant training data is available (the high-data regime), and enhanced robustness when training data is scarce (the low-data regime) are demonstrated using weak supervision for knee and brain MR image reconstructions at 8 and 10 acceleration, respectively. Multi-coil physics-guided MR image reconstruction using both under-sampled and fully sampled datasets is achievable with transfer learning and fine-tuning. This methodology can provide improved reconstruction quality in the high-data regime and improved robustness in the low-data regime at high acceleration rates.

Enhancing adaptivity of active sonar classifier considering loss landscape under dataset shift

We study the loss landscape of deep neural networks and use it to solve dataset shift problems in active sonar classification. The dataset shift degrades the generalization performance of supervised learning-based classifiers. When test data samples are available, fine-tuning methods can be used to mitigate the performance decrease. However, they can induce catastrophic forgetting and negative transfer because fine-tuned weights could be overfitted to the test data distribution. These problems are more severe in active sonar datasets with small amounts and less diversity. To mitigate the two problems in conventional fine-tuning, we explore the loss landscape of supervised learning-based active sonar classifiers and apply it to derive adaptative weights from pre-trained weights. Our results showed promising performance.

SSRL-UAVs: A Self-Supervised Deep Representation Learning Approach for GPS Spoofing Attack Detection in Small Unmanned Aerial Vehicles

Self-Supervised Representation Learning (SSRL) has become a potent strategy for addressing the growing threat of Global Positioning System (GPS) spoofing to small Unmanned Aerial Vehicles (UAVs) by capturing more abstract and high-level contributing features. This study focuses on enhancing attack detection capabilities by incorporating SSRL techniques. An innovative hybrid architecture integrates Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) models to detect attacks on small UAVs alongside two additional architectures, LSTM-Recurrent Neural Network (RNN) and Deep Neural Network (DNN), for detecting GPS spoofing attacks. The proposed model leverages SSRL, autonomously extracting meaningful features without the need for many labelled instances. Key configurations include LSTM-GRU, with 64 neurons in the input and concatenate layers and 32 neurons in the second layer. Ablation analysis explores various parameter settings, with the model achieving an impressive 99.9% accuracy after 10 epoch iterations, effectively countering GPS spoofing attacks. To further enhance this approach, transfer learning techniques are also incorporated, which help to improve the adaptability and generalisation of the SSRL model. By saving and applying pre-trained weights to a new dataset, we leverage prior knowledge to improve performance. This integration of SSRL and transfer learning yields a validation accuracy of 79.0%, demonstrating enhanced generalisation to new data and reduced training time. The combined approach underscores the robustness and efficiency of GPS spoofing detection in UAVs.

An automated framework for pediatric hip surveillance and severity assessment using radiographs.

Hip dysplasia is the second most common orthopedic condition in children with cerebral palsy (CP) and may result in disability and pain. The migration percentage (MP) is a widely used metric in hip surveillance, calculated based on an anterior-posterior pelvis radiograph. However, manual quantification of MP values using hip X-ray scans in current standard practice has challenges including being time-intensive, requiring expert knowledge, and not considering human bias. The purpose of this study is to develop a machine learning algorithm to automatically quantify MP values using a hip X-ray scan, and hence provide an assessment for severity, which then can be used for surveillance, treatment planning, and management. X-ray scans from 210 patients were curated, pre-processed, and manually annotated at our clinical center. Several machine learning models were trained using pre-trained weights from Inception ResNet-V2, VGG-16, and VGG-19, with different strategies (pre-processing, with and without region of interest (ROI) detection, with and without data augmentation) to find an optimal model for automatic hip landmarking. The predicted landmarks were then used by our geometric algorithm to quantify the MP value for the input hip X-ray scan. The pre-trained VGG-19 model, fine-tuned with additional custom layers, outputted the lowest mean squared error values for both train and test data, when ROI cropped images were used along with data augmentation for model training. The MP value calculated by the algorithm was compared to manual ground truth labels from our orthopedic fellows using the hip screen application for benchmarking. The results showed the feasibility of the machine learning model in automatic hip landmark detection for reliably quantifying MP value from hip X-ray scans. The algorithm could be used as an accurate and reliable tool in orthopedic care for diagnosing, severity assessment, and hence treatment and surgical planning for hip displacement.

Self-supervised in-domain representation learning for remote sensing image scene classification

Transferring the ImageNet pre-trained weights to the various remote sensing tasks has produced acceptable results and reduced the need for labeled samples. However, the domain differences between ground imageries and remote sensing images cause the performance of such transfer learning to be limited. The difficulty of annotating remote sensing images is well-known as it requires domain experts and more time, whereas unlabeled data is readily available. Recently, self-supervised learning, which is a subset of unsupervised learning, emerged and significantly improved representation learning. Recent research has demonstrated that self-supervised learning methods capture visual features that are more discriminative and transferable than the supervised ImageNet weights. We are motivated by these facts to pre-train the in-domain representations of remote sensing imagery using contrastive self-supervised learning and transfer the learned features to other related remote sensing datasets. Specifically, we used the SimSiam algorithm to pre-train the in-domain knowledge of remote sensing datasets and then transferred the obtained weights to the other scene classification datasets. Thus, we have obtained state-of-the-art results on five land cover classification datasets with varying numbers of classes and spatial resolutions. In addition, by conducting appropriate experiments, including feature pre-training using datasets with different attributes, we have identified the most influential factors that make a dataset a good choice for obtaining in-domain features. We have transferred the features obtained by pre-training SimSiam on remote sensing datasets to various downstream tasks and used them as initial weights for fine-tuning. Moreover, we have linearly evaluated the obtained representations in cases where the number of samples per class is limited. Our experiments have demonstrated that using a higher-resolution dataset during the self-supervised pre-training stage results in learning more discriminative and general representations.

465 Comparative analysis of semantic segmentation and deep regression models with supervised pre-training for accurate prediction of pig body weight from video data: Insights from industry-scale datasets

Abstract Accurate pig body weight (BW) measurement is essential for producers as it is related to pig growth, health, and marketing, yet conventional manual weighing methods are time-consuming and may cause potential stress to the animals. Although there is a growing trend towards the adoption of three-dimensional cameras coupled with computer vision techniques for pig BW estimation, their validation using industry-scale data are still limited. Among the prevailing methodologies, semantic segmentation and supervised pre-training regression are prominent models currently used. Therefore, the objectives of this study were: 1) to estimate pig BW from repeatedly measured video data obtained from a commercial setting, 2) to compare the performance of the two image analysis methods: thresholding segmentation and deep regression approaches, and 3) to evaluate the predictive ability of BW estimation models. An Intel RealSense D435 camera was installed in the commercial farm to collect top-view videos of 540 pigs biweekly at five different time points over three months. At the same time, manually measured BW records were collected using a digital weighing system. We used an automated video conversion pipeline and fine-tuned YOLOv8 to pre-process the raw depth videos. Subsequently, we acquired a total of 151,756 depth images and depth map files. Adaptive thresholding was applied to segment the pig body from the background. Four image-derived biometric features, including dorsal length, abdominal width, height, and volume, were estimated from the segmented images and fitted using ordinary least squares and random forest models. We applied transfer learning by initializing the weights of five deep learning models, including ResNet50, Xception, EfficientNetV2S, ConvNeXtBase, and Vision Transformer, with pre-trained weights from ImageNet. We then fine-tuned these models on the pig depth images. The last layer of each model was adapted to linear regression, enabling direct estimation of BW for each image without the need for additional image pre-processing steps. We employed random repeated subsampling cross-validation, dividing 80% of the pigs for training and 20% for testing, to evaluate prediction performance at each time point. The best prediction coefficients of determination and mean absolute percentage error for each time point were 0.76, 0.86, 0.90, 0.83, 0.90, and 4.57%, 3.76%, 3.01%, 3.41%, 4.84%, respectively. On average, the Xception model resulted in the best prediction coefficient of determination and mean absolute percentage error of 0.90 and 3.01%. Our results suggest that deep learning-based supervised learning models improve the prediction performance of pig BW from industry-scale depth video data.

A model for investment type recommender system based on the potential investors based on investors and experts feedback using ANFIS and MNN

This article presents an investment recommender system based on an Adaptive Neuro-Fuzzy Inference System (ANFIS) and pre-trained weights from a Multimodal Neural Network (MNN). The model is designed to support the investment process for the customers and takes into consideration seven factors to implement the proposed investment system model through the customer or potential investor data set. The system takes input from a web-based questionnaire that collects data on investors' preferences and investment goals. The data is then preprocessed and clustered using ETL tools, JMP, MATLAB, and Python. The ANFIS-based recommender system is designed with three inputs and one output and trained using a hybrid approach over three epochs with 188 data pairs and 18 fuzzy rules. The system's performance is evaluated using metrics such as RMSE, accuracy, precision, recall, and F1-score. The system is also designed to incorporate expert feedback and opinions from investors to customize and improve investment recommendations. The article concludes that the proposed ANFIS-based investment recommender system is effective and accurate in generating investment recommendations that meet investors' preferences and goals.Graphical abstract

Focal cortical dysplasia lesion segmentation using multiscale transformer

ObjectivesAccurate segmentation of focal cortical dysplasia (FCD) lesions from MR images plays an important role in surgical planning and decision but is still challenging for radiologists and clinicians. In this study, we introduce a novel transformer-based model, designed for the end-to-end segmentation of FCD lesions from multi-channel MR images.MethodsThe core innovation of our proposed model is the integration of a convolutional neural network-based encoder-decoder structure with a multiscale transformer to augment the feature representation of lesions in the global field of view. Transformer pathways, composed of memory- and computation-efficient dual-self-attention modules, leverage feature maps from varying depths of the encoder to discern long-range interdependencies among feature positions and channels, thereby emphasizing areas and channels relevant to lesions. The proposed model was trained and evaluated on a public-open dataset including MR images of 85 patients using both subject-level and voxel-level metrics.ResultsExperimental results indicate that our model offers superior performance both quantitatively and qualitatively. It successfully identified lesions in 82.4% of patients, with a low false-positive lesion cluster rate of 0.176 ± 0.381 per patient. Furthermore, the model achieved an average Dice coefficient of 0.410 ± 0.288, outperforming five established methods.ConclusionIntegration of the transformer could enhance the feature presentation and segmentation performance of FCD lesions. The proposed model has the potential to serve as a valuable assistive tool for physicians, enabling rapid and accurate identification of FCD lesions. The source code and pre-trained model weights are available at https://github.com/zhangxd0530/MS-DSA-NET.Critical relevance statementThis multiscale transformer-based model performs segmentation of focal cortical dysplasia lesions, aiming to help radiologists and clinicians make accurate and efficient preoperative evaluations of focal cortical dysplasia patients from MR images.Key PointsThe first transformer-based model was built to explore focal cortical dysplasia lesion segmentation.Integration of global and local features enhances the segmentation performance of lesions.A valuable benchmark for model development and comparative analyses was provided.Graphical

Detecting floating litter in freshwater bodies with semi-supervised deep learning

Researchers and practitioners have extensively utilized supervised Deep Learning methods to quantify floating litter in rivers and canals. These methods require the availability of large amount of labeled data for training. The labeling work is expensive and laborious, resulting in small open datasets available in the field compared to the comprehensive datasets for computer vision, e.g., ImageNet. Fine-tuning models pre-trained on these larger datasets helps improve litter detection performances and reduces data requirements. Yet, the effectiveness of using features learned from generic datasets is limited in large-scale monitoring, where automated detection must adapt across different locations, environmental conditions, and sensor settings. To address this issue, we propose a two-stage semi-supervised learning method to detect floating litter based on the Swapping Assignments between multiple Views of the same image (SwAV). SwAV is a self-supervised learning approach that learns the underlying feature representation from unlabeled data. In the first stage, we used SwAV to pre-train a ResNet50 backbone architecture on about 100k unlabeled images. In the second stage, we added new layers to the pre-trained ResNet50 to create a Faster R-CNN architecture, and fine-tuned it with a limited number of labeled images (≈1.8k images with 2.6k annotated litter items). We developed and validated our semi-supervised floating litter detection methodology for images collected in canals and waterways of Delft (the Netherlands) and Jakarta (Indonesia). We tested for out-of-domain generalization performances in a zero-shot fashion using additional data from Ho Chi Minh City (Vietnam), Amsterdam and Groningen (the Netherlands). We benchmarked our results against the same Faster R-CNN architecture trained via supervised learning alone by fine-tuning ImageNet pre-trained weights. The findings indicate that the semi-supervised learning method matches or surpasses the supervised learning benchmark when tested on new images from the same training locations. We measured better performances when little data (≈200 images with about 300 annotated litter items) is available for fine-tuning and with respect to reducing false positive predictions. More importantly, the proposed approach demonstrates clear superiority for generalization on the unseen locations, with improvements in average precision of up to 12.7%. We attribute this superior performance to the more effective high-level feature extraction from SwAV pre-training from relevant unlabeled images. Our findings highlight a promising direction to leverage semi-supervised learning for developing foundational models, which have revolutionized artificial intelligence applications in most fields. By scaling our proposed approach with more data and compute, we can make significant strides in monitoring to address the global challenge of litter pollution in water bodies.

Performance prediction and design optimization of a transonic rotor based on deep transfer learning

Deep transfer learning is frequently employed to address the challenges arising from limited or hard-to-obtain training data in the target domain, but its application in axial compressors has been scarcely explored thus far. In this paper, a multi-objective optimization framework of a transonic rotor is established using deep transfer learning. This framework first pre-trains deep neural networks based on the peak efficiency condition of 100% design speed and then fine-tunes the networks to predict the performance of off-design conditions based on the small training dataset. Finally, the design optimization of the transonic rotor is carried out through non-dominated sorting genetic algorithm II. Compared to neural networks that are trained directly, transfer learning models can achieve higher prediction accuracy, particularly in scenarios with small training datasets. This is because the pre-trained weights can offer a better initial state for transfer learning models. Moreover, transfer learning models can use fewer samples to obtain an approximate Pareto front, making the optimized rotor increase the isentropic efficiency at both peak efficiency and high loading conditions. The efficiency improvement of the optimized rotor is attributed to the reduction of the loss associated with the tip leakage flow by adjusting the tip loading distribution. Overall, this study fully demonstrates the effectiveness of transfer learning in predicting compressor performance, which provides a promising approach to solving high-cost compressor design problems.

Are genomic language models all you need? Exploring genomic language models on protein downstream tasks.

Large language models, trained on enormous corpora of biological sequences, are state-of-the-art for downstream genomic and proteomic tasks. Since the genome contains the information to encode all proteins, genomic language models (gLMs) hold the potential to make downstream predictions not only about DNA sequences, but also about proteins. However, the performance of gLMs on protein tasks remains unknown, due to few tasks pairing proteins with the coding DNA sequences (CDS) that can be processed by gLMs. In this work, we curated five such datasets and used them to evaluate the performance of gLMs and proteomic language models (pLMs). We show that gLMs are competitive and even outperform their pLMs counterparts on some tasks. The best performance was achieved using the retrieved CDS compared to sampling strategies. We found that training a joint genomic-proteomic model outperforms each individual approach, showing that they capture different but complementary sequence representations, as we demonstrate through model interpretation of their embeddings. Lastly, we explored different genomic tokenization schemes to improve downstream protein performance. We trained a new Nucleotide Transformer (50M) foundation model with 3mer tokenization that outperforms its 6mer counterpart on protein tasks while maintaining performance on genomics tasks. The application of gLMs to proteomics offers the potential to leverage rich CDS data, and in the spirit of the central dogma, the possibility of a unified and synergistic approach to genomics and proteomics. We make our inference code, 3mer pre-trained model weights and datasets available.

Unveiling the human-like similarities of automatic facial expression recognition: An empirical exploration through explainable ai

AbstractFacial expression recognition is vital for human behavior analysis, and deep learning has enabled models that can outperform humans. However, it is unclear how closely they mimic human processing. This study aims to explore the similarity between deep neural networks and human perception by comparing twelve different networks, including both general object classifiers and FER-specific models. We employ an innovative global explainable AI method to generate heatmaps, revealing crucial facial regions for the twelve networks trained on six facial expressions. We assess these results both quantitatively and qualitatively, comparing them to ground truth masks based on Friesen and Ekman’s description and among them. We use Intersection over Union (IoU) and normalized correlation coefficients for comparisons. We generate 72 heatmaps to highlight critical regions for each expression and architecture. Qualitatively, models with pre-trained weights show more similarity in heatmaps compared to those without pre-training. Specifically, eye and nose areas influence certain facial expressions, while the mouth is consistently important across all models and expressions. Quantitatively, we find low average IoU values (avg. 0.2702) across all expressions and architectures. The best-performing architecture averages 0.3269, while the worst-performing one averages 0.2066. Dendrograms, built with the normalized correlation coefficient, reveal two main clusters for most expressions: models with pre-training and models without pre-training. Findings suggest limited alignment between human and AI facial expression recognition, with network architectures influencing the similarity, as similar architectures prioritize similar facial regions.

MA-SAM: Modality-agnostic SAM adaptation for 3D medical image segmentation

The Segment Anything Model (SAM), a foundation model for general image segmentation, has demonstrated impressive zero-shot performance across numerous natural image segmentation tasks. However, SAM’s performance significantly declines when applied to medical images, primarily due to the substantial disparity between natural and medical image domains. To effectively adapt SAM to medical images, it is important to incorporate critical third-dimensional information, i.e., volumetric or temporal knowledge, during fine-tuning. Simultaneously, we aim to harness SAM’s pre-trained weights within its original 2D backbone to the fullest extent. In this paper, we introduce a modality-agnostic SAM adaptation framework, named as MA-SAM, that is applicable to various volumetric and video medical data. Our method roots in the parameter-efficient fine-tuning strategy to update only a small portion of weight increments while preserving the majority of SAM’s pre-trained weights. By injecting a series of 3D adapters into the transformer blocks of the image encoder, our method enables the pre-trained 2D backbone to extract third-dimensional information from input data. We comprehensively evaluate our method on five medical image segmentation tasks, by using 11 public datasets across CT, MRI, and surgical video data. Remarkably, without using any prompt, our method consistently outperforms various state-of-the-art 3D approaches, surpassing nnU-Net by 0.9%, 2.6%, and 9.9% in Dice for CT multi-organ segmentation, MRI prostate segmentation, and surgical scene segmentation respectively. Our model also demonstrates strong generalization, and excels in challenging tumor segmentation when prompts are used. Our code is available at: https://github.com/cchen-cc/MA-SAM.

A web-based mpox skin lesion detection system using state-of-the-art deep learning models considering racial diversity

The recent ‘Mpox’ outbreak, formerly known as ‘Monkeypox’, has been a significant public health concern and spread to over 110 countries globally. The challenge of clinically diagnosing mpox early on is due, in part, to its similarity to other types of rashes. Computer-aided screening tools have been proven valuable in cases where Polymerase Chain Reaction (PCR) based diagnosis is not immediately available. Deep learning methods excel at learning complex data representations, but their efficacy largely depends on adequate training data. To address this challenge, we present the “Mpox Skin Lesion Dataset Version 2.0 (MSLD v2.0)” as a follow-up to the previously released open-access dataset, one of the first to contain mpox lesion images. This dataset contains images of patients with mpox and five other non-mpox classes (chickenpox, measles, hand-foot-mouth disease, cowpox, and healthy). We benchmark the performance of several state-of-the-art deep learning models, including VGG16, ResNet50, DenseNet121, MobileNetV2, EfficientNetB3, InceptionV3, and Xception, to classify mpox and other infectious skin diseases. We leverage transfer learning implemented with pre-trained weights generated from the HAM10000 dataset, an extensive collection of pigmented skin lesion images. To reduce the impact of racial bias, we employ a color space data augmentation method to increase skin color variability during training and present a rigorous assessment of the technique’s effectiveness in skin lesion classification; we achieved the best overall accuracy of 83.59±2.11%. Finally, the developed models are incorporated within a prototype web application to analyze uploaded skin images by a user and determine whether a subject is a suspected mpox patient. We believe that the presented tool can be of significant impact in screening of mpox in case of a future outbreak.