Pre-trained Model Research Articles

Automatic TNM staging of colorectal cancer radiology reports using pre-trained language models

Automatic TNM staging of colorectal cancer radiology reports using pre-trained language models

Ab-amy 2.0: Predicting light chain amyloidogenic risk of therapeutic antibodies based on antibody language model.

Therapeutic antibodies have emerged as a promising treatment option for a wide range of diseases. However, the light chain of antibodies can potentially induce amyloidosis, a condition characterized by protein misfolding and aggregation, posing a significant safety concern. Therefore, it is crucial to assess the amyloidogenic risk of therapeutic antibodies during the early stages of drug development. In this study, we introduce AB-Amy 2.0, a new computational model with enhanced performance for assessing the light chain amyloidogenic risk of therapeutic antibodies. By employing pretrained protein language models (PLMs) embeddings, AB-Amy 2.0 achieves higher accuracy in amyloidogenic risk prediction compared with traditional features offering a crucial tool for early-stage identification of antibodies with low aggregation propensity. The AB-Amy 2.0 was trained on antiBERTy embeddings and utilizes the SVM algorithm, resulting in superior performance metrics. On an independent test dataset, the model achieved high sensitivity, specificity, ACC, MCC and AUC of 93.47%, 89.23%, 91.92%, 0.8261 and 0.9739, respectively. These results highlight the effectiveness and robustness of AB-Amy 2.0 in predicting light chain amyloidogenic risk accurately. To facilitate user-friendly access, we have developed an online web server (http://i.uestc.edu.cn/AB-Amy2) and a command line tool (https://github.com/zzyywww/ABAmy2). These resources enable the broader application of this advanced model and promise to enhance the development of safer therapeutic antibodies.

PreAlgPro: Prediction of allergenic proteins with pre-trained protein language model and efficient neutral network

Allergy is a prevalent phenomenon, involving allergens such as nuts and milk. Avoiding exposure to allergens is the most effective preventive measure against allergic reactions. However, current homology-based methods for identifying allergenic proteins encounter challenges when dealing with non-homologous data. Traditional machine learning approaches rely on manually extracted features, which lack important protein functional characteristics, including evolutionary information. Consequently, there is still considerable room for improvement in existing methods. In this study, we present PreAlgPro, a method for identifying allergenic proteins based on pre-trained protein language models and deep learning techniques. Specifically, we employed the ProtT5 model to extract protein embedding features, replacing the manual feature extraction step. Furthermore, we devised an Attention-CNN neural network architecture to identify potential features that contribute to the classification of allergenic proteins. The performance of our model was evaluated on four independent test sets, and the experimental results demonstrate that PreAlgPro surpasses existing state-of-the-art methods. Additionally, we collected allergenic protein samples to validate the robustness of the model and conducted an analysis of model interpretability.

CAST: An innovative framework for Cross-dimensional Attention Structure in Transformers

Dominant Transformer-based approaches rely solely on attention mechanisms and their variations, primarily emphasizing capturing crucial information within the temporal dimension. For enhanced performance, we introduce a novel architecture for Cross-dimensional Attention Structure in Transformers (CAST), which presents an innovative approach in Transformer-based models, emphasizing attention mechanisms across both temporal and spatial dimensions. The core component of CAST, the cross-dimensional attention structure (CAS), captures dependencies among multivariable time series in both temporal and spatial dimensions. The Static Attention Mechanism (SAM) is incorporated to simplify and enhance multivariate time series forecasting performance. This integration effectively reduces complexity, leading to a more efficient model. CAST demonstrates robust and efficient capabilities in predicting multivariate time series, with the simplicity of SAM broadening its applicability to various tasks. Beyond time series forecasting, CAST also shows promise in CV classification tasks. By integrating CAS into pre-trained image models, CAST facilitates spatiotemporal reasoning. Experimental results highlight the superior performance of CAST in time series forecasting and its competitive edge in CV classification tasks.

Adaptive class token knowledge distillation for efficient vision transformer

The Vision Transformer (ViT) outperforms Convolutional Neural Networks (CNNs) but at the cost of significantly higher computational demands. Knowledge Distillation (KD) has shown promise in compressing complex networks by transferring knowledge from a large pre-trained model to a smaller one. However, current KD methods for ViT often rely on CNNs as teachers or neglect the importance of class token ([CLS]) information, resulting in ineffective distillation of ViT’s unique knowledge. In this paper, we propose Adaptive Class token Knowledge Distillation ([CLS]-KD), which fully exploits information from the class token and patches in ViT. For class embedding (CLS) distillation, the intermediate CLS of the student model is aligned with the corresponding CLS of the teacher model through a projector. Furthermore, we introduce CLS-patch attention map distillation, where an attention map between the CLS and patch embeddings is generated and matched at each layer. This empowers the student model to learn how to dynamically extract patch embedding information into the CLS under teacher guidance. Finally, we propose Adaptive Layer-wise Distillation (ALD) to mitigate the imbalance in distillation effects varying with the depth of layers. This method assigns greater weight to the losses in layers where the training discrepancies between the teacher and student models are larger during distillation. Through these strategies, [CLS]-KD consistently surpasses existing state-of-the-art methods on the ImageNet-1K dataset across various teacher-student configurations. Furthermore, the proposed method demonstrates its generalization capability through transfer learning experiments on the CIFAR-10, CIFAR-100, and CALTECH-256 datasets.

AfroPALM - Afrocentric Palm oil Adulteration Learning Models: an End-to-end Deep Learning Approach for Detection of Palm Oil Adulteration

In Africa, rapid population growth and a focus on increasing food production have often overlooked the crucial aspect of food safety, leading to the highest per capita incidences of foodborne illness globally. This underscores the imperative to address the food safety challenges on the continent. A critical concern is the widespread adulteration of red palm oil across West Africa, often involving harmful red azo dyes, posing significant health risks. Current detection methods, reliant on laboratory procedures, are not only time-consuming and expensive but also impractical for broad implementation in local markets. To address these limitations, we propose an end-to-end deep learning approach that bypasses the need for manual feature extraction and laboratory-based analyses. Utilizing high- resolution imaging technology, our approach can objectively and reliably detect palm oil adulteration directly from raw image data. In our study, we developed a deep convolutional neural network dubbed AfroPALM-Custom, specifically designed for detecting palm oil adulteration in African markets. AfroPALM-Custom was pivotal in refining our deep learning approach, achieving a test accuracy of 90.63% and an F1 score of 90.98%. We later adapted mobile-efficient pretrained models, namely SqueezeNet1.1 and GhostNetV1—mobile—fine-tuned and as well dubbed AfroPALM-GhostNet and AfroPALM-SqueezeNet. In performance, AfroPALM-GhostNet and AfroPALM-SqueezeNet achieved test accuracies of 96.29% and 91.16%, respectively, and F1-scores of 96.57% and 91.95%. This proficiency in identifying adulterated palm oil demonstrates a significant advancement in food safety solutions for African markets, offering a practical and scalable approach for implementation within local marketplaces where resources are limited and laboratory methods are impractical.© 2001 Elsevier Science. All rights reserved

Advancements in Data Augmentation and Transfer Learning: A Comprehensive Survey to Address Data Scarcity Challenges

Abstract: Deep Learning (DL) models have demonstrated remarkable proficiency in image classification and recognition tasks, surpassing human capabilities. The observed enhancement in performance can be attributed to the utilization of extensive datasets. Nevertheless, DL models have huge data requirements. Widening the learning capability of such models from limited samples even today remains a challenge, given the intrinsic constraints of small datasets. The trifecta of challenges, encompassing limited labeled datasets, privacy, poor generalization performance, and the costliness of annotations, further compounds the difficulty in achieving robust model performance. Overcoming the challenge of expanding the learning capabilities of Deep Learning models with limited sample sizes remains a pressing concern even today. To address this critical issue, our study conducts a meticulous examination of established methodologies, such as Data Augmentation and Transfer Learning, which offer promising solutions to data scarcity dilemmas. Data Augmentation, a powerful technique, amplifies the size of small datasets through a diverse array of strategies. These encompass geometric transformations, kernel filter manipulations, neural style transfer amalgamation, random erasing, Generative Adversarial Networks, augmentations in feature space, and adversarial and meta- learning training paradigms. : Furthermore, Transfer Learning emerges as a crucial tool, leveraging pre-trained models to facilitate knowledge transfer between models or enabling the retraining of models on analogous datasets. Through our comprehensive investigation, we provide profound insights into how the synergistic application of these two techniques can significantly enhance the performance of classification tasks, effectively magnifying scarce datasets. This augmentation in data availability not only addresses the immediate challenges posed by limited datasets but also unlocks the full potential of working with Big Data in a new era of possibilities in DL applications.

Enhancing Prediction Stability and Performance in LIBS Analysis Using Custom CNN Architectures

LIBS-based analysis has experienced an ever-increasing interest in recent years as a well-suited technique for chemical analysis tasks relying on elemental fingerprinting. This method stands out for its ability to offer rapid, simultaneous multi-element analysis with the advantage of portability. In the context of this research, our aim is to bridge the gap between the analysis of simulated and real data to better account for variations in plasma temperature and electron density, which are typically not considered in LIBS analysis. To achieve this, we employ two distinct methodologies, PLS and CNNs, to construct predictive models for predicting the concentration of the 24 elements within each LIBS spectrum. The initial phase of our investigation concentrates on the training and testing of these models using simulated LIBS data, with results evaluated through RMSEP values. The IQR and median RMSEP values for all the elements demonstrate that CNNs consistently achieved values below 0.01, while PLS results ranged from 0.01 to 0.05, highlighting the superior stability and predictive accuracy of CNNs model. In the next phase, we applied the pre-trained models to analyze the real LIBS spectra, consistently identifying Aluminum (Al), Iron (Fe), and Silicon (Si) as having the highest predicted concentrations. The overall predicted values were approximately 0.5 for Al, 0.6 for Si, and 0.04 for Fe. In the third phase, deliberate adjustments are made to the training parameters and architecture of the proposed CNNs model to force the network to emphasize specific elements, prioritizing them over other components present in each real LIBS spectrum. The generation of the three modified versions of the initially proposed CNNs allows us to explore the impact of regularization, sample weighting, and a customized loss function on prediction outcomes. Some elements emerge during the prediction phase, with Calcium (Ca), Magnesium (Mg), Zinc (Zn), Titanium (Ti), and Gallium (Ga) exhibiting more pronounced patterns.

사전학습 모델 기반 발화 동영상 멀티 모달 감정 인식

Recently, as the demand for non-face-to-face counseling has rapidly increased, the need for emotion recognition technology that combines various aspects such as text, voice, and facial expressions is being emphasized. In this paper, we address issues such as the dominance of non-Korean data and the imbalance of emotion labels in existing datasets like FER-2013, CK+, and AFEW by using Korean video data. We propose methods to enhance multimodal emotion recognition performance in videos by integrating the strengths of image modality with text modality. A pre-trained model is used to overcome the limitations caused by small training data. A GPT-4-based LLM model is applied to text, and a pre-trained model based on VGG-19 architecture is fine-tuned to facial expression images. The method of extracting representative emotions by combining the emotional results of each aspect extracted using a pre-trained model is as follows. Emotion information extracted from text was combined with facial expression changes in a video. If there was a sentiment mismatch between the text and the image, we applied a threshold that prioritized the text-based sentiment if it was deemed trustworthy. Additionally, as a result of adjusting representative emotions using emotion distribution information for each frame, performance was improved by 19% based on F1-Score compared to the existing method that used average emotion values for each frame.

Automated anonymization of radiology reports: comparison of publicly available natural language processing and large language models.

Medical reports, governed by HIPAA regulations, contain personal health information (PHI), restricting secondary data use. Utilizing natural language processing (NLP) and large language models (LLM), we sought to employ publicly available methods to automatically anonymize PHI in free-text radiology reports. We compared two publicly available rule-based NLP models (spaCy; NLPac, accuracy-optimized; NLPsp, speed-optimized; iteratively improved on 400 free-text CT-reports (test set)) and one offline LLM approach (LLM-model, LLaMa-2, Meta-AI) for PHI-anonymization. The three models were tested on 100 randomly selected chest CT reports. Two investigators assessed the anonymization of occurring PHI entities and whether clinical information was removed. Subsequently, precision, recall, and F1 scores were calculated. NLPac and NLPsp successfully removed all instances of dates (n = 333), medical record numbers (MRN) (n = 6), and accession numbers (ACC) (n = 92). The LLM model removed all MRNs, 96% of ACCs, and 32% of dates. NLPac was most consistent with a perfect F1-score of 1.00, followed by NLPsp with lower precision (0.86) and F1-score (0.92) for dates. The LLM model had perfect precision for MRNs, ACCs, and dates but the lowest recall for ACC (0.96) and dates (0.52), corresponding F1 scores of 0.98 and 0.68, respectively. Names were removed completely or majorly (i.e., one first or family name non-anonymized) in 100% (NLPac), 72% (NLPsp), and 90% (LLM-model). Importantly, NLPac and NLPsp did not remove medical information, while the LLM model did in 10% (n = 10). Pre-trained NLP models can effectively anonymize free-text radiology reports, while anonymization with the LLM model is more prone to deleting medical information. Question This study compares NLP and locally hosted LLM techniques to ensure PHI anonymization without losing clinical information. Findings Pre-trained NLP models effectively anonymized radiology reports without removing clinical data, while a locally hosted LLM was less reliable, risking the loss of important information. Clinical relevance Fast, reliable, automated anonymization of PHI from radiology reports enables HIPAA-compliant secondary use, facilitating advanced applications like LLM-driven radiology analysis while ensuring ethical handling of sensitive patient data.

Clinical information extraction for lower-resource languages and domains with few-shot learning using pretrained language models and prompting

Abstract A vast amount of clinical data are still stored in unstructured text. Automatic extraction of medical information from these data poses several challenges: high costs of clinical expertise, restricted computational resources, strict privacy regulations, and limited interpretability of model predictions. Recent domain adaptation and prompting methods using lightweight masked language models showed promising results with minimal training data and allow for application of well-established interpretability methods. We are first to present a systematic evaluation of advanced domain-adaptation and prompting methods in a lower-resource medical domain task, performing multi-class section classification on German doctor’s letters. We evaluate a variety of models, model sizes (further-pre)training and task settings, and conduct extensive class-wise evaluations supported by Shapley values to validate the quality of small-scale training data and to ensure interpretability of model predictions. We show that in few-shot learning scenarios, a lightweight, domain-adapted pretrained language model, prompted with just 20 shots per section class, outperforms a traditional classification model, by increasing accuracy from $48.6\%$ to $79.1\%$ . By using Shapley values for model selection and training data optimization, we could further increase accuracy up to $84.3\%$ . Our analyses reveal that pretraining of masked language models on general-language data is important to support successful domain-transfer to medical language, so that further-pretraining of general-language models on domain-specific documents can outperform models pretrained on domain-specific data only. Our evaluations show that applying prompting based on general-language pretrained masked language models combined with further-pretraining on medical-domain data achieves significant improvements in accuracy beyond traditional models with minimal training data. Further performance improvements and interpretability of results can be achieved, using interpretability methods such as Shapley values. Our findings highlight the feasibility of deploying powerful machine learning methods in clinical settings and can serve as a process-oriented guideline for lower-resource languages and domains such as clinical information extraction projects.

A Comprehensive Survey on Brain Tumor Detection and Classification Techniques Using Machine Learning and Deep Learning Models

Abstract: Brain tumor detection and classification have become critical areas of research due to the complexities and challenges involved in accurate diagnosis. With the advancement of machine learning and deep learning techniques, numerous intelligent systems have been proposed to automate and enhance the process of detecting and classifying brain tumors. This survey paper reviews recent approaches and techniques used for brain tumor detection, focusing primarily on deep learning architectures like Convolutional Neural Networks (CNNs) and pre-trained models such as VGG, ResNet, and EfficientNet. In particular, methods using MRI images to classify tumors into categories such as glioma, meningioma, pituitary tumors, and healthy tissue are explored. A detailed analysis of various models, their architectures, training strategies, and evaluation metrics is provided. Moreover, the paper identifies the key challenges faced in the field, including imbalanced datasets, generalization issues, and computational efficiency. By comparing different approaches, we aim to highlight the strengths and limitations of each method and provide insights into how future research can address these challenges. The paper concludes with a discussion on promising future directions in the application of intelligent techniques for brain tumor detection, including the integration of multimodal data and the development of more interpretable models. This survey serves as a comprehensive resource for researchers interested in understanding the current landscape of brain tumor detection using advanced deep learning techniques.

Developing healthcare language model embedding spaces

Pre-trained Large Language Models (LLMs) have revolutionised Natural Language Processing (NLP) tasks, but often struggle when applied to specialised domains such as healthcare. The traditional approach of pre-training on large datasets followed by task-specific fine-tuning is resource-intensive and poorly aligned with the constraints of many healthcare settings. This presents a significant challenge for deploying LLM-based NLP solutions in medical contexts, where data privacy, computational resources, and domain-specific language pose unique obstacles.This study aims to develop and evaluate efficient methods for adapting smaller LLMs to healthcare-specific datasets and tasks. We seek to identify pre-training approaches that can effectively instil healthcare competency in compact LLMs under tight computational budgets, a crucial capability for responsible and sustainable deployment in local healthcare settings.We explore three specialised pre-training methods to adapt smaller LLMs to different healthcare datasets: traditional Masked Language modelling (MLM), Deep Contrastive Learning for Unsupervised Textual Representations (DeCLUTR), and a novel approach utilising metadata categories from healthcare settings. These methods are assessed across multiple healthcare datasets, with a focus on downstream document classification tasks. We evaluate the performance of the resulting LLMs through classification accuracy and analysis of the derived embedding spaces.Contrastively trained models consistently outperform other approaches on classification tasks, delivering strong performance with limited labelled data and fewer model parameter updates. While our novel metadata-based pre-training does not further improve classifications across datasets, it yields interesting embedding cluster separability. Importantly, all domain-adapted LLMs outperform their publicly available, general-purpose base models, validating the importance of domain specialisation.This research demonstrates the efficacy of specialised pre-training methods in adapting compact LLMs to healthcare tasks, even under resource constraints. We provide guidelines for pre-training specialised healthcare LLMs and motivate continued inquiry into contrastive objectives. Our findings underscore the potential of these approaches for aligning small LLMs with privacy-sensitive medical tasks, offering a path toward more efficient and responsible NLP deployment in healthcare settings. This work contributes to the broader goal of making advanced NLP techniques accessible and effective in specialised domains, particularly where resource limitations and data sensitivity are significant concerns.

Cost-Effective Dust Detection on Solar PV Panels through Deep Learning: A Step towards Automated Maintenance Systems

Abstract: Accumulation of dust on solar panels impacts the overall efficiency and the amount of energy it produces. Detecting and mitigating dust accumulation is therefore crucial for optimizing solar energy production. While various techniques exist for detecting dust to schedule cleaning, many of these methods uses licensed software like MATLAB, which can be financially burdensome. This study proposes the use of modified pre-trained ResNet-50 model architecture with an adjusted fully connected layer for binary classification. An experimental setup was installed utilizing a single 75 Wp panel with an inclination maintained at 30-degree angle. The fine dirt particles were artificially introduced and datasets of images of clean and dusty panels were collected from five different sides were taken, to compensate for the surface reflectance from the PV panel due to camera angles. Those datasets were used to train and test the model and the accuracy achieved was 90%. The model's ability to detect dust with minimal false positives ensures more efficient maintenance scheduling. This research demonstrates the potential of AI-driven dust detection systems to enhance the operational efficiency of solar PV installations.

Construction of a combined prognostic model for pancreatic ductal adenocarcinoma based on deep learning and digital pathology images

BackgroundDeep learning has made significant advancements in the field of digital pathology, and the integration of multiple models has further improved accuracy. In this study, we aimed to construct a combined prognostic model using deep learning-extracted features from digital pathology images of pancreatic ductal adenocarcinoma (PDAC) alongside clinical predictive indicators and to explore its prognostic value.MethodsA retrospective analysis was conducted on 142 postoperative pathologically confirmed PDAC cases. These cases were divided into training (n = 114) and testing sets (n = 28) at an 8:2 ratio. Tumor whole-slide imaging features were extracted and screened to construct a pathological risk model based on a pre-trained deep learning model. Clinical and pathological data from the training set were used to select independent predictive factors for PDAC and establish a clinical risk model using LASSO, univariate, and multivariate Cox regression analyses. Based on the pathological and clinical risk models, a combined model was developed. The Harrell concordance index (C-index) was computed to assess the predictive performance of each model for PDAC survival prognosis.ResultsFor the training and testing sets, the C-index values for the clinical risk model were 0.76 and 0.75, respectively; for the pathological risk model, they were 0.82 and 0.73, respectively; and for the combined model, they were 0.86 and 0.77, respectively. The combined model exhibited appropriate calibration at 1-, 3-, and 5-year time points, as well as a superior area under the curve of the receiver operating characteristic curve and clinical net benefit compared to the single models.ConclusionsIntegrating the pathological and clinical risk models may provide a higher predictive value for survival prognosis.

Speech Emotion Recognition Using Transfer Learning: Integration of Advanced Speaker Embeddings and Image Recognition Models

Automatic Speech Emotion Recognition (SER) plays a vital role in making human–computer interactions more natural and effective. A significant challenge in SER development is the limited availability of diverse emotional speech datasets, which hinders the application of advanced deep learning models. Transfer learning is a machine learning technique that helps address this issue by utilizing knowledge from pre-trained models to improve performance on a new task in a target domain, even with limited data. This study investigates the use of transfer learning from various pre-trained networks, including speaker embedding models such as d-vector, x-vector, and r-vector, and image classification models like AlexNet, GoogLeNet, SqueezeNet, ResNet-18, and ResNet-50. We also propose enhanced versions of the x-vector and r-vector models incorporating Multi-Head Attention Pooling and Angular Margin Softmax, alongside other architectural improvements. Additionally, reverberation from the Room Impulse Response datasets was added to the speech utterances to diversify and augment the available data. Notably, the enhanced r-vector model achieved classification accuracies of 74.05% Unweighted Accuracy (UA) and 73.68% Weighted Accuracy (WA) on the IEMOCAP dataset, and 80.25% UA and 79.81% WA on the CREMA-D dataset, outperforming the existing state-of-the-art methods. This study shows that using cross-domain transfer learning is beneficial for low-resource emotion recognition. The enhanced models developed in other domains (for non-emotional tasks) can further improve the accuracy of SER.

Detection of Subarachnoid Hemorrhage Using CNN with Dynamic Factor and Wandering Strategy-Based Feature Selection.

Objectives: Subarachnoid Hemorrhage (SAH) is a serious neurological emergency case with a higher mortality rate. An automatic SAH detection is needed to expedite and improve identification, aiding timely and efficient treatment pathways. The existence of noisy and dissimilar anatomical structures in NCCT images, limited availability of labeled SAH data, and ineffective training causes the issues of irrelevant features, overfitting, and vanishing gradient issues that make SAH detection a challenging task. Methods: In this work, the water waves dynamic factor and wandering strategy-based Sand Cat Swarm Optimization, namely DWSCSO, are proposed to ensure optimum feature selection while a Parametric Rectified Linear Unit with a Stacked Convolutional Neural Network, referred to as PRSCNN, is developed for classifying grades of SAH. The DWSCSO and PRSCNN surpass current practices in SAH detection by improving feature selection and classification accuracy. DWSCSO is proposed to ensure optimum feature selection, avoiding local optima issues with higher exploration capacity and avoiding the issue of overfitting in classification. Firstly, in this work, a modified region-growing method was employed on the patient Non-Contrast Computed Tomography (NCCT) images to segment the regions affected by SAH. From the segmented regions, the wide range of patterns and irregularities, fine-grained textures and details, and complex and abstract features were extracted from pre-trained models like GoogleNet, Visual Geometry Group (VGG)-16, and ResNet50. Next, the PRSCNN was developed for classifying grades of SAH which helped to avoid the vanishing gradient issue. Results: The DWSCSO-PRSCNN obtained a maximum accuracy of 99.48%, which is significant compared with other models. The DWSCSO-PRSCNN provides an improved accuracy of 99.62% in CT dataset compared with the DL-ICH and GoogLeNet + (GLCM and LBP), ResNet-50 + (GLCM and LBP), and AlexNet + (GLCM and LBP), which confirms that DWSCSO-PRSCNN effectively reduces false positives and false negatives. Conclusions: the complexity of DWSCSO-PRSCNN was acceptable in this research, for while simpler approaches appeared preferable, they failed to address problems like overfitting and vanishing gradients. Accordingly, the DWSCSO for optimized feature selection and PRSCNN for robust classification were essential for handling these challenges and enhancing the detection in different clinical settings.

Automated Recognition of Submerged Body-like Objects in Sonar Images Using Convolutional Neural Networks

The Police Robot for Inspection and Mapping of Underwater Evidence (PRIME) is an uncrewed surface vehicle (USV) currently being developed for underwater search and recovery teams to assist in crime scene investigation. The USV maps underwater scenes using sidescan sonar (SSS). Test exercises use a clothed mannequin lying on the seafloor as a target object to evaluate system performance. A robust, automated method for detecting human body-shaped objects is required to maximise operational functionality. The use of a convolutional neural network (CNN) for automatic target recognition (ATR) is proposed. SSS image data acquired from four different locations during previous missions were used to build a dataset consisting of two classes, i.e., a binary classification problem. The target object class consisted of 166 196 × 196 pixel image snippets of the underwater mannequin, whereas the non-target class consisted of 13,054 examples. Due to the large class imbalance in the dataset, CNN models were trained with six different imbalance ratios. Two different pre-trained models (ResNet-50 and Xception) were compared, and trained via transfer learning. This paper presents results from the CNNs and details the training methods used. Larger datasets are shown to improve CNN performance despite class imbalance, achieving average F1 scores of 97% in image classification. Average F1 scores for target vs background classification with unseen data are only 47% but the end result is enhanced by combining multiple weak classification results in an ensemble average. The combined output, represented as a georeferenced heatmap, accurately indicates the target object location with a high detection confidence and one false positive of low confidence. The CNN approach shows improved object detection performance when compared to the currently used ATR method.

ULMFiT: Universal Language Model Fine-Tuning for Text Classification

While inductive transfer learning has revolutionized computer vision, current approaches to natural language processing still need training from the ground up and task-specific adjustments. As a powerful transfer learning approach applicable to any NLP activity, we provide Universal Language Model Fine-tuning (ULMFiT) and outline essential strategies for language model fine-tuning. With an error reduction of 18–24% on most datasets, our technique considerably surpasses the state-of-the-art on six text categorization tasks. Additionally, it achieves the same level of performance as training on 100 times more data with only 100 annotated examples. We have made our pretrained models and code publicly available.

Pop-cosmos: Scaleable Inference of Galaxy Properties and Redshifts with a Data-driven Population Model

We present an efficient Bayesian method for estimating individual photometric redshifts and galaxy properties under a pretrained population model (pop-cosmos) that was calibrated using purely photometric data. This model specifies a prior distribution over 16 stellar population synthesis (SPS) parameters using a score-based diffusion model, and includes a data model with detailed treatment of nebular emission. We use a GPU-accelerated affine-invariant ensemble sampler to achieve fast posterior sampling under this model for 292,300 individual galaxies in the COSMOS2020 catalog, leveraging a neural network emulator (Speculator) to speed up the SPS calculations. We apply both the pop-cosmos population model and a baseline prior inspired by Prospector-α, and compare these results to published COSMOS2020 redshift estimates from the widely used EAZY and LePhare codes. For the ∼12,000 galaxies with spectroscopic redshifts, we find that pop-cosmos yields redshift estimates that have minimal bias (∼10−4), high accuracy (σ MAD = 7 × 10−3), and a low outlier rate (1.6%). We show that the pop-cosmos population model generalizes well to galaxies fainter than its r < 25 mag training set. The sample we have analyzed is ≳3× larger than has previously been possible via posterior sampling with a full SPS model, with average throughput of 15 GPU-sec per galaxy under the pop-cosmos prior, and 0.6 GPU-sec per galaxy under the Prospector prior. This paves the way for principled modeling of the huge catalogs expected from upcoming Stage IV galaxy surveys.